10 Replies Latest reply: May 23, 2012 8:27 AM by user12432153 RSS

    ART Application SB00 takes too long to respond for more than 1 user

    525013
      I am providing a lot of system resources in the System IPC settigs, via the sysctl.conf file and also through the MAXACCSSERS. I am not sure why the SB00 application will only do one transaction at a time. I am quite dumbfounded with this problem. I am not sure why the application is not working properly. From what I understand with Sophia and the other ART Dev, the I should be able to handle concurrent transactions. Thats why I set up the ARTTCPL server -M setting to 900 and -x to 50. I also set the MAX ARTSTRN servers to 50. So far, I am not getting too much. I am not sure where the problem is.

      Any ideas,??

      When I run this and I do a psr, I see a lot of ARTSTRN servers sitting IDLE. I also see some TMACTIVE transactions. I am not sure why is that happening since my Database I have set MaxCur=120.


      Where is the problem in this for it not to behave as specified?


      ipcs -l

      ------ Shared Memory Limits --------
      max number of segments = 4096
      max seg size (kbytes) = 67108864
      max total shared memory (kbytes) = 17179869184
      min seg size (bytes) = 1

      ------ Semaphore Limits --------
      max number of arrays = 1024
      max semaphores per array = 250
      max semaphores system wide = 256000
      max ops per semop call = 100
      semaphore max value = 32767

      ------ Messages: Limits --------
      max queues system wide = 4096
      max size of message (bytes) = 65536
      default max size of queue (bytes) = 65536

      *****************************************************************************


      tmboot -c

      Fixed Minimums Per Processor

      SHMMIN: 1
      SHMALL: 1
      SEMMAP: SEMMNI

      Variable Minimums Per Processor

      SEMUME, A SHMMAX
      SEMMNU, * *
      Node SEMMNS SEMMSL SEMMSL SEMMNI MSGMNI MSGMAP SHMSEG

      x3 375 47 370 A + 1 79 158 6845K

      where 1 <= A <= 8.

      The number of expected application clients per processor should
      be added to each MSGMNI value.


      ***********************************************************************************
      tmunloadcf

      *RESOURCES
      IPCKEY 34701
      MASTER "KIXR"
      UID 0
      GID 0
      PERM 0600
      MAXACCESSERS 400
      MAXACLGROUPS 16384
      MAXGTT 100
      DOMAINID "KIXD"
      MAXGROUPS 100
      MAXNETGROUPS 8
      MAXMACHINES 256
      MAXQUEUES 180
      MAXDRT 0
      MAXRFT 0
      MAXRTDATA 4
      MAXSPDATA 100744
      MAXSERVERS 180
      MAXSERVICES 10000
      MAXCONV 64
      MODEL SHM
      LDBAL Y
      CMTRET COMPLETE
      MAXBUFTYPE 16
      MAXBUFSTYPE 32
      SCANUNIT 10
      SANITYSCAN 12
      DBBLWAIT 2
      BBLQUERY 30
      BLOCKTIME 10
      NOTIFY DIPIN
      SYSTEM_ACCESS FASTPATH
      MAXINTERFACES 150
      MAXOBJECTS 1000
      SIGNATURE_AHEAD 3600
      SIGNATURE_BEHIND 604800
      USIGNAL SIGUSR2
      AUTOTRAN N
      TRANTIME 30

      *MACHINES
      "x3" LMID="KIXR"
      TUXCONFIG="/opt/oracle/art11gR1/Cics_RT/sample/sample_03/config/tux/tuxconfig"
      TUXDIR="/opt/oracle/tuxedo11gR1"
      APPDIR="/opt/oracle/art11gR1/Cics_RT/sample/sample_03"
      TLOGDEVICE="/opt/oracle/art11gR1/Cics_RT/sample/sample_03/sysfile/TLOG"
      TLOGNAME="TLOG"
      TLOGSIZE=100
      ULOGPFX="/opt/oracle/art11gR1/Cics_RT/sample/sample_03/LOGS/log/ULOG"
      ENVFILE="/opt/oracle/art11gR1/Cics_RT/sample/sample_03/config/tux/envfile"
      MAXWSCLIENTS=50
      CMPLIMIT="MAXLONG,MAXLONG"
      NETLOAD=0
      SPINCOUNT=0
      MAXACLCACHE=100
      MAXOBJECTS=1000
      SICACHEENTRIESMAX="500"


      *GROUPS
      "TCP00" LMID="KIXR" GRPNO=1
      TMSCOUNT=3
      MRM=N
      "GRP00" LMID="KIXR" GRPNO=10
      ENVFILE="/opt/oracle/art11gR1/Cics_RT/sample/sample_03/config/tux/envfile"
      TMSCOUNT=3
      MRM=N
      "GRP01" LMID="KIXR" GRPNO=11
      ENVFILE="/opt/oracle/art11gR1/Cics_RT/sample/sample_03/config/tux/envfile"
      TMSCOUNT=3
      MRM=N
      "GRP02" LMID="KIXR" GRPNO=12
      OPENINFO="Oracle_XA:Oracle_XA+Acc=P/sa/passw0rd+SqlNet=orcl+SesTm=600+LogDir=/opt/oracle/art11gR1/Cics_RT/sample/sample_03/LOGS/xa+MaxCur=120+Threads=true"
      TMSNAME="TMS_ORA"
      ENVFILE="/opt/oracle/art11gR1/Cics_RT/sample/sample_03/config/tux/envfile"
      TMSCOUNT=2
      MRM=N

      *RMS

      *NETGROUPS

      *SERVERS
      "ARTADM" SRVGRP="GRP01" SRVID=10
      CLOPT="-o /opt/oracle/art11gR1/Cics_RT/sample/sample_03/LOGS/sysout/stdout_adm -e /opt/oracle/art11gR1/Cics_RT/sample/sample_03/LOGS/sysout/stderr_adm -r --"
      SEQUENCE=1
      RQPERM=0600 REPLYQ=N RPPERM=0600 MIN=1 MAX=1 CONV=N
      SYSTEM_ACCESS=FASTPATH
      MAXGEN=1 GRACE=86400 RESTART=N
      MINDISPATCHTHREADS=0 MAXDISPATCHTHREADS=1 THREADSTACKSIZE=0
      SICACHEENTRIESMAX="500"
      "ARTTCPL" SRVGRP="TCP00" SRVID=101
      CLOPT=" -- -M 900 -m 50 -x 50 -L //192.168.1.2:34702 -n //192.168.1.2:34701 +H -1"
      RQPERM=0600 REPLYQ=N RPPERM=0600 MIN=1 MAX=1 CONV=N
      SYSTEM_ACCESS=FASTPATH
      MAXGEN=1 GRACE=86400 RESTART=N
      MINDISPATCHTHREADS=0 MAXDISPATCHTHREADS=1 THREADSTACKSIZE=0
      SICACHEENTRIESMAX="500"
      "ARTCNX" SRVGRP="GRP01" SRVID=15
      CLOPT="-o /opt/oracle/art11gR1/Cics_RT/sample/sample_03/LOGS/sysout/stdout_cnx -e /opt/oracle/art11gR1/Cics_RT/sample/sample_03/LOGS/sysout/stderr_cnx -r --"
      RQADDR="QCNX015"
      RQPERM=0600 REPLYQ=Y RPPERM=0600 MIN=1 MAX=1 CONV=Y
      SYSTEM_ACCESS=FASTPATH
      MAXGEN=1 GRACE=86400 RESTART=N
      MINDISPATCHTHREADS=0 MAXDISPATCHTHREADS=1 THREADSTACKSIZE=0
      SICACHEENTRIESMAX="500"
      "ARTSTRN" SRVGRP="GRP02" SRVID=20
      CLOPT="-o /opt/oracle/art11gR1/Cics_RT/sample/sample_03/LOGS/sysout/stdout_strn -e /opt/oracle/art11gR1/Cics_RT/sample/sample_03/LOGS/sysout/stderr_strn -r -- -s KIXR -l SIMPAPP"
      RQADDR="QKIX110"
      RQPERM=0600 REPLYQ=Y RPPERM=0600 MIN=1 MAX=50 CONV=Y
      SYSTEM_ACCESS=FASTPATH
      MAXGEN=1 GRACE=86400 RESTART=N
      MINDISPATCHTHREADS=0 MAXDISPATCHTHREADS=1 THREADSTACKSIZE=0
      SICACHEENTRIESMAX="500"

      *MODULES

      *JDBCCONNPOOLS

      *SERVICES

      *INTERFACES

      *ROUTING
      ***********************************************************************************

      Edited by: user522010 on May 17, 2012 7:31 AM
        • 1. Re: ART Application SB00 takes too long to respond for more than 1 user
          784893
          Hi,

          After testing and investigating your tmunloaded UBB config file, I think there are two points for this appearance.

          1. There are two parameters in your UBB may causing this illusion. One is "MAXACCESSERS" in "*RESOURCES" section, and the other one is "MAXWSCLIENTS" in "*MACHINES" section. (You can reference http://docs.oracle.com/cd/E18050_01/tuxedo/docs11gr1/rf5/rf5.html#wp3370051 for details)
          In my test environment, I tested sample_03 (with SB00 transaction) using your UBB by enlarging these two parameters to "1500" and "500" respectively. And I invoked 1000 clients running in background. Tracing "psr" from time to time, I can see the SB00 being handled sometimes. Following is a part of it:
          Prog Name Queue Name Grp Name ID RqDone Load Done Current Service
          ----------------------------------------------------------------------------------------------------------------
          ......
          ARTSTRN QKIX110 GRP02 20 164 8200 SB00
          ARTSTRN QKIX110 GRP02 21 125 6250 SB00
          ARTSTRN QKIX110 GRP02 22 164 8200 SB00
          ARTSTRN QKIX110 GRP02 23 99 4950 SB00
          ARTSTRN QKIX110 GRP02 24 82 4100 SB03
          ARTSTRN QKIX110 GRP02 25 83 4150 SB00
          ARTSTRN QKIX110 GRP02 26 83 4150 SB00
          ARTSTRN QKIX110 GRP02 27 113 5650 SB00
          ARTSTRN QKIX110 GRP02 28 80 4000 SB03
          ARTSTRN QKIX110 GRP02 29 88 4400 SB00
          ARTSTRN QKIX110 GRP02 30 72 3600 SB00
          ARTSTRN QKIX110 GRP02 31 108 5400 SB00
          ARTSTRN QKIX110 GRP02 32 75 3750 SB00
          ARTSTRN QKIX110 GRP02 33 71 3550 SB00
          ARTSTRN QKIX110 GRP02 34 72 3600 SB00
          ARTSTRN QKIX110 GRP02 35 67 3350 SB00
          ARTSTRN QKIX110 GRP02 36 66 3300 SB00
          ARTSTRN QKIX110 GRP02 37 62 3100 SB03
          ARTSTRN QKIX110 GRP02 38 69 3450 SB00
          ARTSTRN QKIX110 GRP02 39 65 3250 SB03
          ARTSTRN QKIX110 GRP02 40 67 3350 SB00
          ARTSTRN QKIX110 GRP02 41 61 3050 SB00
          ARTSTRN QKIX110 GRP02 42 61 3050 SB03
          ARTSTRN QKIX110 GRP02 43 54 2700 SB03
          ARTSTRN QKIX110 GRP02 44 51 2550 SB00
          ARTSTRN QKIX110 GRP02 45 50 2500 SB00
          ARTSTRN QKIX110 GRP02 46 61 3050 SB03
          ARTSTRN QKIX110 GRP02 47 51 2550 SB03
          ARTSTRN QKIX110 GRP02 48 57 2850 SB00
          ARTSTRN QKIX110 GRP02 49 46 2300 SB00
          ARTSTRN QKIX110 GRP02 50 1 50 SB00
          ARTSTRN QKIX110 GRP02 51 0 0 ( IDLE )
          ARTSTRN QKIX110 GRP02 52 0 0 ( IDLE )
          ARTSTRN QKIX110 GRP02 53 0 0 ( IDLE )

          2. Another point, SB00 is a small transaction, which may run very fast. We may can't see all the "handling" status by psr except that you are always watching it. I don't know how many clients your are running. You can take a look at the "RqDone" column to figure out whether the requests have been handled.

          Is this help?

          Thanks,
          Sophia
          • 2. Re: ART Application SB00 takes too long to respond for more than 1 user
            525013
            Hi Sophia, Thank you very much for looking into this. I have never seen a PSR show this much activity, usually in my case, they are all ( IDLE ).

            It is true that SB00 is a small transaction, but the fact that you can get that many servers talking to the clients is very interesting to me.

            What are you using to simulate the clients?
            Can you tell me your ipcs -l settings.? Do you think my ipcs -l setting are fine?

            I will try out your new suggestions and let you know if it maks a difference

            Thanks.
            • 3. Re: ART Application SB00 takes too long to respond for more than 1 user
              525013
              Sophia,

              This is what I get for some of my pt output

              index=0 gtrid=x0 x482f533a x9132
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=1 gtrid=x0 x482f533a x6f89
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=2 gtrid=x0 x482f533a x532f
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=3 gtrid=x0 x482f533a x325c
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=4 gtrid=x0 x482f533a x325b
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=5 gtrid=x0 x482f533a x2636
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=6 gtrid=x0 x482f533a x1b67
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=7 gtrid=x0 x482f533a x1b09
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=8 gtrid=x0 x482f533a x19fd
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=9 gtrid=x0 x482f533a x131c
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=10 gtrid=x0 x482f533a x1232
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=11 gtrid=x0 x482f533a xf90
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=12 gtrid=x0 x482f533a xee5
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=13 gtrid=x0 x482f533a xebd
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=14 gtrid=x0 x482f533a xc05
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=15 gtrid=x0 x482f533a x9cc
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=16 gtrid=x0 x482f533a x668
              : Machine id: KIXR, Transaction status: TMGACTIVE
              Group count: 1
              index=17 gtrid=x0 x482f533a x500


              I increased the MAXACCESSERS to 1400 and the MAXWSCLIENTS TO 500. I am having a lot of pending transactions, do u think can be due to a database bottleneck?

              Is there a way to have multiple domains of ART? maybe this one KIXR domain is exhausted, but I am guessing if you have are gettin ga lot of users, then you are doing something right.

              Edited by: user522010 on May 18, 2012 2:56 PM
              • 4. Re: ART Application SB00 takes too long to respond for more than 1 user
                784893
                user522010 wrote:
                Hi Sophia, Thank you very much for looking into this. I have never seen a PSR show this much activity, usually in my case, they are all ( IDLE ).

                It is true that SB00 is a small transaction, but the fact that you can get that many servers talking to the clients is very interesting to me.

                What are you using to simulate the clients?
                Can you tell me your ipcs -l settings.? Do you think my ipcs -l setting are fine?

                I will try out your new suggestions and let you know if it maks a difference

                Thanks.
                Hi,

                I'm using s3270 commands to simulate the clients. I have a script named s3270.in, which invokes SB00 transaction with "increase, deleting, and modifying" operations. Then I let 1000 clients call "s3270 -model 3278-2 < s3270.in" in the background. That generated the earlier results. (Ref: http://x3270.bgp.nu/s3270-man.html)


                For your reference, following is my ipc settings:
                $ ipcs -l

                ------ Shared Memory Limits --------
                max number of segments = 4096
                max seg size (kbytes) = 4294967296
                max total shared memory (kbytes) = 4294967296
                min seg size (bytes) = 1

                ------ Semaphore Limits --------
                max number of arrays = 2048
                max semaphores per array = 250
                max semaphores system wide = 256000
                max ops per semop call = 100
                semaphore max value = 32767

                ------ Messages: Limits --------
                max queues system wide = 8192
                max size of message (bytes) = 65536
                default max size of queue (bytes) = 65536


                Thanks,
                Sophia
                • 5. Re: ART Application SB00 takes too long to respond for more than 1 user
                  784893
                  user522010 wrote:
                  Sophia,

                  This is what I get for some of my pt output

                  index=0 gtrid=x0 x482f533a x9132
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=1 gtrid=x0 x482f533a x6f89
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=2 gtrid=x0 x482f533a x532f
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=3 gtrid=x0 x482f533a x325c
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=4 gtrid=x0 x482f533a x325b
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=5 gtrid=x0 x482f533a x2636
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=6 gtrid=x0 x482f533a x1b67
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=7 gtrid=x0 x482f533a x1b09
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=8 gtrid=x0 x482f533a x19fd
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=9 gtrid=x0 x482f533a x131c
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=10 gtrid=x0 x482f533a x1232
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=11 gtrid=x0 x482f533a xf90
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=12 gtrid=x0 x482f533a xee5
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=13 gtrid=x0 x482f533a xebd
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=14 gtrid=x0 x482f533a xc05
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=15 gtrid=x0 x482f533a x9cc
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=16 gtrid=x0 x482f533a x668
                  : Machine id: KIXR, Transaction status: TMGACTIVE
                  Group count: 1
                  index=17 gtrid=x0 x482f533a x500


                  I increased the MAXACCESSERS to 1400 and the MAXWSCLIENTS TO 500. I am having a lot of pending transactions, do u think can be due to a database bottleneck?

                  Is there a way to have multiple domains of ART? maybe this one KIXR domain is exhausted, but I am guessing if you have are gettin ga lot of users, then you are doing something right.

                  Edited by: user522010 on May 18, 2012 2:56 PM
                  Hi,

                  I don't think multiple domains are necessary in this case. We have done a lot of tests using one domain, and we also have real customer environment working well with one domain. In another point, I feel this is probably not caused by bottleneck of database. From your above information, I can't figure out the root cause currently. Could you please further provide how you simulate the clients?

                  Thanks,
                  Sophia
                  • 6. Re: ART Application SB00 takes too long to respond for more than 1 user
                    525013
                    Hi Sophia, as you can see on my top command, I am running this against 16 cpus. You can see the ARTTCPH process is maxed out at 100%.

                    Is this a cause for concern? I am running the SB00 transaction and doing just a READ and UPDATE on the same data.

                    I am using SilkPerformer because it gives me monitoring data.

                    top - 17:16:55 up 1 day, 17:18, 1 user, load average: 1.11, 0.81, 0.42
                    Tasks: 409 total, 2 running, 407 sleeping, 0 stopped, 0 zombie
                    Cpu(s): 6.3%us, 0.1%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
                    Mem: 33554432k total, 2036188k used, 31518244k free, 139740k buffers
                    Swap: 69435384k total, 0k used, 69435384k free, 562816k cached

                    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                    26755 root 25 0 93416 7912 6092 R 100.0 0.0 5:06.71 ARTTCPH
                    26759 root 15 0 235m 22m 13m S 0.3 0.1 0:01.62 ARTSTRN
                    26773 root 15 0 235m 22m 13m S 0.3 0.1 0:01.46 ARTSTRN
                    26779 root 15 0 235m 22m 13m S 0.3 0.1 0:01.40 ARTSTRN
                    26791 root 15 0 235m 22m 13m S 0.3 0.1 0:01.76 ARTSTRN
                    26797 root 15 0 235m 22m 13m S 0.3 0.1 0:01.64 ARTSTRN
                    26800 root 15 0 235m 22m 13m S 0.3 0.1 0:01.50 ARTSTRN
                    26816 root 15 0 235m 22m 13m S 0.3 0.1 0:01.37 ARTSTRN


                    Also, I dont know why the system cpu is showig 93% idle. I feel like this is incorrect.

                    Edited by: user522010 on May 22, 2012 2:20 PM
                    • 7. Re: ART Application SB00 takes too long to respond for more than 1 user
                      784893
                      user522010 wrote:
                      Hi Sophia, as you can see on my top command, I am running this against 16 cpus. You can see the ARTTCPH process is maxed out at 100%.

                      Is this a cause for concern? I am running the SB00 transaction and doing just a READ and UPDATE on the same data.

                      I am using SilkPerformer because it gives me monitoring data.

                      top - 17:16:55 up 1 day, 17:18, 1 user, load average: 1.11, 0.81, 0.42
                      Tasks: 409 total, 2 running, 407 sleeping, 0 stopped, 0 zombie
                      Cpu(s): 6.3%us, 0.1%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
                      Mem: 33554432k total, 2036188k used, 31518244k free, 139740k buffers
                      Swap: 69435384k total, 0k used, 69435384k free, 562816k cached

                      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                      26755 root 25 0 93416 7912 6092 R 100.0 0.0 5:06.71 ARTTCPH
                      26759 root 15 0 235m 22m 13m S 0.3 0.1 0:01.62 ARTSTRN
                      26773 root 15 0 235m 22m 13m S 0.3 0.1 0:01.46 ARTSTRN
                      26779 root 15 0 235m 22m 13m S 0.3 0.1 0:01.40 ARTSTRN
                      26791 root 15 0 235m 22m 13m S 0.3 0.1 0:01.76 ARTSTRN
                      26797 root 15 0 235m 22m 13m S 0.3 0.1 0:01.64 ARTSTRN
                      26800 root 15 0 235m 22m 13m S 0.3 0.1 0:01.50 ARTSTRN
                      26816 root 15 0 235m 22m 13m S 0.3 0.1 0:01.37 ARTSTRN


                      Also, I dont know why the system cpu is showig 93% idle. I feel like this is incorrect.

                      Edited by: user522010 on May 22, 2012 2:20 PM
                      Hi,

                      I see that your ARTTCPH has run more than 5 hours, while ARTSTRN servers only run about one minute.This is a big difference. I guess that might be the cause. The ARTTCPH may be some "dead" processes in your last application(s). Actually, there is a situation, when there exists some clients which haven't completed their transactions correctly, and you have "tmshutdown" the servers, then the "ARTTCPH" can't be totally shutdown. You can see the ARTTCPH process by "ps" command. In this way, we need to kill these processes manually before the next boot up. Could you please kill the ARTTCPH processes first, and run your application again? Let's see if it can work normally in this way.

                      Thanks,
                      Sophia
                      • 8. Re: ART Application SB00 takes too long to respond for more than 1 user
                        525013
                        Sophia wrote:
                        user522010 wrote:
                        Hi Sophia, as you can see on my top command, I am running this against 16 cpus. You can see the ARTTCPH process is maxed out at 100%.

                        Is this a cause for concern? I am running the SB00 transaction and doing just a READ and UPDATE on the same data.

                        I am using SilkPerformer because it gives me monitoring data.

                        top - 17:16:55 up 1 day, 17:18, 1 user, load average: 1.11, 0.81, 0.42
                        Tasks: 409 total, 2 running, 407 sleeping, 0 stopped, 0 zombie
                        Cpu(s): 6.3%us, 0.1%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
                        Mem: 33554432k total, 2036188k used, 31518244k free, 139740k buffers
                        Swap: 69435384k total, 0k used, 69435384k free, 562816k cached

                        PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                        26755 root 25 0 93416 7912 6092 R 100.0 0.0 5:06.71 ARTTCPH
                        26759 root 15 0 235m 22m 13m S 0.3 0.1 0:01.62 ARTSTRN
                        26773 root 15 0 235m 22m 13m S 0.3 0.1 0:01.46 ARTSTRN
                        26779 root 15 0 235m 22m 13m S 0.3 0.1 0:01.40 ARTSTRN
                        26791 root 15 0 235m 22m 13m S 0.3 0.1 0:01.76 ARTSTRN
                        26797 root 15 0 235m 22m 13m S 0.3 0.1 0:01.64 ARTSTRN
                        26800 root 15 0 235m 22m 13m S 0.3 0.1 0:01.50 ARTSTRN
                        26816 root 15 0 235m 22m 13m S 0.3 0.1 0:01.37 ARTSTRN


                        Also, I dont know why the system cpu is showig 93% idle. I feel like this is incorrect.

                        Edited by: user522010 on May 22, 2012 2:20 PM
                        Hi,

                        I see that your ARTTCPH has run more than 5 hours, while ARTSTRN servers only run about one minute.This is a big difference. I guess that might be the cause. The ARTTCPH may be some "dead" processes in your last application(s). Actually, there is a situation, when there exists some clients which haven't completed their transactions correctly, and you have "tmshutdown" the servers, then the "ARTTCPH" can't be totally shutdown. You can see the ARTTCPH process by "ps" command. In this way, we need to kill these processes manually before the next boot up. Could you please kill the ARTTCPH processes first, and run your application again? Let's see if it can work normally in this way.

                        Thanks,
                        Sophia
                        I think those stale ARTTCPH might be hanging up my runs. I have had situations where only some of the clients run, the others are sort of hung, and there is som TMACTIVE transactions. Howevever, when I close my clients, the transactions go away, but maybe they are still stuck in ARTTCPH. This is why I thnk we should have more than one ARTTCPH server.

                        Do you think this is a bug?
                        • 9. Re: ART Application SB00 takes too long to respond for more than 1 user
                          784893
                          user522010 wrote:
                          Sophia wrote:
                          user522010 wrote:
                          Hi Sophia, as you can see on my top command, I am running this against 16 cpus. You can see the ARTTCPH process is maxed out at 100%.

                          Is this a cause for concern? I am running the SB00 transaction and doing just a READ and UPDATE on the same data.

                          I am using SilkPerformer because it gives me monitoring data.

                          top - 17:16:55 up 1 day, 17:18, 1 user, load average: 1.11, 0.81, 0.42
                          Tasks: 409 total, 2 running, 407 sleeping, 0 stopped, 0 zombie
                          Cpu(s): 6.3%us, 0.1%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
                          Mem: 33554432k total, 2036188k used, 31518244k free, 139740k buffers
                          Swap: 69435384k total, 0k used, 69435384k free, 562816k cached

                          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                          26755 root 25 0 93416 7912 6092 R 100.0 0.0 5:06.71 ARTTCPH
                          26759 root 15 0 235m 22m 13m S 0.3 0.1 0:01.62 ARTSTRN
                          26773 root 15 0 235m 22m 13m S 0.3 0.1 0:01.46 ARTSTRN
                          26779 root 15 0 235m 22m 13m S 0.3 0.1 0:01.40 ARTSTRN
                          26791 root 15 0 235m 22m 13m S 0.3 0.1 0:01.76 ARTSTRN
                          26797 root 15 0 235m 22m 13m S 0.3 0.1 0:01.64 ARTSTRN
                          26800 root 15 0 235m 22m 13m S 0.3 0.1 0:01.50 ARTSTRN
                          26816 root 15 0 235m 22m 13m S 0.3 0.1 0:01.37 ARTSTRN


                          Also, I dont know why the system cpu is showig 93% idle. I feel like this is incorrect.

                          Edited by: user522010 on May 22, 2012 2:20 PM
                          Hi,

                          I see that your ARTTCPH has run more than 5 hours, while ARTSTRN servers only run about one minute.This is a big difference. I guess that might be the cause. The ARTTCPH may be some "dead" processes in your last application(s). Actually, there is a situation, when there exists some clients which haven't completed their transactions correctly, and you have "tmshutdown" the servers, then the "ARTTCPH" can't be totally shutdown. You can see the ARTTCPH process by "ps" command. In this way, we need to kill these processes manually before the next boot up. Could you please kill the ARTTCPH processes first, and run your application again? Let's see if it can work normally in this way.

                          Thanks,
                          Sophia
                          I think those stale ARTTCPH might be hanging up my runs. I have had situations where only some of the clients run, the others are sort of hung, and there is som TMACTIVE transactions. Howevever, when I close my clients, the transactions go away, but maybe they are still stuck in ARTTCPH. This is why I thnk we should have more than one ARTTCPH server.

                          Do you think this is a bug?
                          Hi,

                          Thanks for your feedback. It seems that the TMACTIVE transactions are probably caused by the hanged clients which are related to the stale ARTTCPH. As you said, I think there is a bug in ARTTCPH. When tmshutdown, at least we should print a warning messages if ARTTCPH can't be shutdown successfully. We'll look into this bug as soon as possible. Hopefully it can be fixed in later releases or patches. For now, I think you can kill the stale ARTTCPH manually as a workaround. Any further questions, please feel free to post it out. Thanks!

                          Regards,
                          Sophia
                          • 10. Re: ART Application SB00 takes too long to respond for more than 1 user
                            user12432153
                            Hi,

                            I’m from Tuxedo ART dev team, and I’ll follow up the issue you found. In fact, we have done a lot of concurrent tests as you did before, but all ARL servers could work normally. So, before we start to check your problems, could you help us to collect some traces and detailed information as below:

                            1. Please describe the details of the problem you found, and what are the abnormal behaviors ?

                            2. Please provide us your reproducer if it’s possible, seems you set up your application runtime based on sample 03, you could compress the directory which includes the CICS programs, ubbconfig, envfile, resources and etc, and then send to us via mail "sean.zhang@oracle.com"

                            3. Please describe your reproduce steps (it’s very important how you start the clients concurrently in your side), it’s better to provide your scripts (include shell script and x3270 script) used.

                            4. If you could not send us the reproducer you used, here is some way to catch the traces of ART runtime as below, could you help to collect them and send to us via mail ?
                            4.1 Enable the traces of ARTTCPL and ARTTCPH. Change the CLOPT of ARTTCPL in ubb file as below, add “-D +H 0”.
                            ARTTCPL SRVGRP=TCP00
                            SRVID=101
                            CLOPT=" -- -M 4 -m 1 -L //10.182.55.250:38159 -n //10.182.55.250:38158 -D +H 0"
                            The traces of ARTTCPL will be located at /tmp with the name “tcxtcpl.dbg-time-pid”, and the traces of ARTTCPH will be located at /tmp with the name “tcxtcph.dbg-time-pid”,
                            each ARTTCPH process will have one trace file, so please send them all to us.

                            4.2     Enable the trace of BMS, “export KCPDEBUG=Y” before tmboot. The traces of BMS will be located at $APPDIR with name “kcptrc.pid”, please send them all to us.

                            4.3     Enable the trace of ART CICS servers, such as ARTSTRN, ARTATRN. Before tmboot, please add the following lines in your envfile.
                            KIX_TRACE_LEVEL=9
                            KIX_DEBUG_LEVEL=9
                            TP_USER_TRACE=SID
                            The traces of all ART CICS servers will be located at $APPDIR/LOGS/sysout and $APPDIR/LOGS/traces.

                            4.4     Enable the TM trace of Tuxedo, export TMTRACE=”*:ulog:dye” before tmboot. All tuxedo traces will be located in $APPDIR/LOGS/log/ULOG.xxxx.

                            I understand it will take some time to prepare the items above, but these items are very important for us to locate the root cause.

                            Regards,
                            Sean
                            Tuxedo Dev Team