4 Replies Latest reply on Nov 24, 2011 12:50 AM by 858779

    App Server and Process Scheduler stuck while booting

    868539
      Hi All,

      We are facing a big issue in our production instance.The application server showed down while accessing the PIA.When we tried to reboot the same it didnt start and was getting stuck after some steps..few snipets are below after the last PSSRENSRV process does not proceed further.

      exec PSSAMSRV  --A  --  -C  psappsrv.cfg  -D FNPROD  -S PSSAMSRV:
              CMDTUX_CAT:819: INFO:Process id=5584 Assume started <pipe>.
      exec PSSAMSRV  --A  --  -C  psappsrv.cfg  -D FNPROD  -S PSSAMSRV:
              CMDTUX_CAT:819: INFO:Process id=5600 Assume started <pipe>.
      exec PSSAMSRV  --A  --  -C  psappsrv.cfg  -D FNPROD  -S PSSAMSRV:
              CMDTUX_CAT:819: INFO:Process id=5616 Assume started <pipe>.
      exec PSSAMSRV  --A  --  -C  psappsrv.cfg  -D FNPROD  -S PSSAMSRV:
              CMDTUX_CAT:819: INFO:Process id=5632 Assume started <pipe>.
      exec PSSAMSRV  --A  --  -C  psappsrv.cfg  -D FNPROD  -S PSSAMSRV:
              CMDTUX_CAT:819: INFO:Process id=5648 Assume started <pipe>.
      exec PSSAMSRV  --A  --  -C  psappsrv.cfg  -D FNPROD  -S PSSAMSRV:
              CMDTUX_CAT:819: INFO:Process id=5664 Assume started <pipe>.
      exec PSRENSRV  --A  --   -C  psappsrv.cfg  -D FNPROD  -S PSRENSRV:
      The tuxedo,appserv and renserv logs shows,

      renserv log
      ----- Begin booting PSRENSRV -------------------------
      0.5608 [14/Jul/2011 07:51:54](ERROR) CRenTuxSvcMgr::ReqBootParameters - Tux service call failed.
      ----- Begin booting PSRENSRV -------------------------
      0.5056 [14/Jul/2011 07:58:47](ERROR) CRenTuxSvcMgr::ReqBootParameters - Tux service call failed.
      ----- Begin booting PSRENSRV -------------------------
      0.780 [14/Jul/2011 08:21:04](ERROR) CRenTuxSvcMgr::ReqBootParameters - Tux service call failed.
      ------------------------------------------------------------------------------------------
      tuxedo log
      receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV
      082104.UKLO2K3S55!JSH.5060.5084.-2: JOLT_CAT:1107: "ERROR: Error receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV"
      082104.UKLO2K3S55!WSH.4364.1752.0: WSNAT_CAT:1107: ERROR: Error receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV
      082104.UKLO2K3S55!WSH.4472.4416.0: WSNAT_CAT:1107: ERROR: Error receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV
      082104.UKLO2K3S55!WSH.1684.4128.0: WSNAT_CAT:1107: ERROR: Error receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV
      082104.UKLO2K3S55!WSH.4256.4172.0: WSNAT_CAT:1107: ERROR: Error receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV
      082104.UKLO2K3S55!JSH.4876.496.-2: JOLT_CAT:1107: "ERROR: Error receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV"
      082104.UKLO2K3S55!WSH.4896.4284.0: WSNAT_CAT:1107: ERROR: Error receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV
      082104.UKLO2K3S55!JSH.4800.1036.-2: JOLT_CAT:1107: "ERROR: Error receiving messages, tperrno = TPEOS, Uunixerr = UMSGRCV"
      082104.UKLO2K3S55!WSL.3904.4032.0: LIBTUX_CAT:703: ERROR: Cannot get statistics on message queue 43
      082104.UKLO2K3S55!JSL.4316.4024.0: JOLT_CAT:1196: "INFO: Terminating handlers in preparation for shutdown"
      082104.UKLO2K3S55!WSL.3904.4032.0: WSNAT_CAT:1197: INFO: Exiting system
      082104.UKLO2K3S55!JREPSVR.5768.5800.0: LIBTUX_CAT:271: ERROR: System lock semop failure, key = 61480 (errno = 22)
      082104.UKLO2K3S55!WSL.3904.4032.0: LIBTUX_CAT:268: ERROR: Failed to stop serving
      082104.UKLO2K3S55!JSL.4316.4024.0: JOLT_CAT:1197: "INFO: Exiting system"
      082104.UKLO2K3S55!JREPSVR.5768.5800.0: LIBTUX_CAT:268: ERROR: Failed to stop serving
      082104.UKLO2K3S55!JSL.4316.4024.0: LIBTUX_CAT:268: ERROR: Failed to stop serving
      082105.UKLO2K3S55!PSWATCHSRV.6124.568.-2: ERROR: msgrcv err(LIBTUX_CAT:669: ERROR: Message operation failed because of the invalid message queue identifier): errno=22,qid=2,buf=9580216,bytes=4572,type=-1073741824,flag=2048
      082105.UKLO2K3S55!PSWATCHSRV.6124.568.-2: LIBTUX_CAT:268: ERROR: Failed to stop serving
      ------------------------------------------------------------------------------------------
      app serv log
      PSADMIN.4944 (0) [07/14/11 07:48:28](0) Begin boot attempt on domain FNPROD
      PSWATCHSRV.4720 (0) [07/14/11 07:48:33] Starting
      PSADMIN.4944 (0) [07/14/11 07:52:29](0) End boot attempt on domain FNPROD
      PSADMIN.4944 (0) [07/14/11 07:55:46](0) Begin boot attempt on domain FNPROD
      PSWATCHSRV.1412 (0) [07/14/11 07:55:50] Starting
      PSADMIN.4892 (0) [07/14/11 08:08:07](0) Begin boot attempt on domain FNPROD
      PSWATCHSRV.5032 (0) [07/14/11 08:08:13] Starting
      PSADMIN.4892 (0) [07/14/11 08:08:27](0) End boot attempt on domain FNPROD
      PSADMIN.6104 (0) [07/14/11 08:20:28](0) Begin boot attempt on domain FNPROD
      PSWATCHSRV.6124 (0) [07/14/11 08:20:32] Starting
      -----------------------------------------------------------------------------------------
      tuxedo log
      213815.UKLO2K3S55!PSSAMSRV.6072.6040.0: LIBTUX_CAT:262: INFO: Standard main starting
      213815.UKLO2K3S55!PSRENSRV.2440.2444.-2: 07-13-2011: Tuxedo Version 8.1 32-bit Windows.
      213815.UKLO2K3S55!PSRENSRV.2440.2444.-2: LIBTUX_CAT:262: INFO: Standard main starting
      214434.UKLO2K3S55!BBL.5964.4464.0: ERROR: msgrcv err(LIBTUX_CAT:666: ERROR: Message operation failed because the queue was removed): errno=100,qid=257,buf=10285048,bytes=4572,type=-1073741824,flag=0
      214434.UKLO2K3S55!PSRENSRV.2440.2444.-2: ERROR: msgrcv err(LIBTUX_CAT:666: ERROR: Message operation failed because the queue was removed): errno=100,qid=277,buf=11551960,bytes=8668,type=805306369,flag=0
      214434.UKLO2K3S55!BBL.5964.4464.0: ERROR: msgrcv err(LIBTUX_CAT:669: ERROR: Message operation failed because of the invalid message queue identifier): errno=22,qid=257,buf=10285048,bytes=4572,type=0,flag=0
      214434.UKLO2K3S55!BBL.5964.4464.0: LIBTUX_CAT:271: ERROR: System lock semop failure, key = 33242 (errno = 22)
      214434.UKLO2K3S55!BBL.5964.4464.0: LIBTUX_CAT:268: ERROR: Failed to stop serving
      214434.UKLO2K3S55!BBL.5964.4464.0: CMDTUX_CAT:26: INFO: The BBL is exiting system
      214434.UKLO2K3S55!PSWATCHSRV.5208.4528.-2: ERROR: msgrcv err(LIBTUX_CAT:669: ERROR: Message operation failed because of the invalid message queue identifier): errno=22,qid=258,buf=9654368,bytes=27996,type=-1073741824,flag=2048
      214434.UKLO2K3S55!PSWATCHSRV.5208.4528.-2: LIBTUX_CAT:268: ERROR: Failed to stop serving
      214507.UKLO2K3S55!tmboot.4616.5152.-2: 07-13-2011: Tuxedo Version 8.1 32-bit Windows.
      214507.UKLO2K3S55!tmboot.4616.5152.-2: TUXNT_CAT:4: ERROR: Cannot access ipc server (The system cannot find the file specified.
      )
      214507.UKLO2K3S55!tmboot.4616.5152.-2: tmboot: CMDTUX_CAT:827: ERROR: Fatal error encountered; initiating user error handler
      214508.UKLO2K3S55!PSADMIN.868: End boot attempt on domain FNPROD
      Also the process scheduler was not getting booted the logs are shown below,

      scheduler log
      085646.UKLO2K3S55!restartsrv.2484.4116.-2: server BASE/101: CMDTUX_CAT:587: INFO: Cannot restart server, scheduling for cleanup
      085653.UKLO2K3S55!BBL.4044.5996.0: LIBTUX_CAT:550: WARN: Cleaning up restartable server BASE/101
      085653.UKLO2K3S55!BBL.4044.5996.0: LIBTUX_CAT:216: WARN: Process 4300 died; removing from BB
      085653.UKLO2K3S55!BBL.4044.5996.0: LIBTUX_CAT:216: WARN: Process 5372 died; removing from BB
      085653.UKLO2K3S55!BBL.4044.5996.0: LIBTUX_CAT:216: WARN: Process 5640 died; removing from BB
      085653.UKLO2K3S55!BBL.4044.5996.0: LIBTUX_CAT:216: WARN: Process 5768 died; removing from BB
      085653.UKLO2K3S55!cleanupsrv.4148.4364.-2: 07-14-2011: Tuxedo Version 8.1 32-bit Windows.
      085653.UKLO2K3S55!cleanupsrv.4148.4364.-2: GP_CAT:1077: ERROR: Unable to open environment file E:\Psoft\PRDAPP\appserv\prcs\FNPROD\psprcsrv.env for reading
      085653.UKLO2K3S55!cleanupsrv.4148.4364.-2: LIBTUX_CAT:353: ERROR: Error putting ENVFILE E:\Psoft\PRDAPP\appserv\prcs\FNPROD\psprcsrv.env into environment
      085653.UKLO2K3S55!cleanupsrv.4148.4364.-2: server BASE/101: CMDTUX_CAT:551: INFO: server removed
      085743.UKLO2K3S55!BBL.4044.5996.0: ERROR: msgrcv err(LIBTUX_CAT:666: ERROR: Message operation failed because the queue was removed): errno=100,qid=80,buf=10304328,bytes=4572,type=-1073741824,flag=0
      085743.UKLO2K3S55!BBL.4044.5996.0: ERROR: msgrcv err(LIBTUX_CAT:669: ERROR: Message operation failed because of the invalid message queue identifier): errno=22,qid=80,buf=10304328,bytes=4572,type=-1073741824,flag=0
      085743.UKLO2K3S55!BBL.4044.5996.0: LIBTUX_CAT:271: ERROR: System lock semop failure, key = 64479 (errno = 22)
      085743.UKLO2K3S55!BBL.4044.5996.0: LIBTUX_CAT:268: ERROR: Failed to stop serving
      085743.UKLO2K3S55!BBL.4044.5996.0: CMDTUX_CAT:26: INFO: The BBL is exiting system
      Is this a problem with Tuxedo if so what has to be done...Pls help its a bit urgent...


      Thanks in advance...