8 Replies Latest reply: Feb 13, 2013 7:12 PM by sb92075 RSS

    Listener Crash without error and logs

    user10388158
      Hi,

      i need a little help..

      for some reason, my prodcution listerner crashes.. but when looking into the logs of my listener, no error is shown.
      even if i check the time stamp, there is no error of the log.

      here is the cut of the logs. if you notice, the listener was started twice.. one at 1am and the other at 12PM..
      but there is no log in bet the 1am and 12pm that it was stopped or an error.

      TIA..



      09-FEB-2013 21:21:01 * service_update * CRIBS * 0
      09-FEB-2013 21:21:02 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=34139)) * establish * CRIBS * 0
      09-FEB-2013 21:31:01 * service_update * CRIBS * 0
      09-FEB-2013 21:34:04 * (CONNECT_DATA=(SID=CRIBS)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.122.18)(PORT=41583)) * establish * CRIBS *
      0
      09-FEB-2013 21:38:28 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=58234)) * establish * CRIBS * 0
      09-FEB-2013 21:38:28 * service_update * CRIBS * 0
      09-FEB-2013 21:42:46 * (CONNECT_DATA=(SID=CRIBS)(CID=(PROGRAM=emagent)(HOST=cribsibmdb01)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.32.151)(PORT=34528)) * est
      ablish * CRIBS * 12518
      TNS-12518: TNS:listener could not hand off client connection
      TNS-12547: TNS:lost contact
      TNS-12560: TNS:protocol adapter error
      TNS-00517: Lost contact
      IBM/AIX RISC System/6000 Error: 32: Broken pipe
      09-FEB-2013 21:42:52 * service_died * CRIBS * 12537
      No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cribsibmdb01)(PORT=1521)))
      No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
      Listener completed notification to CRS on stop
      09-FEB-2013 21:43:07 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * stop * 0

      TNSLSNR for IBM/AIX RISC System/6000: Version 10.2.0.4.0 - Production on 10-FEB-2013 01:31:33

      Copyright (c) 1991, 2007, Oracle. All rights reserved.

      System parameter file is /oracle/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
      Log messages written to /oracle/app/oracle/product/10.2.0/db_1/network/log/listener.log
      Trace information written to /oracle/app/oracle/product/10.2.0/db_1/network/trace/listener.trc
      Trace level is currently 0

      Started with pid=9699554
      Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cribsibmdb01)(PORT=1521)))
      Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
      Listener completed notification to CRS on start

      TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
      WARNING: Subscription for node down event still pending
      10-FEB-2013 01:31:33 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
      WARNING: Subscription for node down event still pending
      10-FEB-2013 01:31:59 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0

      .
      .
      .
      .
      .
      .
      .

      10-FEB-2013 11:30:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=cribs))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
      10-FEB-2013 11:30:09 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
      2.151)(PORT=32978)) * establish * CRIBS * 0
      10-FEB-2013 11:30:09 * service_update * CRIBS * 0
      10-FEB-2013 11:30:11 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=38577)) * establish * CRIBS * 0
      10-FEB-2013 11:30:15 * service_update * CRIBS * 0
      10-FEB-2013 11:31:06 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=35548)) * establish * CRIBS * 0
      10-FEB-2013 11:31:06 * service_update * CRIBS * 0
      10-FEB-2013 11:31:09 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
      2.151)(PORT=33941)) * establish * CRIBS * 0
      10-FEB-2013 11:31:09 * service_update * CRIBS * 0
      10-FEB-2013 11:31:11 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=63985)) * establish * CRIBS * 0
      10-FEB-2013 11:32:06 * service_update * CRIBS * 0
      10-FEB-2013 11:32:07 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=56060)) * establish * CRIBS * 0
      10-FEB-2013 11:32:09 * service_update * CRIBS * 0
      10-FEB-2013 11:32:10 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
      2.151)(PORT=52442)) * establish * CRIBS * 0
      10-FEB-2013 11:32:10 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=38479)) * establish * CRIBS * 0
      10-FEB-2013 11:32:12 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=45212)) * establish * CRIBS * 0
      10-FEB-2013 11:32:12 * service_update * CRIBS * 0
      10-FEB-2013 11:32:14 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
      2.151)(PORT=55982)) * establish * CRIBS * 0
      10-FEB-2013 11:32:15 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=55149)) * establish * CRIBS * 0
      10-FEB-2013 11:32:15 * service_update * CRIBS * 0

      TNSLSNR for IBM/AIX RISC System/6000: Version 10.2.0.4.0 - Production on 10-FEB-2013 12:10:20

      Copyright (c) 1991, 2007, Oracle. All rights reserved.

      System parameter file is /oracle/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
      Log messages written to /oracle/app/oracle/product/10.2.0/db_1/network/log/listener.log
      Trace information written to /oracle/app/oracle/product/10.2.0/db_1/network/trace/listener.trc
      Trace level is currently 0

      Started with pid=56885624
      Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cribsibmdb01)(PORT=1521)))
      Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
      Listener completed notification to CRS on start

      TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
      WARNING: Subscription for node down event still pending
      10-FEB-2013 12:10:21 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
      WARNING: Subscription for node down event still pending
      10-FEB-2013 12:10:28 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
      10-FEB-2013 12:11:19 * service_register * CRIBS * 0
      WARNING: Subscription for node down event still pending
      10-FEB-2013 12:15:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=cribs))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
      10-FEB-2013 12:15:09 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
      2.151)(PORT=48086)) * establish * CRIBS * 0
      10-FEB-2013 12:15:10 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=63500)) * establish * CRIBS * 0
      10-FEB-2013 12:19:07 * (CONNECT_DATA=(SID=CRIBS)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.122.18)(PORT=62563)) * establish * CRIBS *
      0
      10-FEB-2013 12:21:19 * service_update * CRIBS * 0
      WARNING: Subscription for node down event still pending
      10-FEB-2013 12:30:01 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=cribs))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
      10-FEB-2013 12:30:02 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
      32.151)(PORT=43204)) * establish * CRIBS * 0
        • 1. Re: Listener Crash without error and logs
          sb92075
          user10388158 wrote:
          Hi,

          i need a little help..

          for some reason, my prodcution listerner crashes.. but when looking into the logs of my listener, no error is shown.
          even if i check the time stamp, there is no error of the log.
          forgive me, but I need to ask a simple question.

          what proof exists that outage was not an Operating System crash?
          if the system crashed, then the Oracle listener would no longer be available to log any connection requests.

          Handle:     user10388158
          Status Level:     Newbie
          Registered:     Mar 10, 2010
          Total Posts:     25
          Total Questions:     9 (8 unresolved)

          why do you waste time here when your questions RARELY get answered?
          • 2. Re: Listener Crash without error and logs
            Osama_Mustafa
            Refer to
            TROUBLESHOOTING GUIDE TNS-12518 TNS listener could not hand off client connection [ID 550859.1]
            ORA-12518 / TNS-12518 Troubleshooting [ID 556428.1]
            • 3. Re: Listener Crash without error and logs
              EdStevens
              Usually the error message at the bottom of a series is most meaningful. In this case I'd start by focusing on
              IBM/AIX RISC System/6000 Error: 32: Broken pipe
              Looks like the listener was just a victim of something at the OS level.
              • 4. Re: Listener Crash without error and logs
                user10388158
                Thanks for the reply..

                i have checked also the syslogs..
                looking at this /var/adm/ras/syslog.caa.0 and errpt -a|more

                but there seems to be no significant errors occured on the syslogs. is there any other logs i should look at?

                Thank you.
                • 5. Re: Listener Crash without error and logs
                  user10388158
                  could this be the culprit ORA-12518 ?
                  looking at the logs, my issue happend in between the two startups of the listener..
                  the fisrt stop was because we stopped the listener for an offline backup.

                  the next was the first startup of the listener..
                  next was the unknown crash...
                  then the startup after the crash.
                  • 6. Re: Listener Crash without error and logs
                    EdStevens
                    user10388158 wrote:
                    could this be the culprit ORA-12518 ?
                    looking at the logs, my issue happend in between the two startups of the listener..
                    the fisrt stop was because we stopped the listener for an offline backup.
                    I can't say if there is a connection or not, but I can say there is no reason to stop the listener for an offline backup. What did you think was being accomplished by stopping the listener in that scenario?

                    the next was the first startup of the listener..
                    next was the unknown crash...
                    then the startup after the crash.
                    • 7. Re: Listener Crash without error and logs
                      user10388158
                      thanks. actually it is included in our script to stop the listener. just to make sure that nothing runs during the offline backup.

                      but in betwen the two startups, can you suggest on something there i can get an idea how it died?
                      • 8. Re: Listener Crash without error and logs
                        sb92075
                        user10388158 wrote:
                        thanks. actually it is included in our script to stop the listener. just to make sure that nothing runs during the offline backup.
                        HUH?
                        How in the world can any client access the DB after it has been SHUTDOWN?