This discussion is archived
8 Replies Latest reply: Feb 13, 2013 5:12 PM by sb92075 RSS

Listener Crash without error and logs

user10388158 Newbie
Currently Being Moderated
Hi,

i need a little help..

for some reason, my prodcution listerner crashes.. but when looking into the logs of my listener, no error is shown.
even if i check the time stamp, there is no error of the log.

here is the cut of the logs. if you notice, the listener was started twice.. one at 1am and the other at 12PM..
but there is no log in bet the 1am and 12pm that it was stopped or an error.

TIA..



09-FEB-2013 21:21:01 * service_update * CRIBS * 0
09-FEB-2013 21:21:02 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=34139)) * establish * CRIBS * 0
09-FEB-2013 21:31:01 * service_update * CRIBS * 0
09-FEB-2013 21:34:04 * (CONNECT_DATA=(SID=CRIBS)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.122.18)(PORT=41583)) * establish * CRIBS *
0
09-FEB-2013 21:38:28 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=58234)) * establish * CRIBS * 0
09-FEB-2013 21:38:28 * service_update * CRIBS * 0
09-FEB-2013 21:42:46 * (CONNECT_DATA=(SID=CRIBS)(CID=(PROGRAM=emagent)(HOST=cribsibmdb01)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.32.151)(PORT=34528)) * est
ablish * CRIBS * 12518
TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
IBM/AIX RISC System/6000 Error: 32: Broken pipe
09-FEB-2013 21:42:52 * service_died * CRIBS * 12537
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cribsibmdb01)(PORT=1521)))
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
Listener completed notification to CRS on stop
09-FEB-2013 21:43:07 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=stop)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * stop * 0

TNSLSNR for IBM/AIX RISC System/6000: Version 10.2.0.4.0 - Production on 10-FEB-2013 01:31:33

Copyright (c) 1991, 2007, Oracle. All rights reserved.

System parameter file is /oracle/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
Log messages written to /oracle/app/oracle/product/10.2.0/db_1/network/log/listener.log
Trace information written to /oracle/app/oracle/product/10.2.0/db_1/network/trace/listener.trc
Trace level is currently 0

Started with pid=9699554
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cribsibmdb01)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
Listener completed notification to CRS on start

TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
WARNING: Subscription for node down event still pending
10-FEB-2013 01:31:33 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
WARNING: Subscription for node down event still pending
10-FEB-2013 01:31:59 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0

.
.
.
.
.
.
.

10-FEB-2013 11:30:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=cribs))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
10-FEB-2013 11:30:09 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
2.151)(PORT=32978)) * establish * CRIBS * 0
10-FEB-2013 11:30:09 * service_update * CRIBS * 0
10-FEB-2013 11:30:11 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=38577)) * establish * CRIBS * 0
10-FEB-2013 11:30:15 * service_update * CRIBS * 0
10-FEB-2013 11:31:06 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=35548)) * establish * CRIBS * 0
10-FEB-2013 11:31:06 * service_update * CRIBS * 0
10-FEB-2013 11:31:09 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
2.151)(PORT=33941)) * establish * CRIBS * 0
10-FEB-2013 11:31:09 * service_update * CRIBS * 0
10-FEB-2013 11:31:11 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=63985)) * establish * CRIBS * 0
10-FEB-2013 11:32:06 * service_update * CRIBS * 0
10-FEB-2013 11:32:07 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=56060)) * establish * CRIBS * 0
10-FEB-2013 11:32:09 * service_update * CRIBS * 0
10-FEB-2013 11:32:10 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
2.151)(PORT=52442)) * establish * CRIBS * 0
10-FEB-2013 11:32:10 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=38479)) * establish * CRIBS * 0
10-FEB-2013 11:32:12 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=45212)) * establish * CRIBS * 0
10-FEB-2013 11:32:12 * service_update * CRIBS * 0
10-FEB-2013 11:32:14 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
2.151)(PORT=55982)) * establish * CRIBS * 0
10-FEB-2013 11:32:15 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=55149)) * establish * CRIBS * 0
10-FEB-2013 11:32:15 * service_update * CRIBS * 0

TNSLSNR for IBM/AIX RISC System/6000: Version 10.2.0.4.0 - Production on 10-FEB-2013 12:10:20

Copyright (c) 1991, 2007, Oracle. All rights reserved.

System parameter file is /oracle/app/oracle/product/10.2.0/db_1/network/admin/listener.ora
Log messages written to /oracle/app/oracle/product/10.2.0/db_1/network/log/listener.log
Trace information written to /oracle/app/oracle/product/10.2.0/db_1/network/trace/listener.trc
Trace level is currently 0

Started with pid=56885624
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cribsibmdb01)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
Listener completed notification to CRS on start

TIMESTAMP * CONNECT DATA [* PROTOCOL INFO] * EVENT [* SID] * RETURN CODE
WARNING: Subscription for node down event still pending
10-FEB-2013 12:10:21 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
WARNING: Subscription for node down event still pending
10-FEB-2013 12:10:28 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=oracle))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
10-FEB-2013 12:11:19 * service_register * CRIBS * 0
WARNING: Subscription for node down event still pending
10-FEB-2013 12:15:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=cribs))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
10-FEB-2013 12:15:09 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlldr)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.3
2.151)(PORT=48086)) * establish * CRIBS * 0
10-FEB-2013 12:15:10 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=63500)) * establish * CRIBS * 0
10-FEB-2013 12:19:07 * (CONNECT_DATA=(SID=CRIBS)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.122.18)(PORT=62563)) * establish * CRIBS *
0
10-FEB-2013 12:21:19 * service_update * CRIBS * 0
WARNING: Subscription for node down event still pending
10-FEB-2013 12:30:01 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=cribsibmdb01)(USER=cribs))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=169870336)) * status * 0
10-FEB-2013 12:30:02 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=CRIBS)(CID=(PROGRAM=sqlplus)(HOST=cribsibmdb01)(USER=cribs))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.30.
32.151)(PORT=43204)) * establish * CRIBS * 0
  • 1. Re: Listener Crash without error and logs
    sb92075 Guru
    Currently Being Moderated
    user10388158 wrote:
    Hi,

    i need a little help..

    for some reason, my prodcution listerner crashes.. but when looking into the logs of my listener, no error is shown.
    even if i check the time stamp, there is no error of the log.
    forgive me, but I need to ask a simple question.

    what proof exists that outage was not an Operating System crash?
    if the system crashed, then the Oracle listener would no longer be available to log any connection requests.

    Handle:     user10388158
    Status Level:     Newbie
    Registered:     Mar 10, 2010
    Total Posts:     25
    Total Questions:     9 (8 unresolved)

    why do you waste time here when your questions RARELY get answered?
  • 2. Re: Listener Crash without error and logs
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Refer to
    TROUBLESHOOTING GUIDE TNS-12518 TNS listener could not hand off client connection [ID 550859.1]
    ORA-12518 / TNS-12518 Troubleshooting [ID 556428.1]
  • 3. Re: Listener Crash without error and logs
    EdStevens Guru
    Currently Being Moderated
    Usually the error message at the bottom of a series is most meaningful. In this case I'd start by focusing on
    IBM/AIX RISC System/6000 Error: 32: Broken pipe
    Looks like the listener was just a victim of something at the OS level.
  • 4. Re: Listener Crash without error and logs
    user10388158 Newbie
    Currently Being Moderated
    Thanks for the reply..

    i have checked also the syslogs..
    looking at this /var/adm/ras/syslog.caa.0 and errpt -a|more

    but there seems to be no significant errors occured on the syslogs. is there any other logs i should look at?

    Thank you.
  • 5. Re: Listener Crash without error and logs
    user10388158 Newbie
    Currently Being Moderated
    could this be the culprit ORA-12518 ?
    looking at the logs, my issue happend in between the two startups of the listener..
    the fisrt stop was because we stopped the listener for an offline backup.

    the next was the first startup of the listener..
    next was the unknown crash...
    then the startup after the crash.
  • 6. Re: Listener Crash without error and logs
    EdStevens Guru
    Currently Being Moderated
    user10388158 wrote:
    could this be the culprit ORA-12518 ?
    looking at the logs, my issue happend in between the two startups of the listener..
    the fisrt stop was because we stopped the listener for an offline backup.
    I can't say if there is a connection or not, but I can say there is no reason to stop the listener for an offline backup. What did you think was being accomplished by stopping the listener in that scenario?

    the next was the first startup of the listener..
    next was the unknown crash...
    then the startup after the crash.
  • 7. Re: Listener Crash without error and logs
    user10388158 Newbie
    Currently Being Moderated
    thanks. actually it is included in our script to stop the listener. just to make sure that nothing runs during the offline backup.

    but in betwen the two startups, can you suggest on something there i can get an idea how it died?
  • 8. Re: Listener Crash without error and logs
    sb92075 Guru
    Currently Being Moderated
    user10388158 wrote:
    thanks. actually it is included in our script to stop the listener. just to make sure that nothing runs during the offline backup.
    HUH?
    How in the world can any client access the DB after it has been SHUTDOWN?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points