6 Replies Latest reply on May 9, 2012 9:08 AM by John-MK

    LIstener damaged

    John-MK
      Hi,

      Two days ago, I got an alarm that Listener is damaged/stopped (ORA-12541: TNS:no listener) in Production and Test RAC. DB=10.2.0.4, OS=RHL

      I found out by crs_stat that the following is shown OFFLINE. Though I started it by: crsctl start resources ora.db02.LISTENER.lsnr, but I wanted to know the reason(s) behind it. Could you please guide me when or why this ORA-12541 occurs? Also, this error message regarding Oracle Port that Connection is refused? I did not find any message in the Listener log file. It will be very helpful, If I could find the reason(s) behind it with your kind help.
      [oracle@db02 ~]$ crs_stat
      
      - - 
      NAME=ora.db02.LISTENER.lsnr
      TYPE=application
      TARGET=ONLINE
      STATE=ONLINE on db02
      I will appreciate your suggestion(s) regarding the issue.

      Thanks alot.

      Regards

      Edited by: 910385 on May 8, 2012 4:15 AM
        • 1. Re: LIstener damaged
          kuljeet singh -
          review log file located under crs_home/log.
          drwxr-x---    2 oracle   oinstall        256 Nov 24 17:41 admin
          drwxr-x---    2 oracle   oinstall        256 Nov 24 18:14 evmd
          drwxr-x---    2 root     oinstall        256 Nov 24 18:14 crsd
          drwxr-x---    4 oracle   oinstall        256 Nov 25 10:34 cssd
          drwxrwxr-t    5 oracle   oinstall        256 Nov 30 15:08 racg
          drwxr-x---    2 oracle   oinstall       4096 Apr 30 16:16 client
          -rw-rw-r--    1 root     oinstall       8195 Apr 30 16:16 alertRAC1.log
          • 2. Re: LIstener damaged
            John-MK
            Hi,

            Thank you.

            Following are the lines from CRS_HOME/log/alert.log. Are these lines suggest the cause?. I am very poor in RAC's
            2012-05-05 20:13:58.673
            [cssd(11928)]CRS-1612:node db02 (2) at 50% heartbeat fatal, eviction in 29.180 seconds
            2012-05-06 08:17:36.257
            [cssd(11928)]CRS-1612:node db02 (2) at 50% heartbeat fatal, eviction in 29.058 seconds
            2012-05-06 08:17:37.249
            [cssd(11928)]CRS-1612:node db02 (2) at 50% heartbeat fatal, eviction in 28.068 seconds
            2012-05-06 08:17:51.258
            [cssd(11928)]CRS-1611:node db02 (2) at 75% heartbeat fatal, eviction in 14.068 seconds
            2012-05-06 08:17:52.260
            [cssd(11928)]CRS-1611:node db02 (2) at 75% heartbeat fatal, eviction in 13.058 seconds
            Regards

            Edited by: 910385 on May 8, 2012 4:37 AM
            • 3. Re: LIstener damaged
              kuljeet singh -
              appear your node was evicted ,can you check the uptime of your node.
              • 4. Re: LIstener damaged
                John-MK
                uptime
                15:25:51 up 324 days, 1:35, 2 users, load average: 0.51, 0.34, 0.29

                I have checked uptime of all the four machines, i.e., 2 test and 2 prod but in all the cases its more than 150 days. Then it means, messages in CRS log file doesn't mean such meanings or what?

                Edited by: 910385 on May 8, 2012 6:29 AM

                Edited by: 910385 on May 8, 2012 7:46 AM
                • 5. Re: LIstener damaged
                  onedbguru
                  It appears more like a network issue... Should it have reached 100% it would have been fatal. So, at some point something network related started causing problems - could have been an overloaded nic or switch (although, best practices, the cluster interconnect should be a dedicated switch.) Look in the listener logs, the CRS logs for indications of problems.
                  1 person found this helpful
                  • 6. Re: LIstener damaged
                    John-MK
                    Hi,

                    Thank you for your input.

                    The lines from CRS log I have pasted above at the time of issue, and below lines are from listener log file. According to my weak judgement, I dont see any problem in the listener.log file, isn't it? Confusing... CRS log file suggests some NODE EVICTION, but UPTIME is more than 150 days. Listener log file to me doesnt says anything like service_down or node_down.

                    But the thing to note here is that in both CRS log file and LISTENER log file, entries are starting from the time when the ORA error come...NO LISTENER....in both the log files previous entries are TWO years old..
                    05-MAY-2012 20:26:01 * service_register * +ASM1 * 0
                    05-MAY-2012 20:26:06 * service_register * db1 * 0
                    05-MAY-2012 20:26:06 * service_update * db1 * 0
                    05-MAY-2012 20:26:07 * service_update * +ASM1 * 0
                    05-MAY-2012 20:26:41 * 12502
                    
                    TNS-12502: TNS:listener received no CONNECT_DATA from client
                    Thank you.

                    Regards.