9 Replies Latest reply: Jan 6, 2011 4:07 AM by user633661 RSS

    Why "lsnrctl status" works on only one node?

    user633661
      Hi,

      I have a 2 node 11.1.0.7 RAC cluster running on Oracle Enterprise Linux 5. My listener.ora files are as follows:

      Node RAC1:
      LISTENER_RAC1 =
        (DESCRIPTION_LIST =
          (DESCRIPTION =
            (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521)(IP = FIRST))
            (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.1)(PORT = 1521)(IP = FIRST))
          )
        )
      Node RAC2:
      LISTENER_RAC2 =
        (DESCRIPTION_LIST =
          (DESCRIPTION =
            (ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521)(IP = FIRST))
            (ADDRESS = (PROTOCOL = TCP)(HOST = 10.65.1.118)(PORT = 1521)(IP = FIRST))
          )
        )
      The listener was created by means of netca on the rac1 node and the default 'LISTENER' name was used. The two listeners are up on both nodes:

      Node RAC1:
      [oracle@rac1 admin]$ ps -ef | grep tns | grep -v grep
      oracle   21460     1  0 01:01 ?        00:00:00 /u01/app/oracle/product/11.1.0/db_1/bin/tnslsnr LISTENER_RAC1 -inherit
      Node RAC2:
      [oracle@rac2 admin]$ ps -ef | grep tns | grep -v grep
      oracle   21153     1  0 05:16 ?        00:00:00 /u01/app/oracle/product/11.1.0/db_1/bin/tnslsnr LISTENER_RAC2 -inherit
      What bothers me is why the "lsnrctl status" command on the rac1 node returns the status of the listener but the same command on the rac2 node doesn't?


      Node RAC1:
      [oracle@rac1 admin]$ lsnrctl status
      
      LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 04-JAN-2011 01:05:33
      
      Copyright (c) 1991, 2008, Oracle.  All rights reserved.
      
      Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
      STATUS of the LISTENER
      ------------------------
      Alias                     LISTENER_RAC1
      Version                   TNSLSNR for Linux: Version 11.1.0.7.0 - Production
      Start Date                04-JAN-2011 01:01:42
      Uptime                    0 days 0 hr. 3 min. 50 sec
      Trace Level               off
      Security                  ON: Local OS Authentication
      SNMP                      OFF
      Listener Parameter File   /u01/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
      Listener Log File         /u01/app/oracle/diag/tnslsnr/rac1/listener_rac1/alert/log.xml
      Listening Endpoints Summary...
        (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.65.2.162)(PORT=1521)))
        (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
      Services Summary...
      Service "+ASM" has 1 instance(s).
        Instance "+ASM1", status READY, has 1 handler(s) for this service...
      Service "+ASM_XPT" has 1 instance(s).
        Instance "+ASM1", status READY, has 1 handler(s) for this service...
      The command completed successfully
      [oracle@rac1 admin]$
      Node RAC2:
      [oracle@rac2 admin]$ lsnrctl status
      
      LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 04-JAN-2011 05:20:23
      
      Copyright (c) 1991, 2008, Oracle.  All rights reserved.
      
      Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
      TNS-12541: TNS:no listener
       TNS-12560: TNS:protocol adapter error
        TNS-00511: No listener
         Linux Error: 111: Connection refused
      [oracle@rac2 admin]$
      However, if i add the listener name, i get its status:
      [oracle@rac2 admin]$ lsnrctl status LISTENER_RAC2
      
      LSNRCTL for Linux: Version 11.1.0.7.0 - Production on 04-JAN-2011 05:20:47
      
      Copyright (c) 1991, 2008, Oracle.  All rights reserved.
      
      Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac2-vip)(PORT=1521)(IP=FIRST)))
      STATUS of the LISTENER
      ------------------------
      Alias                     LISTENER_RAC2
      Version                   TNSLSNR for Linux: Version 11.1.0.7.0 - Production
      Start Date                04-JAN-2011 05:16:32
      Uptime                    0 days 0 hr. 4 min. 15 sec
      Trace Level               off
      Security                  ON: Local OS Authentication
      SNMP                      OFF
      Listener Parameter File   /u01/app/oracle/product/11.1.0/db_1/network/admin/listener.ora
      Listener Log File         /u01/app/oracle/diag/tnslsnr/rac2/listener_rac2/alert/log.xml
      Listening Endpoints Summary...
        (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.65.2.163)(PORT=1521)))
        (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.65.1.118)(PORT=1521)))
      The listener supports no services
      The command completed successfully
      [oracle@rac2 admin]$
      Actually I'm new to RAC administration, but on single instance systems, if the listener has a non-default name, it's mandatory to include it in the status command (e.g. "lsnrctl status LISTENER_TEST") otherwise an "TNS-12541: TNS:no listener" error is returned. So on this RAC system I actually don't understand why "lsnrctl status" on rac1 even works? It seems to me that the behavior on rac2 is correct....


      The reason I'm asking this question is because I believe it has some other implications. For example, my newly created ASM instance on the node rac2 doesn't automatically register with the listener, although on node rac1 it does. I checked the init parameters local_listener and remote_listener and they are unset on both asm instances:

      Node RAC1 and RAC2:
      SQL> show parameter listener
      
      NAME                                 TYPE        VALUE
      ------------------------------------ ----------- ------------------------------
      local_listener                       string
      remote_listener                      string
      Maybe if I configured the local_listener init parameter on rac2 to resolve to the listener address it would work, but I'd like to first understand why this happens.


      So if anyone has an explanation or a pointer in the documentation, I'll be very grateful.

      Thanks in advance and regards,
      Jure
        • 1. Re: Why "lsnrctl status" works on only one node?
          Levi Pereira
          Hi,

          Post here...
          /etc/hosts (both nodes)
          Output of this command
          $ hostname (both nodes)
          Regards,
          Levi Pereira
          http://levipereira.wordpress.com
          • 2. Re: Why "lsnrctl status" works on only one node?
            Ehtiram Hasanov
            actually when you use lsnrctl status command, it does not necessary your default listener's alias should be LISTENER as soon as your listener runs on default port(i.e. 1521). Saying with other words, if you rename your listeners alias - LISTENER to for example LISTENER_1 in your listener.ora file, and after it run lsnrctl status command, the status should be displayed normally. It means as soon as your listener is running on default port, then naming alias on listener.ora file does not matter and you are not required to specify it on lsnrctl status command, even it renamed from default -LISTENER alias to something other in your listener.ora file.
            • 3. Re: Why "lsnrctl status" works on only one node?
              user633661
              Hi,

              Node RAC1:
              [oracle@rac1 ~]$ hostname
              rac1.localdomain
              
              [oracle@rac1 ~]$ cat /etc/hosts
              # Do not remove the following line, or various programs
              # that require network functionality will fail.
              127.0.0.1               rac1.localdomain rac1 localhost.localdomain localhost
              ::1             localhost6.localdomain6 localhost6
              
              # Public
              10.65.1.117   rac1.localdomain        rac1
              10.65.1.118   rac2.localdomain        rac2
              #Private
              192.168.0.101   rac1-priv.localdomain   rac1-priv
              192.168.0.102   rac2-priv.localdomain   rac2-priv
              #Virtual
              10.65.2.162   rac1-vip.localdomain    rac1-vip
              10.65.2.163   rac2-vip.localdomain    rac2-vip
              Node RAC2:
              [oracle@rac1 ~]$ ssh oracle@rac2
              Last login: Wed Jan  5 08:32:22 2011 from rac1.localdomain
              
              [oracle@rac2 ~]$ hostname
              rac2.localdomain
              
              [oracle@rac2 ~]$ cat /etc/hosts
              # Do not remove the following line, or various programs
              # that require network functionality will fail.
              127.0.0.1       rac2.localdomain rac2 localhost.localdomain localhost
              ::1             localhost6.localdomain6 localhost6
              
              # Public
              10.65.1.117   rac1.localdomain        rac1
              10.65.1.118   rac2.localdomain        rac2
              #Private
              192.168.0.101   rac1-priv.localdomain   rac1-priv
              192.168.0.102   rac2-priv.localdomain   rac2-priv
              #Virtual
              10.65.2.162   rac1-vip.localdomain    rac1-vip
              10.65.2.163   rac2-vip.localdomain    rac2-vip
              [oracle@rac2 ~]$
              • 4. Re: Why "lsnrctl status" works on only one node?
                user633661
                actually when you use lsnrctl status command, it does not necessary your default listener's alias should be LISTENER as soon as your listener runs on default port(i.e. 1521).
                Thanks for the explanation. I was under the wrong conviction that non-default listener meant non-default 1521 port OR if its name wasn't 'LISTENER'. Thanks again for clarifying it out.
                • 5. Re: Why "lsnrctl status" works on only one node?
                  Levi Pereira
                  Hi,

                  127.0.0.1 rac1.localdomain rac1 localhost.localdomain localhost
                  127.0.0.1 rac2.localdomain rac2 localhost.localdomain localhost
                  10.65.1.117 rac1.localdomain rac1
                  10.65.1.118 rac2.localdomain rac2
                  Your entries in the hosts file is incorrect. The hosts file does not resolve the same hostname in two IP different. Only the first entry is resolved.

                  Please remove rac1.localdomain rac1 and rac2.localdomain rac2 of the line that starts with 127.0.0.1 from file hosts in both nodes.

                  Try lnsrctl status again on node 2


                  Correct the configuration of the node1, because after the changes in the host file you may have trouble connecting on node1:

                  Node RAC1:
                  LISTENER_RAC1 =
                    (DESCRIPTION_LIST =
                      (DESCRIPTION =
                        (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521)(IP = FIRST))
                        (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.1)(PORT = 1521)(IP = FIRST))  # remove this 
                        (ADDRESS = (PROTOCOL = TCP)(HOST = 10.65.1.117)(PORT = 1521)(IP = FIRST))  # put this 
                      )
                    )
                  Stop and start listener on node 1.


                  If works post here...


                  Regards,
                  Levi Pereira

                  Edited by: Levi Pereira on Jan 5, 2011 11:18 AM
                  • 6. Re: Why "lsnrctl status" works on only one node?
                    user633661
                    Thanks, that solved the problem. So the rule is to never resolve the hostname to the 127.0.0.1 loopback address, or at least not resolve the same hostname to two different different IPs (127.0.0.1 and 10.65.1.117 in my case)?

                    Thanks again

                    Edited by: user633661 on Jan 5, 2011 8:18 AM

                    Edited by: user633661 on Jan 5, 2011 8:19 AM
                    • 7. Re: Why "lsnrctl status" works on only one node?
                      Levi Pereira
                      Hi,
                      So the rule is to never resolve the hostname to the 127.0.0.1 loopback address, or at least not resolve the same hostname to two different different IPs (127.0.0.1 and 10.65.1.117 in my case)?
                      Yes.

                      If during the installation of linux you do not configure the IP manually, Linux will put its hostname in loopback, this is necessary because several services that use the Linux hostname depend on the network to start.

                      But during the installation of Oracle this should be corrected, added the public IP on hostname and removing hostname from loopback.

                      Hosts file always resolves only the first entry associated with the hostname. For a hostname can resolve more than one IP, you must use the feature Round Robim available on DNS servers.

                      Understanding your problem

                      The Oracle use gethostbyname() library function may query the DNS server(s), the /etc/hosts file, the NIS service and perhaps other methods, based on the system configuration.
                      As your host in RAC2 was returning the IP 127.0.0.1, Oracle Net could not find the listener configured on the IP 127.0.0.1. So he did not find the Listener default.

                      Regards,
                      Levi Pereira

                      Edited by: Levi Pereira on Jan 5, 2011 2:39 PM
                      • 8. Re: Why "lsnrctl status" works on only one node?
                        807722
                        Hello

                        My problem was listener was hanging while startup or status in newly created oracle home,after trying some options
                        i had reinstalled oracle ,but one strange thing when i name oracle home as say orahome1 it gives same problem and if i give anyother name to oralce home say orahome2 this issue is not there listener start and work as normal

                        please share your views

                        thanks
                        • 9. Re: Why "lsnrctl status" works on only one node?
                          user633661
                          Thank you very much for the clear explanation.

                          Since this is not a production system I just wanted to see if it would work without changing the /etc/hosts file by removing the "(IP=FIRST)" part in the listener.ora file so that the listener would not be bound to only the specified IP. It didin't work. And also if it would work, I see that it wouldn't be a good idea: Re: RAC Listener Configuration what is IP=FIRST Means (TCP timeouts in case of node failure when the VIP migrates to another node). Also, since I don't quite understand all the nuances of IP=FIRST (I think there's a difference in behavior if a hostname or IP is specified in the listener.ora file) I'll first have to read about that subject and then if something won't be clear I might ask.

                          So for now thanks again for all the answers.

                          Regards,
                          Jure