8 Replies Latest reply: May 15, 2013 10:23 AM by 1008097 RSS

    RAC11gR2 - unable to connect via SCAN - can connect directly on nodes

    1008097
      Hello,

      I recently installed a two node RAC 11gR2 database on Oracle Linux. I'm trying to connect from an external windows client, using SQL Developer, but I keep getting the error "IO Error: The Network adapter could not establish the connection".

      If I attempt to connect directly to one of the nodes (instead of putting the scan hostname I just use the node's hostname) it works without a problem. I'm thinking this might have something to do with a bad configuration on the scan listeners or the VIP.

      After installing clusterware and the database I never manually configured any listener. I assume this was done automatically. Let me know what kind of information I should provide to make this easier to troubleshoot. I'm not using any hosts file entries (except the private interconnect hostnames) since everything has been configured on my DNS. This is the network schema I had in mind when creating the cluster (and it's how my DNS is configured):

      #10.16.1.93 rac01-dev.elqrd.local rac01-dev -- public interfaces
      #10.16.1.94 rac02-dev.elqrd.local rac02-dev

      #10.16.1.95 rac01-vip.elqrd.local rac01-vip -- vip interfaces
      #10.16.1.96 rac02-vip.elqrd.local rac02-vip

      172.16.125.1 rac01-priv.elqrd.local rac01-priv -- private interconnect
      172.16.125.2 rac02-priv.elqrd.local rac02-priv

      #10.16.1.97 rac01-scan -- scan ip's
      #10.16.1.98
      #10.16.1.99

      Here's the scan status of my cluster:

      [grid@rac01-dev ~]$ srvctl config scan
      SCAN name: rac01-scan.elqrd.local, Network: 1/10.16.1.0/255.255.255.0/eth0
      SCAN VIP name: scan1, IP: /rac01-scan.elqrd.local/10.16.1.97
      SCAN VIP name: scan2, IP: /rac01-scan.elqrd.local/10.16.1.98
      SCAN VIP name: scan3, IP: /rac01-scan.elqrd.local/10.16.1.99


      And here's the vip configuration:
      [grid@rac01-dev ~]$ srvctl config vip -n rac01-dev
      VIP exists.:rac01-dev
      VIP exists.: /rac01-vip/10.16.1.95/255.255.255.0/eth0
      [grid@rac01-dev ~]$ srvctl config vip -n rac02-dev
      VIP exists.:rac02-dev
      VIP exists.: /rac02-vip/10.16.1.96/255.255.255.0/eth0
      [grid@rac01-dev ~]$


      Here is the listener status:
      grid@rac01-dev ~]$ lsnrctl service

      LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 09-MAY-2013 17:01:01

      Copyright (c) 1991, 2009, Oracle. All rights reserved.

      Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
      Services Summary...
      Service "+ASM" has 1 instance(s).
      Instance "+ASM1", status READY, has 1 handler(s) for this service...
      Handler(s):
      "DEDICATED" established:70 refused:0 state:ready
      LOCAL SERVER
      Service "ORCL" has 1 instance(s).
      Instance "ORCL1", status READY, has 1 handler(s) for this service...
      Handler(s):
      "DEDICATED" established:83 refused:0 state:ready
      LOCAL SERVER
      Service "ORCLXDB" has 1 instance(s).
      Instance "ORCL1", status READY, has 1 handler(s) for this service...
      Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
      DISPATCHER <machine: rac01-dev.elqrd.local, pid: 9331>
      (ADDRESS=(PROTOCOL=tcp)(HOST=rac01-dev.elqrd.local)(PORT=64963))
      The command completed successfully

      It's the same on the second node, with the difference that it refers to the other instances (+ASM2, ORCL2)



      A similar thread suggested I post this information also:
      select inst_id, name, value from gv$parameter where name like '%_listener%'

      1     local_listener     (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac01-vip)(PORT=1521))))
      1     remote_listener     rac01-scan.elqrd.local:1521
      2     local_listener     (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac02-vip)(PORT=1521))))
      2     remote_listener     rac01-scan.elqrd.local:1521

      Edited by: 1005094 on May 9, 2013 2:01 PM
      Added listener status information

      Edited by: 1005094 on May 9, 2013 2:06 PM

      Edited by: 1005094 on May 9, 2013 2:17 PM
        • 1. Re: RAC11gR2 - unable to connect via SCAN - can connect directly on nodes
          damorgan
          The scan listener is installed, configured, and managed as part of clusterware.

          Can you ping it from the client?

          Wen you do ps -ef can you see one running?
          • 2. Re: RAC11gR2 - unable to connect via SCAN - can connect directly on nodes
            Levi Pereira
            Hi,
            Try use this checklist:
            http://levipereira.wordpress.com/2011/05/03/configuring-client-to-use-scan-11-2-0/
            • 3. Re: RAC11gR2 - unable to connect via SCAN - can connect directly on nodes
              1008097
              [grid@rac01-dev ~]$ ps -ef | grep SCAN
              grid 8317 1 0 May09 ? 00:00:03 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
              grid 8326 1 0 May09 ? 00:00:02 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit

              [oracle@rac02-dev ~]$ ps -ef | grep SCAN
              grid 5828 1 0 May09 ? 00:00:02 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit

              So it appears the scan listeners are working on both nodes.


              From the client I can telnet and ping both nodes and their vip as well as the scan dnsname & ip.

              Now the odd thing is this:
              When I do lsnrctl service listener_scan1 (| 2 | 3) I notice that the 3 scan listeners are running on either node 01 or node 02. This is intended. However take a look at the output:

              [grid@rac01-dev ~]$ lsnrctl service listener_scan3 (it's more or less the same for the other 2 scan listeners)

              LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 10-MAY-2013 09:13:43

              Copyright (c) 1991, 2009, Oracle. All rights reserved.

              Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
              Services Summary...
              Service "ORCL" has 2 instance(s).
              Instance "ORCL1", status READY, has 1 handler(s) for this service...
              Handler(s):
              "DEDICATED" established:0 refused:0 state:ready
              REMOTE SERVER
              (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac01-vip)(PORT=1521)))
              Instance "ORCL2", status READY, has 1 handler(s) for this service...
              Handler(s):
              "DEDICATED" established:2 refused:0 state:ready
              REMOTE SERVER
              (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=rac02-vip)(PORT=1521)))
              Service "ORCLXDB" has 2 instance(s).
              Instance "ORCL1", status READY, has 1 handler(s) for this service...
              Handler(s):
              "D000" established:0 refused:0 current:0 max:1022 state:ready
              DISPATCHER <machine: rac01-dev.elqrd.local, pid: 9331>
              (ADDRESS=(PROTOCOL=tcp)(HOST=rac01-dev.elqrd.local)(PORT=64963))
              Instance "ORCL2", status READY, has 1 handler(s) for this service...
              Handler(s):
              "D000" established:0 refused:0 current:0 max:1022 state:ready
              DISPATCHER <machine: rac02-dev.elqrd.local, pid: 6174>
              (ADDRESS=(PROTOCOL=tcp)(HOST=rac02-dev.elqrd.local)(PORT=63648))
              The command completed successfully


              What exactly is the ORCLXDB service? When I installed the database the provided SID was ORCL (With the instances ORCL1 & ORCL2 running on each of the nodes). I am able to connect on ORCLXDB (via SCAN) from my client but not on ORCL. I can still connect normally to the database if I specify the node hostname instead of the scan. But this is not desired as it kind of negates the point of having SCAN (and removes the load balancing functionality).

              P.S I'm trying to connect to the scan via SQL Developer and without providing any tnsnames.ora file. It fails for scan (for ORCL service - but works if it is ORCLXDB) but works for the nodes directly. The same thing happens from a second client when I try via sqlplus ("The network adapter could not establish the connection")

              Edited by: 1005094 on May 10, 2013 6:26 AM

              Edited by: 1005094 on May 10, 2013 6:31 AM
              • 4. Re: RAC11gR2 - unable to connect via SCAN - can connect directly on nodes
                1008097
                Also I noticed this:

                [grid@rac02-dev ~]$ cat /u01/app/11.2.0/grid/network/admin/endpoints_listener.ora
                LISTENER_RAC02-DEV=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac02-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=10.16.1.94)(PORT=1521)(IP=FIRST))))

                [grid@rac01-dev admin]$ cat endpoints_listener.ora
                LISTENER_RAC01-DEV=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=10.16.1.93)(PORT=1521)(IP=FIRST))))          # line added by Agent

                According to my initial design, 10.16.1.93 and 94 are the public interfaces for the two nodes. The VIPs are defined on my dns as .95 (rac01-vip) and .96 (rac02-vip):

                rac01-dev = .93
                rac01-vip = .95

                rac02-dev = .94
                rac02-vip = .96

                Why is the endpoint listener containing both the vip and the public ip? Is this as intended?
                • 5. Re: RAC11gR2 - unable to connect via SCAN - can connect directly on nodes
                  onedbguru
                  https://www.lmgtfy.com/?q=oracle+scan+pdf

                  make sure your local and remote_listener parameters are properly configured on all nodes.
                  • 6. Re: RAC11gR2 - unable to connect via SCAN - can connect directly on nodes
                    986120
                    Hi,

                    Is your scan IP resolvable from windows client.
                    i.e: ping scan hostname as well ip.
                    example: ping rac01-scan

                    Try to add *"elqrd.local"* to your windows DNS suffix

                    --
                    Bala:)
                    • 7. Re: RAC11gR2 - unable to connect via SCAN - can connect directly on nodes
                      1008097
                      I've tried this from a windows, a linux and my macbook host. I can ping and telnet the SCAN ip & hostanme without a problem from anywhere.

                      From linux it seems I can tnsping the service but not able to login via sqlplus:

                      [oracle@queue02-sbox admin]$ tnsping ORCL

                      TNS Ping Utility for Linux: Version 11.2.0.1.0 - Production on 14-MAY-2013 10:21:18

                      Copyright (c) 1997, 2009, Oracle. All rights reserved.

                      Used parameter files:
                      /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora


                      Used TNSNAMES adapter to resolve the alias
                      Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac01-scan.elqrd.local)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = ORCL)))
                      OK (0 msec)
                      [oracle@queue02-sbox admin]$ sqlplus sys@ORCL as sysdba

                      SQL*Plus: Release 11.2.0.1.0 Production on Tue May 14 10:21:30 2013

                      Copyright (c) 1982, 2009, Oracle. All rights reserved.

                      Enter password:
                      ERROR:
                      ORA-12537: TNS:connection closed
                      • 8. Re: RAC11gR2 - unable to connect via SCAN - can connect directly on nodes
                        1008097
                        Apparently this was a silly problem that was missed by me due to making attempts on various OS's. I just had to add a dns suffix on windows (and update /etc/resolv.conf on my linux client). For some reason when trying to make connections to the SCAN hostname without having a DNS search domain configured it fails. Even if I explicitly set the hostname as fully qualified. I was still able to ping the fully qualified name but sqlplus (and sql developer) failed to login. Thanks for the feedback.