11 Replies Latest reply: May 21, 2012 11:54 PM by 932792 RSS

    scan-vip running only on one RAC node

    932792
      Hi ,

      While setting up RAC11.2 on Centos 5.7 , I was getting this error during the grid installation:

      PRCR-1079 : Failed to start resource ora.scan1.vip
      CRS-5005: IP Address: 192.168.100.208 is already in use in the network
      CRS-2674: Start of 'ora.scan1.vip' on 'falcen6b' failed
      CRS-2632: There are no more servers to try to place resource 'ora.scan1.vip' on that would satisfy its placement policy
      PRCR-1079 : Failed to start resource ora.scan2.vip
      CRS-5005: IP Address: 192.168.100.209 is already in use in the network
      CRS-2674: Start of 'ora.scan2.vip' on 'falcen6b' failed
      CRS-2632: There are no more servers to try to place resource 'ora.scan2.vip' on that would satisfy its placement policy
      PRCR-1079 : Failed to start resource ora.scan3.vip
      CRS-5005: IP Address: 192.168.100.210 is already in use in the network
      CRS-2674: Start of 'ora.scan3.vip' on 'falcen6b' failed
      CRS-2632: There are no more servers to try to place resource 'ora.scan3.vip' on that would satisfy its placement policy


      I figured that the scan service is able to run only on one node at a time. When I stopped the service on rac1 and started it on rac2 the service is starting.

      But I think for the grid installation the scan service has to simultaneously run on both the nodes.

      How do I resolve it?

      Any suggestions please.

      PS - I am planning to try with the patch 11.0.2.3 but it will be a while till i get access to it.

      Till then can someone suggest a workaround?
        • 1. Re: scan-vip running only on one RAC node
          Balazs Papp
          I figured that the scan service is able to run only on one node at a time. When I stopped the service on rac1 and started it on rac2 the service is starting.
          thats right, one scan vip can run on one node at a time
          But I think for the grid installation the scan service has to simultaneously run on both the nodes.
          no it doesnt have to
          Any suggestions please.
          post the output of the following commands:

          $GRID_HOME/bin/crsctl stat res -t

          from both nodes:

          ifconfig
          cat /etc/hosts (if you dont use DNS for pub/vip/priv addresses)
          nslookup [your_scan_address]
          • 2. Re: scan-vip running only on one RAC node
            932792
            Hi Balazs Papp,

            Thanks for your mail.

            Please find the output:

            RAC1:
            $GRID_HOME/bin/crsctl stat res -t

            Output :
            [oracle@falcen6a bin]$ ./crsctl stat res -t
            --------------------------------------------------------------------------------
            NAME TARGET STATE SERVER STATE_DETAILS
            --------------------------------------------------------------------------------
            Local Resources
            --------------------------------------------------------------------------------
            ora.DATA.dg
            ONLINE ONLINE falcen6a
            ora.LISTENER.lsnr
            ONLINE ONLINE falcen6a
            ora.asm
            ONLINE ONLINE falcen6a Started
            ora.eons
            ONLINE ONLINE falcen6a
            ora.gsd
            OFFLINE OFFLINE falcen6a
            ora.net1.network
            ONLINE ONLINE falcen6a
            ora.ons
            ONLINE ONLINE falcen6a
            --------------------------------------------------------------------------------
            Cluster Resources
            --------------------------------------------------------------------------------
            ora.LISTENER_SCAN1.lsnr
            1 OFFLINE OFFLINE
            ora.LISTENER_SCAN2.lsnr
            1 OFFLINE OFFLINE
            ora.LISTENER_SCAN3.lsnr
            1 OFFLINE OFFLINE
            ora.falcen6a.vip
            1 ONLINE ONLINE falcen6a
            ora.oc4j
            1 OFFLINE OFFLINE
            ora.scan1.vip
            1 ONLINE OFFLINE
            ora.scan2.vip
            1 ONLINE OFFLINE
            ora.scan3.vip
            1 ONLINE OFFLINE
            [oracle@falcen6a bin]$

            [oracle@falcen6a bin]$ nslookup falcen6-scan
            Server: 192.168.100.21
            Address: 192.168.100.21#53

            Name: falcen6-scan.a.pri
            Address: 192.168.100.210
            Name: falcen6-scan.a.pri
            Address: 192.168.100.208
            Name: falcen6-scan.a.pri
            Address: 192.168.100.209

            IFCONFIG:
            [oracle@falcen6a bin]$ ifconfig
            eth0 Link encap:Ethernet HWaddr 00:0C:29:22:41:83
            inet addr:192.168.100.218 Bcast:192.168.100.255 Mask:255.255.255.0
            inet6 addr: fe80::20c:29ff:fe22:4183/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            RX packets:41395 errors:0 dropped:0 overruns:0 frame:0
            TX packets:28200 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:36489898 (34.7 MiB) TX bytes:4947781 (4.7 MiB)
            Interrupt:67 Base address:0x2024

            eth0:4 Link encap:Ethernet HWaddr 00:0C:29:22:41:83
            inet addr:192.168.100.212 Bcast:192.168.100.255 Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            Interrupt:67 Base address:0x2024

            eth1 Link encap:Ethernet HWaddr 00:0C:29:22:41:8D
            inet addr:192.168.210.101 Bcast:192.168.210.255 Mask:255.255.255.0
            inet6 addr: fe80::20c:29ff:fe22:418d/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            RX packets:5410 errors:0 dropped:0 overruns:0 frame:0
            TX packets:2425 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:639655 (624.6 KiB) TX bytes:289470 (282.6 KiB)
            Interrupt:67 Base address:0x20a4

            lo Link encap:Local Loopback
            inet addr:127.0.0.1 Mask:255.0.0.0
            inet6 addr: ::1/128 Scope:Host
            UP LOOPBACK RUNNING MTU:16436 Metric:1
            RX packets:15879 errors:0 dropped:0 overruns:0 frame:0
            TX packets:15879 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:7493976 (7.1 MiB) TX bytes:7493976 (7.1 MiB)


            RAC2:
            [oracle@falcen6b bin]$ ./crsctl stat res -t
            --------------------------------------------------------------------------------
            NAME TARGET STATE SERVER STATE_DETAILS
            --------------------------------------------------------------------------------
            Local Resources
            --------------------------------------------------------------------------------
            ora.DATA.dg
            ONLINE ONLINE falcen6b
            ora.asm
            ONLINE ONLINE falcen6b Started
            ora.eons
            ONLINE ONLINE falcen6b
            ora.gsd
            OFFLINE OFFLINE falcen6b
            ora.net1.network
            ONLINE ONLINE falcen6b
            ora.ons
            ONLINE ONLINE falcen6b
            --------------------------------------------------------------------------------
            Cluster Resources
            --------------------------------------------------------------------------------
            ora.LISTENER_SCAN1.lsnr
            1 ONLINE ONLINE falcen6b
            ora.LISTENER_SCAN2.lsnr
            1 ONLINE ONLINE falcen6b
            ora.LISTENER_SCAN3.lsnr
            1 ONLINE ONLINE falcen6b
            ora.falcen6b.vip
            1 ONLINE ONLINE falcen6b
            ora.oc4j
            1 OFFLINE OFFLINE
            ora.scan1.vip
            1 ONLINE ONLINE falcen6b
            ora.scan2.vip
            1 ONLINE ONLINE falcen6b
            ora.scan3.vip
            1 ONLINE ONLINE falcen6b


            NSLOOKUP:
            [oracle@falcen6b bin]$ nslookup falcen6-scan
            Server: 192.168.100.252
            Address: 192.168.100.252#53

            Name: falcen6-scan.a.pri
            Address: 192.168.100.210
            Name: falcen6-scan.a.pri
            Address: 192.168.100.208
            Name: falcen6-scan.a.pri
            Address: 192.168.100.209

            IFCONFIG:
            [oracle@falcen6b bin]$ ifconfig
            eth0 Link encap:Ethernet HWaddr 00:0C:29:A4:5E:96
            inet addr:192.168.100.219 Bcast:192.168.100.255 Mask:255.255.255.0
            inet6 addr: fe80::20c:29ff:fea4:5e96/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            RX packets:2232694 errors:0 dropped:0 overruns:0 frame:0
            TX packets:321133 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:3274457044 (3.0 GiB) TX bytes:23178566 (22.1 MiB)
            Interrupt:67 Base address:0x2024

            eth0:1 Link encap:Ethernet HWaddr 00:0C:29:A4:5E:96
            inet addr:192.168.100.213 Bcast:192.168.100.255 Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            Interrupt:67 Base address:0x2024

            eth0:2 Link encap:Ethernet HWaddr 00:0C:29:A4:5E:96
            inet addr:192.168.100.208 Bcast:192.168.100.255 Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            Interrupt:67 Base address:0x2024

            eth0:3 Link encap:Ethernet HWaddr 00:0C:29:A4:5E:96
            inet addr:192.168.100.209 Bcast:192.168.100.255 Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            Interrupt:67 Base address:0x2024

            eth0:4 Link encap:Ethernet HWaddr 00:0C:29:A4:5E:96
            inet addr:192.168.100.210 Bcast:192.168.100.255 Mask:255.255.255.0
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            Interrupt:67 Base address:0x2024

            eth1 Link encap:Ethernet HWaddr 00:0C:29:A4:5E:A0
            inet addr:192.168.210.102 Bcast:192.168.210.255 Mask:255.255.255.0
            inet6 addr: fe80::20c:29ff:fea4:5ea0/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            RX packets:26490 errors:0 dropped:0 overruns:0 frame:0
            TX packets:8663 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:3160363 (3.0 MiB) TX bytes:1020401 (996.4 KiB)
            Interrupt:67 Base address:0x20a4

            lo Link encap:Local Loopback
            inet addr:127.0.0.1 Mask:255.0.0.0
            inet6 addr: ::1/128 Scope:Host
            UP LOOPBACK RUNNING MTU:16436 Metric:1
            RX packets:84137 errors:0 dropped:0 overruns:0 frame:0
            TX packets:84137 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:42990747 (40.9 MiB) TX bytes:42990747 (40.9 MiB)
            • 3. Re: scan-vip running only on one RAC node
              932792
              This was the error I got when executing the root.sh on rac2:

              PRCR-1079 : Failed to start resource ora.scan1.vip
              CRS-5005: IP Address: 192.168.100.208 is already in use in the network
              CRS-2674: Start of 'ora.scan1.vip' on 'falcen6b' failed
              CRS-2632: There are no more servers to try to place resource 'ora.scan1.vip' on that would satisfy its placement policy
              PRCR-1079 : Failed to start resource ora.scan2.vip
              CRS-5005: IP Address: 192.168.100.209 is already in use in the network
              CRS-2674: Start of 'ora.scan2.vip' on 'falcen6b' failed
              CRS-2632: There are no more servers to try to place resource 'ora.scan2.vip' on that would satisfy its placement policy
              PRCR-1079 : Failed to start resource ora.scan3.vip
              CRS-5005: IP Address: 192.168.100.210 is already in use in the network
              CRS-2674: Start of 'ora.scan3.vip' on 'falcen6b' failed
              CRS-2632: There are no more servers to try to place resource 'ora.scan3.vip' on that would satisfy its placement policy
              • 4. Re: scan-vip running only on one RAC node
                Balazs Papp
                something went wrong during configuration
                the scan resources are all online on falcen6b according to your output, and all the virtual interfaces (eth0:2-3-4) are up and running also on falcen6b
                you have 2 nodes but they doesnt seem to be members of the same cluster

                when running root.sh on the first node, did you wait until it finished, before starting root.sh on second node?

                what is the output of this command (lists the names of cluster nodes):
                $GRID_HOME/bin/olsnodes

                can you upload somewhere the logs of your root.sh sessions from the 2 nodes? $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<hostname>.log
                • 5. Re: scan-vip running only on one RAC node
                  932792
                  Hi,

                  Thanks for your reply.

                  The first output which I had displayed was when I had recently manually shutdown the service on rac1 and started the service on rac2.

                  When I was installing using the grid installer, it was the other way around. Just wanted to clarify.
                  when running root.sh on the first node, did you wait until it finished, before starting root.sh on second node?
                  I waited till the root.sh was completed in the first node and then started with the second node.

                  I will display the requested output tomorrow.
                  • 6. Re: scan-vip running only on one RAC node
                    onedbguru
                    Please read and understand the SCAN documentation - what it is and how it works. Are you using DNS or GNS/DHCP? However you configured this and based on your output below, you are creating a scenario where you are using the same ip-address more than once.

                    post the output of the following:

                    cat /etc/hosts |grep <scan-name>

                    nslookup <scan-name>

                    srvctl config scan
                    srvctl config scan_listener

                    Go through this document from start to finish and FOLLOW THE INSTRUCTIONS - you will more than likely determine how it was mis-configured.
                    http://www.oracle.com/technetwork/products/clustering/overview/scan-129069.pdf


                    The advice I give to ALL my DBA trainee's and system/net/san administrators working on RAC:

                    RAC really is NOT that hard. Read the documentation and Follow the instructions. Don't try to think you know more than the instructions, Don't try to take short-cuts. Oh, did I mention READ AND FOLLOW THE DOCUMENTATION AND INSTRUCTIONS - ALL OF THEM!!!
                    • 7. Re: scan-vip running only on one RAC node
                      932792
                      Hi Balazs Papp,

                      Please find the output:

                      RAC1:

                      [oracle@falcen6a bin]$ ./olsnodes
                      falcen6a

                      RAC2:
                      [root@falcen6b bin]# ./olsnodes
                      falcen6b

                      Im not sure where to upload , however i will post (not the fulll) output of root.sh on rac2:

                      2012-05-20 19:43:36: Start of resource "ora.mdnsd -init" Succeeded
                      2012-05-20 19:43:38: Start of resource "ora.gipcd -init" Succeeded
                      2012-05-20 19:43:39: Start of resource "ora.gpnpd -init" Succeeded
                      2012-05-20 19:43:39: Successfully started requested Oracle stack daemons
                      2012-05-20 19:43:39: Starting CSS in clustered mode
                      2012-05-20 19:44:35: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'falcen6b'
                      2012-05-20 19:44:35: CRS-2676: Start of 'ora.cssdmonitor' on 'falcen6b' succeeded
                      2012-05-20 19:44:35: CRS-2672: Attempting to start 'ora.cssd' on 'falcen6b'
                      2012-05-20 19:44:35: CRS-2672: Attempting to start 'ora.diskmon' on 'falcen6b'
                      2012-05-20 19:44:35: CRS-2676: Start of 'ora.diskmon' on 'falcen6b' succeeded
                      2012-05-20 19:44:35: CRS-2676: Start of 'ora.cssd' on 'falcen6b' succeeded
                      2012-05-20 19:44:36: Start of resource "ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE" Succeeded
                      2012-05-20 19:44:49: Start of resource "ora.asm -init" Succeeded
                      2012-05-20 19:44:50: Start of resource "ora.crsd -init" Succeeded
                      2012-05-20 19:44:52: Start of resource "ora.evmd -init" Succeeded
                      2012-05-20 19:44:52: Successfully started Oracle clusterware stack
                      2012-05-20 19:44:52: Waiting for Oracle CRSD and EVMD to start
                      2012-05-20 19:44:57: Oracle CRS stack installed and running
                      2012-05-20 19:44:57: Configuring node
                      2012-05-20 19:44:57: adding nodeapps...
                      2012-05-20 19:44:57: upgrade_opt=
                      2012-05-20 19:44:57: nodevip=falcen6b-vip/255.255.255.0/eth0
                      2012-05-20 19:44:57: DHCP_flag=0
                      2012-05-20 19:44:57: nodes_to_add=falcen6b
                      2012-05-20 19:44:57: add nodeapps for static IP
                      2012-05-20 19:44:57: Running srvctl config nodeapps to detect if VIP exists
                      2012-05-20 19:44:58: output=PRKO-2312 : GSD does not exist.
                      PRKO-2331 : ONS daemon does not exist.
                      PRKO-2339 : eONS daemon does not exist.

                      2012-05-20 19:44:58: Invoking "/home/gridhome/bin/srvctl add nodeapps -n falcen6b -A "falcen6b-vip/255.255.255.0/eth0" "
                      2012-05-20 19:44:58: trace file=/home/gridhome/cfgtoollogs/crsconfig/srvmcfg0.log
                      2012-05-20 19:45:17: add nodeapps -n falcen6b -A falcen6b-vip/255.255.255.0/eth0 on node=falcen6b ... passed
                      2012-05-20 19:45:17: nodes_to_start=falcen6b
                      2012-05-20 19:45:17: Configuring first node
                      2012-05-20 19:45:17: DHCP_flag=0
                      2012-05-20 19:45:17: nodes_to_start=falcen6b
                      2012-05-20 19:45:17: check ASM exists done and ASM does not exist
                      2012-05-20 19:45:17: Prior version ASM does not exist , Invoking add asm
                      2012-05-20 19:45:17: Invoking "/home/gridhome/bin/srvctl add asm"
                      2012-05-20 19:45:17: trace file=/home/gridhome/cfgtoollogs/crsconfig/srvmcfg1.log
                      2012-05-20 19:45:17: Running as user oracle: /home/gridhome/bin/srvctl add asm
                      2012-05-20 19:45:17: Invoking "/home/gridhome/bin/srvctl add asm" as user "oracle"
                      2012-05-20 19:45:23: add asm ... success
                      2012-05-20 19:45:23: Adding ASM diskgroup resource
                      2012-05-20 19:45:23: Running as user oracle: /home/gridhome/bin/crsctl create diskgroup DATA
                      2012-05-20 19:45:23: Invoking "/home/gridhome/bin/crsctl create diskgroup DATA" as user "oracle"
                      2012-05-20 19:45:25: create diskgroup DATA ... success
                      2012-05-20 19:45:25: Successfully created disk group resource
                      2012-05-20 19:45:25: GNS is not to be configured - skipping
                      2012-05-20 19:45:25: Invoking "/home/gridhome/bin/srvctl add scan -n falcen6-scan.aqua.pri"
                      2012-05-20 19:45:25: trace file=/home/gridhome/cfgtoollogs/crsconfig/srvmcfg2.log
                      2012-05-20 19:45:28: add scan=falcen6-scan.aqua.pri ... success
                      2012-05-20 19:45:28: Invoking "/home/gridhome/bin/srvctl add scan_listener -p 1521"
                      2012-05-20 19:45:28: trace file=/home/gridhome/cfgtoollogs/crsconfig/srvmcfg3.log
                      2012-05-20 19:45:28: Running as user oracle: /home/gridhome/bin/srvctl add scan_listener -p 1521
                      2012-05-20 19:45:28: Invoking "/home/gridhome/bin/srvctl add scan_listener -p 1521" as user "oracle"
                      2012-05-20 19:45:32: add scan listener ... success
                      2012-05-20 19:45:32: Running as user oracle: /home/gridhome/bin/srvctl add oc4j
                      2012-05-20 19:45:32: Invoking "/home/gridhome/bin/srvctl add oc4j" as user "oracle"
                      2012-05-20 19:45:34: J2EE (OC4J) Container Resource Add ... passed ...
                      2012-05-20 19:45:34: starting nodeapps...
                      2012-05-20 19:45:34: DHCP_flag=0
                      2012-05-20 19:45:34: nodes_to_start=falcen6b
                      2012-05-20 19:45:45: exit value of start nodeapps/vip is 0
                      2012-05-20 19:45:45: GNS is not to be configured - skipping
                      2012-05-20 19:45:45: Invoking "/home/gridhome/bin/srvctl start scan"
                      2012-05-20 19:45:45: trace file=/home/gridhome/cfgtoollogs/crsconfig/srvmcfg5.log
                      2012-05-20 19:46:46: "/home/gridhome/bin/srvctl start scan" failed with status 512.
                      2012-05-20 19:46:46: start scan ... failed
                      2012-05-20 19:46:46: Install cvuqdisk rpm on Linux...
                      2012-05-20 19:46:46: New package to install is /home/gridhome/cv/rpm/cvuqdisk-1.0.7-1.rpm
                      2012-05-20 19:46:46: Invoking "/bin/rpm -q cvuqdisk" command
                      2012-05-20 19:46:46: no existing cvuqdisk found
                      2012-05-20 19:46:46: removing old rpm
                      2012-05-20 19:46:46: installing/upgrading new rpm
                      2012-05-20 19:46:47: Configure Oracle Grid Infrastructure for a Cluster ... failed
                      • 8. Re: scan-vip running only on one RAC node
                        932792
                        Hi onedbguru,

                        Thanks for your mail.

                        I am using DNS and I read the document which you shared.

                        Please find the requested output:

                        RAC1:
                        [oracle@falcen6a bin]$ cat /etc/hosts | grep falcen6-scan
                        #192.168.100.208 falcen6-scan.a.pri falcen6-scan
                        #192.168.100.209 falcen6-scan.a.pri falcen6-scan
                        #192.168.100.210 falcen6-scan.a.pri falcen6-scan

                        [oracle@falcen6a bin]$ nslookup falcen6-scan
                        Server: 192.168.100.21
                        Address: 192.168.100.21#53

                        Name: falcen6-scan.a.pri
                        Address: 192.168.100.210
                        Name: falcen6-scan.a.pri
                        Address: 192.168.100.208
                        Name: falcen6-scan.a.pri
                        Address: 192.168.100.209

                        [oracle@falcen6a bin]$ ./srvctl config scan
                        SCAN name: falcen6-scan.a.pri, Network: 1/192.168.100.0/255.255.255.0/eth0
                        SCAN VIP name: scan1, IP: /falcen6-scan.a.pri/192.168.100.208
                        SCAN VIP name: scan2, IP: /falcen6-scan.a.pri/192.168.100.209
                        SCAN VIP name: scan3, IP: /falcen6-scan.a.pri/192.168.100.210

                        [oracle@falcen6a bin]$ ./srvctl config scan_listener
                        SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
                        SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
                        SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

                        RAC2:
                        [oracle@falcen6b gridhome]$ cat /etc/hosts| grep falcen6-scan
                        #192.168.100.208 falcen6-scan.a.pri falcen6-scan
                        #192.168.100.209 falcen6-scan.a.pri falcen6-scan
                        #192.168.100.210 falcen6-scan.a.pri falcen6-scan

                        [oracle@falcen6b gridhome]$ nslookup falcen6-scan
                        Server: 192.168.100.252
                        Address: 192.168.100.252#53

                        Name: falcen6-scan.a.pri
                        Address: 192.168.100.210
                        Name: falcen6-scan.a.pri
                        Address: 192.168.100.208
                        Name: falcen6-scan.a.pri
                        Address: 192.168.100.209

                        [oracle@falcen6b bin]$ ./srvctl config scan
                        SCAN name: falcen6-scan.a.pri, Network: 1/192.168.100.0/255.255.255.0/eth0
                        SCAN VIP name: scan1, IP: /falcen6-scan.a.pri/192.168.100.208
                        SCAN VIP name: scan2, IP: /falcen6-scan.a.pri/192.168.100.209
                        SCAN VIP name: scan3, IP: /falcen6-scan.a.pri/192.168.100.210

                        [oracle@falcen6b bin]$ ./srvctl config scan_listener
                        SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
                        SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
                        SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
                        • 9. Re: scan-vip running only on one RAC node
                          932792
                          Hi Balazs Papp and onedbguru,

                          I was able to resolve that error by running the following command on rac2, now that part of the installer passed.

                          crsctl start res ora.scan1.vip

                          However the cluster verification utility is failing at the end of installer.

                          When I executed the below command, this is my output:

                          [oracle@falcen6a grid]$ ./runcluvfy.sh stage -post crsinst -n falcen6a,falcen6b -verbose

                          Performing post-checks for cluster services setup

                          Checking node reachability...

                          Check: Node reachability from node "falcen6a"
                          Destination Node Reachable?
                          ------------------------------------ ------------------------
                          falcen6a yes
                          falcen6b yes
                          Result: Node reachability check passed from node "falcen6a"


                          Checking user equivalence...

                          Check: User equivalence for user "oracle"
                          Node Name Comment
                          ------------------------------------ ------------------------
                          falcen6b passed
                          falcen6a passed
                          Result: User equivalence check passed for user "oracle"
                          Checking time zone consistency...
                          Time zone consistency check passed.


                          Checking Cluster manager integrity...

                          Checking CSS daemon...

                          Node Name Status
                          ------------------------------------ ------------------------
                          falcen6b running
                          falcen6a running

                          Oracle Cluster Synchronization Services appear to be online.

                          Cluster manager integrity check passed


                          UDev attributes check for OCR locations started...
                          Result: UDev attributes check passed for OCR locations


                          UDev attributes check for Voting Disk locations started...
                          Result: UDev attributes check passed for Voting Disk locations


                          Check default user file creation mask
                          Node Name Available Required Comment
                          ------------ ------------------------ ------------------------ ----------
                          falcen6b 0022 0022 passed
                          falcen6a 0022 0022 passed
                          Result: Default user file creation mask check passed

                          Checking cluster integrity...

                          Cluster is divided into 2 partitions
                          Partition 1 consists of the following members:
                          Node Name
                          ------------------------------------
                          falcen6b

                          Partition 2 consists of the following members:
                          Node Name
                          ------------------------------------
                          falcen6a


                          Cluster integrity check failed. Cluster is divided into 2 partition(s).

                          Checking OCR integrity...

                          Checking the absence of a non-clustered configuration...
                          All nodes free of non-clustered, local-only configurations


                          ERROR:

                          PRVF-4193 : Asm is not running on the following nodes. Proceeding with the remaining nodes.


                          Checking OCR config file "/etc/oracle/ocr.loc"...

                          OCR config file "/etc/oracle/ocr.loc" check successful


                          ERROR:

                          PRVF-4195 : Disk group for ocr location "+DATA" not available on the following nodes:


                          Checking size of the OCR location "+DATA" ...

                          Size check for OCR location "+DATA" successful...

                          OCR integrity check failed

                          Checking CRS integrity...

                          ERROR:
                          PRVF-5316 : Failed to retrieve version of CRS installed on node "falcen6b"
                          The Oracle clusterware is healthy on node "falcen6b"
                          The Oracle clusterware is healthy on node "falcen6a"

                          CRS integrity check failed

                          Checking node application existence...

                          Checking existence of VIP node application
                          Node Name Required Status Comment
                          ------------ ------------------------ ------------------------ ----------
                          falcen6b yes unknown failed
                          falcen6a yes unknown failed
                          Result: Check failed.

                          Checking existence of ONS node application
                          Node Name Required Status Comment
                          ------------ ------------------------ ------------------------ ----------
                          falcen6b no unknown ignored
                          falcen6a no online passed
                          Result: Check ignored.

                          Checking existence of GSD node application
                          Node Name Required Status Comment
                          ------------ ------------------------ ------------------------ ----------
                          falcen6b no unknown ignored
                          falcen6a no does not exist ignored
                          Result: Check ignored.

                          Checking existence of EONS node application
                          Node Name Required Status Comment
                          ------------ ------------------------ ------------------------ ----------
                          falcen6b no unknown ignored
                          falcen6a no online passed
                          Result: Check ignored.

                          Checking existence of NETWORK node application
                          Node Name Required Status Comment
                          ------------ ------------------------ ------------------------ ----------
                          falcen6b no unknown ignored
                          falcen6a no online passed
                          Result: Check ignored.


                          Checking Single Client Access Name (SCAN)...
                          SCAN VIP name Node Running? ListenerName Port Running?
                          ---------------- ------------ ------------ ------------ ------------ ------------
                          falcen6-scan unknown false LISTENER 1521 false

                          WARNING:
                          PRVF-5056 : Scan Listener "LISTENER" not running

                          Checking name resolution setup for "falcen6-scan"...
                          SCAN Name IP Address Status Comment
                          ------------ ------------------------ ------------------------ ----------
                          falcen6-scan 192.168.100.210 passed
                          falcen6-scan 192.168.100.208 passed
                          falcen6-scan 192.168.100.209 passed

                          Verification of SCAN VIP and Listener setup failed
                          OCR detected on ASM. Running ACFS Integrity checks...

                          Starting check to see if ASM is running on all cluster nodes...
                          PRVF-5137 : Failure while checking ASM status on node "falcen6b"

                          Starting Disk Groups check to see if at least one Disk Group configured...
                          Disk Group Check passed. At least one Disk Group configured

                          Task ACFS Integrity check failed

                          Checking Oracle Cluster Voting Disk configuration...

                          Oracle Cluster Voting Disk configuration check passed

                          Checking to make sure user "oracle" is not in "root" group
                          Node Name Status Comment
                          ------------ ------------------------ ------------------------
                          falcen6b does not exist passed
                          falcen6a does not exist passed
                          Result: User "oracle" is not part of "root" group. Check passed

                          Checking if Clusterware is installed on all nodes...
                          Check of Clusterware install passed

                          Checking if CTSS Resource is running on all nodes...
                          Check: CTSS Resource running on all nodes
                          Node Name Status
                          ------------------------------------ ------------------------
                          falcen6b passed
                          falcen6a passed
                          Result: CTSS resource check passed


                          Querying CTSS for time offset on all nodes...
                          Result: Query of CTSS for time offset passed

                          Check CTSS state started...
                          Check: CTSS state
                          Node Name State
                          ------------------------------------ ------------------------
                          falcen6b Observer
                          falcen6a Observer
                          CTSS is in Observer state. Switching over to clock synchronization checks using NTP


                          Starting Clock synchronization checks using Network Time Protocol(NTP)...

                          NTP Configuration file check started...
                          The NTP configuration file "/etc/ntp.conf" is available on all nodes
                          NTP Configuration file check passed

                          Checking daemon liveness...

                          Check: Liveness for "ntpd"
                          Node Name Running?
                          ------------------------------------ ------------------------
                          falcen6b yes
                          falcen6a yes
                          Result: Liveness check passed for "ntpd"

                          Checking NTP daemon command line for slewing option "-x"
                          Check: NTP daemon command line
                          Node Name Slewing Option Set?
                          ------------------------------------ ------------------------
                          falcen6b yes
                          falcen6a yes
                          Result:
                          NTP daemon slewing option check passed

                          Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"

                          Check: NTP daemon's boot time configuration
                          Node Name Slewing Option Set?
                          ------------------------------------ ------------------------
                          falcen6b yes
                          falcen6a yes
                          Result:
                          NTP daemon's boot time configuration check for slewing option passed

                          NTP common Time Server Check started...
                          NTP Time Server "133.243.236.19" is common to all nodes on which the NTP daemon is running
                          NTP Time Server "133.243.236.18" is common to all nodes on which the NTP daemon is running
                          NTP Time Server "210.173.160.86" is common to all nodes on which the NTP daemon is running
                          NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
                          Check of common NTP Time Server passed

                          Clock time offset check from NTP Time Server started...
                          Checking on nodes "[falcen6b, falcen6a]"...
                          Check: Clock time offset from NTP Time Server

                          Time Server: 133.243.236.19
                          Time Offset Limit: 1000.0 msecs
                          Node Name Time Offset Status
                          ------------ ------------------------ ------------------------
                          falcen6b 15.332 passed
                          falcen6a -1.503 passed
                          Time Server "133.243.236.19" has time offsets that are within permissible limits for nodes "[falcen6b, falcen6a]".

                          Time Server: 133.243.236.18
                          Time Offset Limit: 1000.0 msecs
                          Node Name Time Offset Status
                          ------------ ------------------------ ------------------------
                          falcen6b 15.115 passed
                          falcen6a -1.614 passed
                          Time Server "133.243.236.18" has time offsets that are within permissible limits for nodes "[falcen6b, falcen6a]".

                          Time Server: 210.173.160.86
                          Time Offset Limit: 1000.0 msecs
                          Node Name Time Offset Status
                          ------------ ------------------------ ------------------------
                          falcen6b 15.219 passed
                          falcen6a -1.527 passed
                          Time Server "210.173.160.86" has time offsets that are within permissible limits for nodes "[falcen6b, falcen6a]".

                          Time Server: .LOCL.
                          Time Offset Limit: 1000.0 msecs
                          Node Name Time Offset Status
                          ------------ ------------------------ ------------------------
                          falcen6b 0.0 passed
                          falcen6a 0.0 passed
                          Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[falcen6b, falcen6a]".
                          Clock time offset check passed

                          Result: Clock synchronization check using Network Time Protocol(NTP) passed


                          Oracle Cluster Time Synchronization Services check passed

                          Post-check for cluster services setup was unsuccessful on all the nodes.
                          [oracle@falcen6a grid]$

                          Any suggestions?
                          • 10. Re: scan-vip running only on one RAC node
                            932792
                            This is my output for ocr check from both the nodes:
                            [oracle@falcen6a bin]$ ./ocrcheck
                            Status of Oracle Cluster Registry is as follows :
                            Version : 3
                            Total space (kbytes) : 262120
                            Used space (kbytes) : 2204
                            Available space (kbytes) : 259916
                            ID : 960988220
                            Device/File Name : +DATA
                            Device/File integrity check succeeded

                            Device/File not configured

                            Device/File not configured

                            Device/File not configured

                            Device/File not configured

                            Cluster registry integrity check succeeded

                            Logical corruption check bypassed due to non-privileged user

                            RAC2:
                            [oracle@falcen6b bin]$ ./ocrcheck
                            Status of Oracle Cluster Registry is as follows :
                            Version : 3
                            Total space (kbytes) : 262120
                            Used space (kbytes) : 2056
                            Available space (kbytes) : 260064
                            ID : 1720050235
                            Device/File Name : +DATA
                            Device/File integrity check succeeded

                            Device/File not configured

                            Device/File not configured

                            Device/File not configured

                            Device/File not configured

                            Cluster registry integrity check succeeded

                            Logical corruption check bypassed due to non-privileged user
                            • 11. Re: scan-vip running only on one RAC node
                              932792
                              Hi Moderator,

                              Please delete my above post as It is not related to the subject and I will post it in a new thread.

                              The solution related to the thread subject has been provided.