4 Replies Latest reply: Mar 3, 2014 11:48 AM by Shivendra Narain Nirala RSS

    CRS fail to start because of VIP changes

    user1675141

      Hi,

       

       

      Using EMC RecoverPoint, I cloned the LUNs of our Billing RAC Database. This includes all ASM Diskgroups in a consistent state at the destination using the RMAN Host Backup mode.

       

      I backed up the Operating System of Production node01, and restored it to a new server at the destination, and presented it to the cloned disks.

       

       

      So, basically the only difference is the IP Address assigned to Linux interface eth0.

       

       

      I am able to start ASM and mount all disk groups successfully, but I’m having trouble with CRS.

       

       

      Here are the steps I tried:

       

       

      -- //

       

      [root@node01]# crsctl start crs

      CRS-4640: Oracle High Availability Services is already active

      CRS-4000: Command Start failed, or completed with errors.

       

      [root@node01]# crsctl stat res -t

      CRS-4535: Cannot communicate with Cluster Ready Services

      CRS-4000: Command Status failed, or completed with errors.

       

      [root@node01]# vi /oracle/grid/11.2.0/log/node01/crsd/crsd.log

       

      ##############################################################################################

      2014-03-03 09:38:59.013: [ CRSMAIN][2856722928] Initializing OCR

      [   CLWAL][2856722928]clsw_Initialize: OLR initlevel [70000]

      2014-03-03 09:38:59.077: [ OCRAPI][2856722928]clsu_get_private_ip_addresses: no ip addresses found.

      [  OCRAPI][2856722928]a_init_clsss: failed to call clsu_get_private_ip_addr (7)

      2014-03-03 09:38:59.286: [ OCRAPI][2856722928]a_init:13!: Clusterware init unsuccessful : [44]

      2014-03-03 09:38:59.286: [  CRSOCR][2856722928] OCR context init failure.  Error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]

      2014-03-03 09:38:59.287: [ CRSMAIN][2856722928] Created alert : (:CRSD00111:) :  Could not init OCR, error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]

      2014-03-03 09:38:59.287: [ CRSD][2856722928][PANIC] CRSD exiting: Could not init OCR, code: 44

      2014-03-03 09:38:59.287: [ CRSD][2856722928] Done.

       

      2014-03-03 09:39:00.107: [ default][972362224] First attempt: init CSS context succeeded.

      [  clsdmt][1094203712]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=node01DBG_CRSD))

      2014-03-03 09:39:00.109: [  clsdmt][1094203712]PID for the Process [18319], connkey 1

      2014-03-03 09:39:00.110: [  clsdmt][1094203712]Creating PID [18319] file for home /oracle/grid/11.2.0 host node01 bin crs to /oracle/grid/11.2.0/crs/init/

      2014-03-03 09:39:00.110: [  clsdmt][1094203712]Writing PID [18319] to the file [/oracle/grid/11.2.0/crs/init/node01.pid]

      2014-03-03 09:39:01.047: [ default][1094203712] Policy Engine is not initialized yet!

      2014-03-03 09:39:01.047: [ default][972362224] CRS Daemon Starting

      ##############################################################################################

       

      -- Check OCR and Voting disks

       

      [oracle@node01]$ ocrcheck|grep -i file

               Device/File Name :       +OCR

      Device/File integrity check succeeded

      Device/File not configured

      Device/File not configured

      Device/File not configured

      Device/File not configured

       

      [oracle@node01]$ crsctl query css votedisk

      ##  STATE    File Universal Id File Name Disk group

      --  ----- ----------------- --------- ---------

      1. ONLINE   24a9d3a3fe594f43bfe92bc4619d17fc (ORCL:RACDB_CRS_DISK001) [OCR]

      Located 1 voting disk(s).

       

      [oracle@node01]$ oifcfg getif

      PRIF-10: failed to initialize the cluster registry

       

      [oracle@node01]$ oifcfg delif -global eth0

      PRIF-10: failed to initialize the cluster registry

       

      -- //

       

       

      So, because CRS cannot start successfully, I cannot change the configuration of the network interfaces using the oifcfg tool.

       

      I was hoping someone could shed some light on this?

       

       

      Regards

        • 1. Re: CRS fail to start because of VIP changes
          Ravinder Rathi-Oracle

          Use oifcfg to add the new private network information, delete the old private network information.


          Please refer the below doc:


          How to Modify Private Network Information in Oracle Clusterware (Doc ID 283684.1)

          How to Modify Public Network Information including VIP in Oracle Clusterware (Doc ID 276434.1)

          • 2. Re: CRS fail to start because of VIP changes
            Vandana B -Oracle

            Hi,

             

            If crsd is running on any of the nodes, then try the following from the node where crsd is running

             

            oifcfg setif -global <INTERFACE>/<SUBNET>:cluster_interconnect

             

            And then reboot the nodes

             

            Else refer to "Clusterware Fails to Start Due to PROC-44: Error in network address and interface operations (Doc ID 1515237.1)"

             

            Regards,

            Vandana - Oracle

            • 3. Re: CRS fail to start because of VIP changes
              BPeaslandDBA

              2014-03-03 09:38:59.077: [ OCRAPI][2856722928]clsu_get_private_ip_addresses: no ip addresses found.

              [  OCRAPI][2856722928]a_init_clsss: failed to call clsu_get_private_ip_addr (7)

              2014-03-03 09:38:59.286: [ OCRAPI][2856722928]a_init:13!: Clusterware init unsuccessful : [44]

               

              How were the private IP's defined on the original hosts? Many people define these in the /etc/hosts file. The private IP's are not VIPs. And they are not needed by any application. The private IPs are only used for communications between the hosts. As such, many people just define them in the local hosts file. My preference is to define them in DNS, but other's see it differently.

               

              Cheers,
              Brian

              • 4. Re: CRS fail to start because of VIP changes
                Shivendra Narain Nirala

                Hi

                 

                Your clusterware stack is not properly configured . First of all check the all IP address that should respond by ping command as well as from named services or DHCP .

                 

                After that run unlock your culsterware configuration then patch it .when you are done , then reboot all nodes and see the clusterware stacks come up or not and share the logs .

                 

                Regards

                 

                Shivendra Narain Nirala