11 Replies Latest reply: Sep 4, 2013 1:45 AM by Pradeepcmst RSS

    one node is up but when we start the crs on node 2 then ASM instance killed.

    dba.ravi

      Hi Friends

      i am facing below strange problem

      one node is up but when we start the crs on node 2 then ASM instance killed. i found that one ip 169.254.98.19 is binding with private interconnect IP rathter than assigned IP.

      Please help me

        • 1. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
          Pradeepcmst

          Hi Ravi,

          Cant understand your issue. Please provide OS, Oracle software details .. post the error message if you get so...

           

           

          Regards,

          Pradeep. V

          • 2. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
            dba.ravi

            Hi Pradeep ,

             

            Oracle S/W verstion is 11.2.0.3. Actually , we start the CRS and DB on node 1 but when we try to start on node 2 , getting below error.

             

            [gpnpd(18284748)]CRS-2328:GPNPD started on node ggnetp04.

            2013-09-03 19:48:29.162

            [cssd(17694820)]CRS-1713:CSSD daemon is started in clustered mode

            2013-09-03 19:48:30.557

            [ohasd(13697160)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE

            2013-09-03 19:48:41.013

            [cssd(17694820)]CRS-1707:Lease acquisition for node ggnetp04 number 2 completed

            2013-09-03 19:48:42.490

            [cssd(17694820)]CRS-1605:CSSD voting file is online: /dev/rhdiskpower0; details in /opt/app/11.2.0.3/grid/log/ggnetp04/cssd/ocssd.log.

            2013-09-03 19:48:45.483

            [cssd(17694820)]CRS-1601:CSSD Reconfiguration complete. Active nodes are ggnetp03 ggnetp04 .

            2013-09-03 19:48:47.257

            [ctssd(18022612)]CRS-2403:The Cluster Time Synchronization Service on host ggnetp04 is in observer mode.

            2013-09-03 19:48:47.632

            [ctssd(18022612)]CRS-2407:The new Cluster Time Synchronization Service reference node is host ggnetp03.

            2013-09-03 19:48:47.633

            [ctssd(18022612)]CRS-2401:The Cluster Time Synchronization Service started on host ggnetp04.

            [client(16056466)]CRS-10001:03-Sep-13 19:48 ACFS-9391: Checking for existing ADVM/ACFS installation.

            [client(16056468)]CRS-10001:03-Sep-13 19:48 ACFS-9392: Validating ADVM/ACFS installation files for operating system.

            [client(16056470)]CRS-10001:03-Sep-13 19:48 ACFS-9393: Verifying ASM Administrator setup.

            [client(16056472)]CRS-10001:03-Sep-13 19:48 ACFS-9308: Loading installed ADVM/ACFS drivers.

            [client(16056478)]CRS-10001:03-Sep-13 19:48 ACFS-9154: Loading 'oracleadvm.ext' driver.

            [client(16056486)]CRS-10001:03-Sep-13 19:48 ACFS-9154: Loading 'oracleacfs.ext' driver.

            [client(16056494)]CRS-10001:03-Sep-13 19:48 ACFS-9327: Verifying ADVM/ACFS devices.

            [client(16056498)]CRS-10001:03-Sep-13 19:48 ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.

            [client(16056504)]CRS-10001:03-Sep-13 19:48 ACFS-9156: Detecting control device '/dev/ofsctl'.

            [client(16056508)]CRS-10001:03-Sep-13 19:48 ACFS-9322: completed

            2013-09-03 19:48:59.667

            [ctssd(18022612)]CRS-2409:The clock on host ggnetp04 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

            2013-09-03 19:50:53.936

            [cssd(17694820)]CRS-1662:Member kill requested by node ggnetp03 for member number 1, group DB+ASM

            2013-09-03 19:50:57.493

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:50:57.494

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log"

            2013-09-03 19:50:58.828

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:50:58.943

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:50:59.235

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:53:00.252

            [cssd(17694820)]CRS-1662:Member kill requested by node ggnetp03 for member number 1, group DB+ASM

            2013-09-03 19:53:03.101

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:53:03.102

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log"

            2013-09-03 19:53:05.430

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:53:05.539

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:53:05.815

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:55:07.436

            [cssd(17694820)]CRS-1662:Member kill requested by node ggnetp03 for member number 1, group DB+ASM

            2013-09-03 19:55:09.673

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:55:09.674

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log"

            2013-09-03 19:55:12.007

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:55:12.115

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:55:12.389

            [/opt/app/11.2.0.3/grid/bin/oraagent.bin(16777322)]CRS-5019:All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/opt/app/11.2.0.3/grid/log/ggnetp04/agent/ohasd/oraagent_grid/oraagent_grid.log".

            2013-09-03 19:55:12.405

            [ohasd(13697160)]CRS-2807:Resource 'ora.asm' failed to start automatically.

            2013-09-03 19:55:12.405

            [ohasd(13697160)]CRS-2807:Resource 'ora.crsd' failed to start automatically.

            2013-09-03 20:23:32.040

            [ctssd(18022612)]CRS-2409:The clock on host ggnetp04 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

            • 4. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
              Pradeepcmst

              Hi,

              Did you completed the installation without any error?

              Please check whether all nodes in the cluster follows same timezone and same time?

              And may i know what commands you used to start the CRS and DB

               

               

              Regards,

              Pradeep. V

              • 5. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
                Pradeepcmst

                Also please execute this command and let us know the status

                 

                cluvfy comp clocksync -n all -verbose

                • 6. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
                  dba.ravi

                  Hi ,

                   

                  We did upgrade successfully. i run the command as suggested by you . below are the o/p

                   

                  $ cluvfy comp clocksync -n all -verbose

                   

                   

                  Verifying Clock Synchronization across the cluster nodes

                   

                   

                  Checking if Clusterware is installed on all nodes...

                  Check of Clusterware install passed

                   

                   

                  Checking if CTSS Resource is running on all nodes...

                  Check: CTSS Resource running on all nodes

                    Node Name                             Status

                    ------------------------------------  ------------------------

                    ggnetp04                              passed

                    ggnetp03                              passed

                  Result: CTSS resource check passed

                   

                   

                   

                   

                  Querying CTSS for time offset on all nodes...

                  Result: Query of CTSS for time offset passed

                   

                   

                  Check CTSS state started...

                  Check: CTSS state

                    Node Name                             State

                    ------------------------------------  ------------------------

                    ggnetp04                              Observer

                    ggnetp03                              Observer

                  CTSS is in Observer state. Switching over to clock synchronization checks using NTP

                   

                   

                   

                   

                  Starting Clock synchronization checks using Network Time Protocol(NTP)...

                   

                   

                  NTP Configuration file check started...

                  The NTP configuration file "/etc/ntp.conf" is available on all nodes

                  NTP Configuration file check passed

                   

                   

                  Checking daemon liveness...

                   

                   

                  Check: Liveness for "xntpd"

                    Node Name                             Running?

                    ------------------------------------  ------------------------

                    ggnetp04                              yes

                    ggnetp03                              yes

                  Result: Liveness check passed for "xntpd"

                  Check for NTP daemon or service alive passed on all nodes

                   

                   

                  Checking NTP daemon command line for slewing option "-x"

                  Check: NTP daemon command line

                    Node Name                             Slewing Option Set?

                    ------------------------------------  ------------------------

                    ggnetp04                              yes

                    ggnetp03                              yes

                  Result:

                  NTP daemon slewing option check passed

                   

                   

                  Checking NTP daemon's boot time configuration, in file "/etc/rc.tcpip", for slewing option "-x"

                   

                   

                  Check: NTP daemon's boot time configuration

                    Node Name                             Slewing Option Set?

                    ------------------------------------  ------------------------

                    ggnetp04                              yes

                    ggnetp03                              no

                  Result:

                  NTP daemon's boot time configuration check for slewing option failed on some nodes

                  PRVF-5470 : The NTP daemon's boot time configuration, in file "/etc/rc.tcpip", on one or more nodes lacks the slewing option "-x"

                  Result: Clock synchronization check using Network Time Protocol(NTP) failed

                   

                   

                   

                   

                  PRVF-9652 : Cluster Time Synchronization Services check failed

                   

                   

                  Verification of Clock Synchronization across the cluster nodes was unsuccessful.

                  Checks did not pass for the following node(s):

                          ggnetp03

                  • 7. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
                    dba.ravi

                    Hi Pardeep ,

                     

                    Before start CRS

                     

                    $ ifconfig en1

                    en1: flags=5e080863,2c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN,MONITOR>

                           inet 172.18.*.* netmask 0xffffff00 broadcast 172.18.114.255

                             tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0

                     

                    After starting CRS

                     

                    en1: flags=5e080863,2c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN,MONITOR>

                            inet 172.18.*.* netmask 0xffffff00 broadcast 172.18.114.255

                            inet 169.254.98.19 netmask 0xffff0000 broadcast 169.254.255.255

                             tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0

                     

                    From where 169.254.98.19 IP is coming. even our unix/Network team not able to understand. N/W team refresh the ARP table and switch both. We took reboot the server . but problem is still same. in ASM alert log , this IP is coming for cluster interconnect rather than 172.18.114.209

                     

                    Regards

                    Ravi

                    • 8. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
                      Pradeepcmst

                      Hi Ravi,

                      Check this node ggnetp03, go to this location " /etc/ and open the file rc.tcpip

                      Configure it according to the MOC ID 1056693.1, restart the service as in DOC and again run cluvfy comp clocksync -n all -verbose to see any changes.

                       

                       

                       

                      Regards,

                      Pradeep. V

                      • 9. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
                        Pradeepcmst

                        Yes Oracle will assign some IP address internally.

                        • 11. Re: one node is up but when we start the crs on node 2 then ASM instance killed.
                          Pradeepcmst

                          Hi,

                          If its internal IP address, then it will be managed by Oracle clusterware. The area where we need to address related to IP is ,

                          i) Static IP address should be configured on all nodes in a cluster. eg : eth0 --> 192.168.1.20 which is public network . Any node in the network can communicate each other.

                          ii) Public VIP should be configured on all nodes in a cluster, It should be same subnet as public network where IP address is different: eg : 192.168.1. <any Ip address which cannot be pinged by any other nodes in the network. Only . Oracle uses a Virtual IP (VIP) for database access.  The VIP is used for RAC failover (TAF). Only RAC nodes within the cluster can communicate

                          iii) Private IP address: Oracle RAC requires "private IP" addresses to manage the CRS, the clusterware heartbeat process and the cache fusion layer.

                          iv) Apart from this SCAN IP which is new in Oracle 11g. If we configure this we can add/remove any node from the cluster without modifying the client tnsnames.ora

                           

                          Oracle have some IP assignments internally, which i dint check yet.

                          So,

                          Did you check these notes??

                           

                          Check this node ggnetp03, go to this location " /etc/ and open the file rc.tcpip

                          Configure it according to the MOC ID 1056693.1, restart the service as in DOC and again run cluvfy comp clocksync -n all -verbose to see any changes.

                           

                          Regards,

                          Pradeep. V