3 Replies Latest reply: Sep 7, 2011 10:05 AM by user179308 RSS

    +ASM2 unable to start following GI Upgrade from 11.2.0.1 to 11.2.0.2

    user179308
      Hi all,

      Not going anywhere fast or far with MOS, so I thought I would ask for help here.

      Following an out of place upgrade of GI 11.2.0.1.2 to 11.2.0.2 on a 2 node cluster running OEL 5 the asm instance on node 2 is failing to start, OCR and voting are stored within asm. Patch 9706490 was applied before starting the upgrade and all but the last step, cluvfy verification, in the upgrade was successful. cluvfy verification failed due to asm2 not running, see below alert logs for both.  It appears that ASM1 is preventing +ASM2 from starting.

      The activeversion and releaseversion from crsctl on both nodes are reporting 11.2.0.2


      <p><i>
      +ASM1 Alert log
      =================

      Mon Sep 05 14:57:01 2011
      LMON (ospid: 2450) detects hung instances during IMR reconfiguration
      LMON (ospid: 2450) tries to kill the instance 2 in 37 seconds.
      Please check instance 2's alert log and LMON trace file for more details.
      Mon Sep 05 14:57:38 2011
      Remote instance kill is issued with system inc 68
      Remote instance kill map (size 1) : 2
      LMON received an instance eviction notification from instance 1
      The instance eviction reason is 0x20000000
      The instance eviction map is 2
      Reconfiguration started (old inc 68, new inc 70)
      List of instances:
      1 (myinst: 1)
      Global Resource Directory frozen
      Communication channels reestablished
      Master broadcasted resource hash value bitmaps
      Non-local Process blocks cleaned out
      Mon Sep 05 14:57:39 2011
      LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
      Set master node info
      Submitted all remote-enqueue requests
      Dwn-cvts replayed, VALBLKs dubious
      All grantable enqueues granted
      Submitted all GCS remote-cache requests
      Fix write in gcs resources
      Reconfiguration complete

      +ASM2 Alert log
      ==================

      Starting up:
      Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
      With the Real Application Clusters and Automatic Storage Management options.
      Using parameter settings in server-side spfile +PRE_FRA/spfileASM.ora
      System parameters with non-default values:
      large_pool_size = 12M
      instance_type = "asm"
      remote_login_passwordfile= "EXCLUSIVE"
      asm_diskgroups = "PRE_FRA"
      asm_diskgroups = "PRE_DATA"
      asm_diskgroups = "PROD_DATA"
      asm_diskgroups = "PROD_FRA"
      asm_power_limit = 1
      diagnostic_dest = "/u01/app/grid"
      Cluster communication is configured to use the following interface(s) for this instance
      169.254.39.173
      cluster interconnect IPC version:Oracle UDP/IP (generic)
      IPC Vendor 1 proto 2
      Mon Sep 05 14:55:40 2011
      PMON started with pid=2, OS id=19969
      Mon Sep 05 14:55:40 2011
      PSP0 started with pid=3, OS id=19971
      Mon Sep 05 14:55:41 2011
      VKTM started with pid=4, OS id=19973 at elevated priority
      VKTM running at (1)millisec precision with DBRM quantum (100)ms
      Mon Sep 05 14:55:41 2011
      GEN0 started with pid=5, OS id=19977
      Mon Sep 05 14:55:41 2011
      DIAG started with pid=6, OS id=19979
      Mon Sep 05 14:55:41 2011
      PING started with pid=7, OS id=19981
      Mon Sep 05 14:55:41 2011
      DIA0 started with pid=8, OS id=19983
      Mon Sep 05 14:55:41 2011
      LMON started with pid=9, OS id=19985
      Mon Sep 05 14:55:41 2011
      LMD0 started with pid=10, OS id=19987
      * Load Monitor used for high load check
      * New Low - High Load Threshold Range = [23040 - 30720]
      Mon Sep 05 14:55:41 2011
      LMS0 started with pid=11, OS id=19989 at elevated priority
      Mon Sep 05 14:55:41 2011
      LMHB started with pid=12, OS id=19993
      Mon Sep 05 14:55:41 2011
      MMAN started with pid=13, OS id=19995
      Mon Sep 05 14:55:41 2011
      DBW0 started with pid=14, OS id=19997
      Mon Sep 05 14:55:41 2011
      LGWR started with pid=15, OS id=19999
      Mon Sep 05 14:55:41 2011
      CKPT started with pid=16, OS id=20001
      Mon Sep 05 14:55:41 2011
      SMON started with pid=17, OS id=20003
      Mon Sep 05 14:55:41 2011
      RBAL started with pid=18, OS id=20005
      Mon Sep 05 14:55:41 2011
      GMON started with pid=19, OS id=20007
      Mon Sep 05 14:55:41 2011
      MMON started with pid=20, OS id=20009
      Mon Sep 05 14:55:41 2011
      MMNL started with pid=21, OS id=20011
      lmon registered with NM - instance number 2 (internal mem no 1)
      Mon Sep 05 14:57:41 2011
      PMON (ospid: 19969): terminating the instance due to error 481
      Mon Sep 05 14:57:41 2011
      System state dump requested by (instance=2, osid=19969 (PMON)), summary=[abnormal instance termination].
      System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_diag_19979.trc
      Dumping diagnostic data in directory=[cdmp_20110905145741], requested by (instance=2, osid=19969 (PMON)), summary=[abnormal instance termination].
      Instance terminated by PMON, pid = 19969
      </i></p>