4 Replies Latest reply on May 26, 2013 8:31 AM by oracle.dba.89

    Faced an issue while setting up RAC on my virtual machine

      Hi All,

      While setting up RAC 10g on VMWare with Cent OS 4, i ran into a few issues which were solved by a reboot (last resort)

      The problem was after installing the database, while checking the cluster resources using crs_stat -t, the status was showing offline for 2 resources.
      [oracle@rac1 bin]$ ./crs_stat -t
      Name           Type           Target    State     Host
      ora....SM1.asm application    ONLINE    OFFLINE   rac1
      ora....C1.lsnr application    ONLINE    OFFLINE   rac1
      ora.rac1.gsd   application    ONLINE    ONLINE    rac1
      ora.rac1.ons   application    ONLINE    ONLINE    rac1
      ora.rac1.vip   application    ONLINE    ONLINE    rac1
      ora....SM2.asm application    ONLINE    ONLINE    rac2
      ora....C2.lsnr application    ONLINE    ONLINE    rac2
      ora.rac2.gsd   application    ONLINE    ONLINE    rac2
      ora.rac2.ons   application    ONLINE    ONLINE    rac2
      ora.rac2.vip   application    ONLINE    ONLINE    rac2
      The +ASM1 instance and listener on RAC1 was not started when I checked. So i started them manually.
      But while starting the instance, i got an error
      WARNING: cache failed to read fn=3  indblk=0 from disk(s): 2
      ORA-15196: invalid ASM block header [kfc.c:7910] [endian_kfbh] [3] [2147483648] [0 != 1]
      NOTE: a corrupted block was dumped to the trace file
      System State dumped to trace file /u01/app/oracle/admin/+ASM/bdump/+asm1_lgwr_12518.trc
      NOTE: cache initiating offline of disk 2  group 1
      WARNING: offlining disk 2.3912953495 (DATA_0002) with mask 0x3
      NOTE: PST update: grp = 1, dsk = 2, mode = 0x6
      Wed Sep 12 22:43:44 2012
      ERROR: too many offline disks in PST (grp 1)
      Wed Sep 12 22:43:44 2012
      NOTE: PST not enabling heartbeating (grp 1): group dismounted
      I was unable to dismount and remount the disk. It was giving an error saying
      ORA-15130: diskgroup "DISK" is being dismounted
      First of all can anyone tell me what "ora....SM1.asm" and "ora....C1.lsnr" are? Why was the TARGET column showing as online?

      I also tried shutting down the resources and then restarting the rawdevice services. But after doing that, while trying to start the resources (crsctl start resources) the command was just hung on both the instances.

      I then didn't know what to do and just rebooted both the machines and then saw that everything was up. I managed to create the database and everything is running fine now.

      I didn't know where else to look for errors. Can anyone help me figure out the issue please?

      Also now after setting everything up, I see the below error in the ASM alert log for both nodes.
      Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm1_rbal_9071.trc:
      ORA-15183: ASMLIB initialization error [/opt/oracle/extapi/32/asm/orcl/1/libasm.so]
      ORA-15183: ASMLIB initialization error [driver/agent not installed]
      I checked oracle support doc ID 1054033.1 and tried checking the permissions. But I don't see the file asmadmin present in $ORACLE_HOME/bin

      From the support doc
      1. Manually change the oracle binary ownership and permission:
      as root user:
      # cd $ORACLE_HOME/bin
      # chgrp asmadmin oracle
      # chmod 6751 oracle
      # ls -l oracle
        • 1. Re: Faced an issue while setting up RAC on my virtual machine
          I only read your first sentence ... let me repeat what I think I just read.
          While setting up RAC using a version of Oracle currently in desupport in a virtual environment never certified or supported by Oracle 
          with an operating system never certified or supported by Oracle, you ran into a few issues.
          did I miss anything?

          My recommendation, without reading past what I just parodied above, is that you format the drive, install Oracle Enterprise Linux, Oracle VM, and or above following the installation documentation published by Oracle Corp.

          Then if you have issues people like me will read your entire post.

          My point is not to be rude but rather the point out what should be obvious. If you don't follow the docs, don't use supported software and configurations, and have problems it is essentially impossible for us to know what might have gone wrong. We can guess ... but we can not do more than that.
          • 2. Re: Faced an issue while setting up RAC on my virtual machine

            I agree with damorgan.

            we should follow Oracle Docs guideline and use only Cerrtified OS for Production and other environment.

            Edited by: 942626 on Sep 12, 2012 11:47 PM
            • 3. Re: Faced an issue while setting up RAC on my virtual machine

              Need to start servuces in sequence.

              can u stop service properly as follows and make sure they are stopped properly by checking status using crscrtl.

              (1) stop db services ,instance services
              (2) stop asm
              (3) stop nodeapps services
              (4) stop CRS SERVICES

              now start services in seqeunce.

              (1) start CRS services
              (2) start nodeapp services
              (3) start asm
              (4) start instance ,db services

              Edited by: Rajesh.Rathod on Sep 12, 2012 11:50 PM

              Edited by: Rajesh.Rathod on Sep 12, 2012 11:51 PM

              Edited by: Rajesh.Rathod on Sep 13, 2012 2:51 AM
              • 4. Re: Faced an issue while setting up RAC on my virtual machine
                I am gonna find a supported virtual solution for this and set up everything on that :)

                I'll close this thread for now