3 Replies Latest reply: Feb 18, 2013 8:56 AM by onedbguru RSS

    DiskGroup falls over when creating tablespace - ORA-15196

    Red Hand Luke
      Running RedHat6 on Linux x86-64 platform, and have installed ASM and Database versions 11.2.0.3 on RAC 2 node cluster
      All appears ok, and I can connect to database and also to ASM database.
      When I attempt to create tablespace on +data diskgroup, I get the following:

      CREATE SMALLFILE TABLESPACE DATA01_TBS DATAFILE '+DATA' SIZE 16G AUTOEXTEND ON NEXT 2G MAXSIZE 32G LOGGING EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO
      *
      ERROR at line 1:
      ORA-01119: error in creating database file '+DATA'
      ORA-17502: ksfdcre:4 Failed to create file +DATA
      ORA-15196: invalid ASM block header [kfc.c:19572] [check_kfbh] [1] [268]
      [537466657 != 2985693367]
      ORA-15196: invalid ASM block header [kfc.c:19572] [check_kfbh] [268]
      [2147483648] [0 != 283281628]
      ORA-15196: invalid ASM block header [kfc.c:19572] [check_kfbh] [1] [268]
      [537466657 != 2985693367]
      ORA-15196: invalid ASM block header [kfc.c:19572] [check_kfbh] [8] [60]
      [18515249 != 18516943]
      ORA-15196: invalid ASM block header [kfc.c:19572] [check_kfbh] [2147483648] [1]
      [2469852149 != 2469824884]
      ORA-15130: diskgroup "DATA" is being dismounted
      ORA-15066: offlining disk "DATA_0000" in group "DATA" may result in a data loss
      ORA-15196: invalid ASM block header [kfc.c:26076] [check_kfbh] [2147483648] [4]
      [2349829711 != 1625096967]
      ORA-15196: invalid ASM block header [kfc.c:26076] [check_kfbh] [2147483648] [4]
      [2349829711 != 1625096967]

      Any help appreciated.

      Thanks.

      Edited by: Red Hand Luke on Feb 16, 2013 4:08 PM
        • 1. Re: DiskGroup falls over when creating tablespace
          onedbguru
          what does your disk partition list look like? Did you partition off the first 1Mb? your ASM partition should start at cylinder 2?

          Could you read and write to all devices as the grid and oracle user?

          What does the alert log for ASM and the cluster and the database look like?
          • 2. Re: DiskGroup falls over when creating tablespace
            Red Hand Luke
            Extract from ASM alert log file:

            WARNING: cache read a corrupt block: group=1(DATA) dsk=0 blk=4 disk=0 (DATA_0000) incarn=3915954476 au=0 blk=4 count=1
            Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4988.trc:
            ORA-15196: invalid ASM block header [kfc.c:26076] [check_kfbh] [2147483648] [4] [2349829711 != 1625096967]
            NOTE: a corrupted block from group DATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4988.trc
            WARNING: cache read (retry) a corrupt block: group=1(DATA) dsk=0 blk=4 disk=0 (DATA_0000) incarn=3915954476 au=0 blk=4 count=1
            Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4988.trc:
            ORA-15196: invalid ASM block header [kfc.c:26076] [check_kfbh] [2147483648] [4] [2349829711 != 1625096967]
            ORA-15196: invalid ASM block header [kfc.c:26076] [check_kfbh] [2147483648] [4] [2349829711 != 1625096967]
            ERROR: cache failed to read group=1(DATA) dsk=0 blk=4 from disk(s): 0(DATA_0000)
            ORA-15196: invalid ASM block header [kfc.c:26076] [check_kfbh] [2147483648] [4] [2349829711 != 1625096967]
            ORA-15196: invalid ASM block header [kfc.c:26076] [check_kfbh] [2147483648] [4] [2349829711 != 1625096967]
            NOTE: cache initiating offline of disk 0 group DATA
            NOTE: process user4988+asm1 (4988) initiating offline of disk 0.3915954476 (DATA_0000) with mask 0x7e in group 1
            WARNING: Disk 0 (DATA_0000) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
            NOTE: initiating PST update: grp = 1, dsk = 0/0xe968b92c, mask = 0x6a, op = clear
            Fri Feb 15 11:04:24 2013
            GMON updating disk modes for group 1 at 7 for pid 33, osid 4988
            ERROR: Disk 0 cannot be offlined, since diskgroup has external redundancy.
            ERROR: too many offline disks in PST (grp 1)
            ------------------------------------------------------------------------------------------------------------------------------------------
            Yes first section of asm disk was set aside, I used parted command to create a whole partition disk starting at 1 and ending at -1

            Also, should mention I am using mutli-pathed disks.
            ------------------------------------------------------------------------------------------------------------------------------------------
            Tried to create some test tablespaces as follows:
            Created a 32mb tablespace in +flash - worked
            Created a 8gb tablespace in flash - disk group flash dismounted
            Created a 32mb tablespace in +data - worked
            Created a 512mb tablespace in +data - worked
            Created a 1gb tablespace in data - disk group data dismounted

            Creating a tablespace >=1gb appears to cause problems!!

            Edited by: Red Hand Luke on Feb 16, 2013 5:45 PM
            • 3. Re: DiskGroup falls over when creating tablespace
              onedbguru
              Any time you are dealing with corruption like this, you need to call Oracle Support. Hopefully you have a backup, but it sounds like this is a *** completely new install that has never worked ***.

              In that situation, ** I ** would wipe out all of the ASM configurations (find $GRID_HOME/ -name roothas.pl ) and google/bing for delete grid from node (something like roothas.pl -deconfig crs -f). Do this on all nodes.

              Do not delete the binaries!!!

              shut down the rdbms instances
              I would then wipe all of the ASM devices (dd if=/dev/zero of=/dev/mapper/{all_asm_devices})
              Verify that the grid and oracle users can read and write to ALL devices on ALL nodes.
              Once you have confirmed the read AND write tests on ALL nodes for both grid and oracle users has succeeded,
              go to node 1 and rerun root.sh
              go to node 2 and rerun root.sh
              etc...

              Now reconfigure the diskgroups and try to create the database.