3 Replies Latest reply: Jun 12, 2014 12:08 AM by Aritra-Oracle RSS

    ASM diskgroup space not getting freed up after RMAN archivelog delete

    Omega3

      Grid Infra version: 11.2.0.4

      DB version: 11.2.0.4

      OS : Oracle Linux 6.4

      Storage : Enterprise Class storage ( EMC VMAX )

       

      We are into banking we have a highly critical 2-Node Production RAC DB.

      Archive logs are stored in ARCH_DG diskgroup. Due to unexpected heavy load, our ARCH_DG diskgroup became full and we got the usual error

       

      ORA-00257: archiver error. Connect internal only, until freed.

       

       

      To quickly bring the DB back up for business, I tried deleting old archive logs older than 1 day from this diskgroup using the following command first

       

      RMAN> delete noprompt archivelog until time 'SYSDATE-1';

       

      So, I started seeing the following message


      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_04/thread_2_seq_10066.331.849396309 RECID=19807 STAMP=849396325

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_04/thread_2_seq_10067.274.849396925 RECID=19809 STAMP=849396940

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_04/thread_2_seq_10068.341.849397529 RECID=19810 STAMP=849397543

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_04/thread_2_seq_10069.298.849398167 RECID=19812 STAMP=849398182

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_05/thread_2_seq_10070.317.849398783 RECID=19813 STAMP=849398806

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_05/thread_2_seq_10071.342.849399327 RECID=19814 STAMP=849399337

       

       

      Although I could see 'deleted archived log' messages as shown above there were 2 inconsistencies (Weird behaviour )

       

      Inconsistency 1. Not a single byte was freed up from the archive diskgroup. Each archive log is 4GB in size. I got 'deleted archived log' message for at least 25 archived log files. So, 100 GB should have been freed up.

       

      The archive diskgroup remained at 99.9% as shown below and the DB was inaccessible.

       

      SQL> select name,

      round(total_mb/1024,2) Total_GB,

      round(free_mb/1024,2) Free_GB,

      free_mb/total_mb*100 "%Free" ,

      abs((free_mb/total_mb*100)-100) "Percent Full"

      from v$asm_diskgroup;  2    3 4    5    6

      NAME                             TOTAL_GB    FREE_GB      %Free Percent Full
      ------------------------------ ---------- ---------- ---------- ------------
      ARCH_DG                              1000        .78 .077636719   99.9223633
      DATA_DG                          10000.02    1188.19 11.8818616   88.1181384
      OCR_VOTE_DG                            12      11.09 92.4623525   7.53764754
      <snipped>


      Inconsistency2. The count of available archived logs remained unchanged. It was always showing 301

       

      SQL> select count(*) from v$archived_log where status = 'A';

       

      COUNT(*)

      ----------

      301

       

      Later RMAN> delete obsolete ; command deleted 17 archived logs .

      This means 17 x 4 GB = 68 GB should have been freed up.

       

      But Diskgroup free space and the count of v$archived_log.status='A' remained unchanged as mentioned above.

       

      So, from ASMCMD , I manually deleted a directory full of archive logs using rm -rf command.

       

      Even after running rm -rf on an entire directory full of archived logs the space was not freed up and we were still getting "ORA-00257: archiver error" .

      So, we had to bring down the cluster using "crsctl stop crs" . After the cluster was brought up using "crsctl start crs" on both nodes,

      the space was freed up and the DB became accesible again.

       

      I am wondering why the ASM diskgroup space was not freeing up despite deleting archive logs . Are Inconsistencies 1&2 mentioned above caused by some open file handle thing ?