3 Replies Latest reply on Jun 12, 2014 5:08 AM by Aritra-Oracle

    ASM diskgroup space not getting freed up after RMAN archivelog delete


      Grid Infra version:

      DB version:

      OS : Oracle Linux 6.4

      Storage : Enterprise Class storage ( EMC VMAX )


      We are into banking we have a highly critical 2-Node Production RAC DB.

      Archive logs are stored in ARCH_DG diskgroup. Due to unexpected heavy load, our ARCH_DG diskgroup became full and we got the usual error


      ORA-00257: archiver error. Connect internal only, until freed.



      To quickly bring the DB back up for business, I tried deleting old archive logs older than 1 day from this diskgroup using the following command first


      RMAN> delete noprompt archivelog until time 'SYSDATE-1';


      So, I started seeing the following message

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_04/thread_2_seq_10066.331.849396309 RECID=19807 STAMP=849396325

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_04/thread_2_seq_10067.274.849396925 RECID=19809 STAMP=849396940

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_04/thread_2_seq_10068.341.849397529 RECID=19810 STAMP=849397543

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_04/thread_2_seq_10069.298.849398167 RECID=19812 STAMP=849398182

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_05/thread_2_seq_10070.317.849398783 RECID=19813 STAMP=849398806

      deleted archived log

      archived log file name=+ARCH_DG/bsoaprd/archivelog/2014_06_05/thread_2_seq_10071.342.849399327 RECID=19814 STAMP=849399337



      Although I could see 'deleted archived log' messages as shown above there were 2 inconsistencies (Weird behaviour )


      Inconsistency 1. Not a single byte was freed up from the archive diskgroup. Each archive log is 4GB in size. I got 'deleted archived log' message for at least 25 archived log files. So, 100 GB should have been freed up.


      The archive diskgroup remained at 99.9% as shown below and the DB was inaccessible.


      SQL> select name,

      round(total_mb/1024,2) Total_GB,

      round(free_mb/1024,2) Free_GB,

      free_mb/total_mb*100 "%Free" ,

      abs((free_mb/total_mb*100)-100) "Percent Full"

      from v$asm_diskgroup;  2    3 4    5    6

      NAME                             TOTAL_GB    FREE_GB      %Free Percent Full
      ------------------------------ ---------- ---------- ---------- ------------
      ARCH_DG                              1000        .78 .077636719   99.9223633
      DATA_DG                          10000.02    1188.19 11.8818616   88.1181384
      OCR_VOTE_DG                            12      11.09 92.4623525   7.53764754

      Inconsistency2. The count of available archived logs remained unchanged. It was always showing 301


      SQL> select count(*) from v$archived_log where status = 'A';






      Later RMAN> delete obsolete ; command deleted 17 archived logs .

      This means 17 x 4 GB = 68 GB should have been freed up.


      But Diskgroup free space and the count of v$archived_log.status='A' remained unchanged as mentioned above.


      So, from ASMCMD , I manually deleted a directory full of archive logs using rm -rf command.


      Even after running rm -rf on an entire directory full of archived logs the space was not freed up and we were still getting "ORA-00257: archiver error" .

      So, we had to bring down the cluster using "crsctl stop crs" . After the cluster was brought up using "crsctl start crs" on both nodes,

      the space was freed up and the DB became accesible again.


      I am wondering why the ASM diskgroup space was not freeing up despite deleting archive logs . Are Inconsistencies 1&2 mentioned above caused by some open file handle thing ?