This discussion is archived
1 2 Previous Next 25 Replies Latest reply: Feb 13, 2013 11:43 AM by 864103 RSS

background dump

864103 Newbie
Currently Being Moderated
Hi expert ,

oracle 11g.2 ASM RAC under RHEL 5 2 nodes

today i face an issue with my database
./sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Tue Feb 12 15:19:35 2013

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

ERROR:
ORA-09817: Write to audit file failed.
Linux-x86_64 Error: 28: No space left on device
Additional information: 12
ORA-01075: you are currently logged on
i found these alert and trace file take more space from /home file system so i clean up this file system vie delete some logs
but i found there is error is alert log
WARNING: Read Failed. group:1 disk:4 AU:11852 offset:892928 size:8192
WARNING: failed to read mirror side 1 of virtual extent 1722 logical extent 0 of file 285 in group [1.4277692974] from disk DATA01_0004  allocation unit 11852 reason error; if possible,will try another mirror side
Errors in file /home/app/oracle/diag/rdbms/absnl/ABSNL1/trace/ABSNL1_ora_24108.trc:
WARNING: Read Failed. group:1 disk:4 AU:11852 offset:892928 size:8192
WARNING: failed to read mirror side 1 of virtual extent 1722 logical extent 0 of file 285 in group [1.4277692974] from disk DATA01_0004  allocation unit 11852 reason error; if possible,will try another mirror side
also the /home increases every minute
[oracle@ccbs01 alert]$ df -h

/dev/cciss/c0d0p9      48G   40G  5.5G  88% /home

after 2 min

[oracle@ccbs01 trace]$ df -h

/dev/cciss/c0d0p9      48G   44G  1.9G  96% /home
please advice ...
  • 1. Re: background dump
    sb92075 Guru
    Currently Being Moderated
    identify what is writing data under /home & kill it.

    lsof reports open files.
  • 2. Re: background dump
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Check MOS Notes about this problem ,
    WARNING: Failed To Read Mirror Side 1 continuously reported in the alert log [ID 1289905.1]
    WARNING: IO Failed…’ ‘WARNING: failed to read mirror side 1 of virtual extent …’ [ID 1274852.1]
    Bug 10422126: FAILED TO READ MORROR SIDE 1 and there is a 70KB patch for 11.2.0.2.

    Download the patch apply it
  • 3. Re: background dump
    864103 Newbie
    Currently Being Moderated
    how ???
  • 4. Re: background dump
    jgarry Guru
    Currently Being Moderated
    Every time you do a sysdba or sysoper operation there is an operating system audit file generated. This includes sys as sysdba and rman backups. Check syslog.conf for where they are going and show parameter audit_trail. These can really add up after a period of time.
  • 5. Re: background dump
    sb92075 Guru
    Currently Being Moderated
    861100 wrote:
    how ???
    culprit files will have recent timestamp
  • 6. Re: background dump
    864103 Newbie
    Currently Being Moderated
    sorry guys , i don't know what i do this file system is increased
    and i just remove the old logs i can't find any thing if the issue is with DISKGROUP why there is no issue
    in node 2 ??? how i kill any process from writing in /home ????
  • 7. Re: background dump
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Check MOS note that i posted.
  • 8. Re: background dump
    864103 Newbie
    Currently Being Moderated
    but you told me i need to apply the patch it's normal when i have error i must upgrade my database
    i have 11.2.0.2.0
  • 9. Re: background dump
    jgarry Guru
    Currently Being Moderated
    There is likely more than one thing wrong. Did you check the note that Osama posted? That fits the error you see in the alert log. Do you have lsof on your system? sb's idea of seeing what is open and killing that off is a good one, although if some runaway forms (or something) process is rapidly opening and closing new files that could be confusing.

    Some background processes open and close the same trace files, making them huge, since the filename is based partly on the process id. I sometimes do simple-minded searches like ls -Rl|sort -nrk5 (on my system, that recursively searches directories and sorts files by size, your system may have different commands to do the same thing), sometimes it makes it obvious what is going wrong (like having giant shared server process trace files on a little XE system). There may also be some adrci and xml things to clean up, I don't know much about that. Sometimes merely taking a long time to cd to a trace directory could be a clue there are thousands of unnecessary files there. Sometimes simply creating a file that is a long listing of all the files on the device and looking at it can identify the problem.
  • 10. Re: background dump
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    861100 wrote:
    but you told me i need to apply the patch it's normal when i have error i must upgrade my database
    i have 11.2.0.2.0
    I didn't explain it well , you have to option after checking MOS notes 1- Apply patch to fix the bug 2- Upgrade database to 11.2.0.3
    so you have two options.
    also Joel explain it well .
  • 11. Re: background dump
    864103 Newbie
    Currently Being Moderated
    okay guys , i will send an email to my manager first to tell him we have bug in our system and we need to apply patch
    but for a temporary sol can i delete the alert log files and then oracle can generated
     ls -ltrh alert_ABSNL1.log
    -rw-r----- 1 oracle dba 22G Feb 12 22:15 alert_ABSNL1.log
    it's possible right ??? once i rename it or remove it oracle can gerenarted it self???
  • 12. Re: background dump
    jgarry Guru
    Currently Being Moderated
    If you have that much in your normal alert log, you might want to check whether adcri is purging too.

    You might want to tail the alert log into another file before deleting it, perhaps the last couple thousand lines, just so you can look for other errors.
  • 13. Re: background dump
    864103 Newbie
    Currently Being Moderated
    okay this is good info. but how can i connect to adrci???
  • 14. Re: background dump
    sb92075 Guru
    Currently Being Moderated
    861100 wrote:
    okay this is good info. but how can i connect to adrci???
    when all else fails, Read The Fine Manual

    http://docs.oracle.com/cd/E11882_01/server.112/e25494/diag001.htm#ADMIN11266
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points