1 2 Previous Next 23 Replies Latest reply on Apr 1, 2010 6:53 PM by lrp1 Go to original post
      • 15. Re: ORA-27102 SVR4 Error: 12: Not enough space
        lrp
        Absolutely no /var/adm/messages related to the failure (which is what keeps pissing me off, since Metalink's notes on the 27063 error routinely point me to the OS logs) -- I just spoke with the SA, and most of the messages get piped there. He did have sar available with a viewer, but i also had charts for the disk i/o at the time -- nothing stood out.
        • 16. Re: ORA-27102 SVR4 Error: 12: Not enough space
          561365
          solaris 10 require the setting of memory limit

          projadd -U oracle -K "project.max-shm-memory=(priv,4096MB,deny)" user.oracle

          The memory specified above should be more than the actual sga &pga you are using.

          Thanks
          1 person found this helpful
          • 17. Re: ORA-27102 SVR4 Error: 12: Not enough space
            lrp
            That was the first thing we increased -- Our 11g memory target (which is the combined SGA+PGA, automatically managed by the instance) is at 5GB, with a memory_max_size = 8G. While it is the only database in the solaris project there ARE other databases on the prod server, contained within their own project (ie. oraproj2) and ORACLE_HOME.

            If I'm reading the results below correctly, I believe we have 16GB of shared memory available:
            oracle@server1:PROD01:/var/adm> projects -l oraproj
            oraproj
                    projid : 102
                    comment: ""
                    users  : oracle
                    groups : (none)
                    attribs: project.max-sem-ids=(priv,200,deny)
                             project.max-sem-nsems=(priv,512,deny)
                             project.max-shm-ids=(priv,200,deny)
                             project.max-shm-memory=(priv,17179869184,deny)
            • 18. Re: ORA-27102 SVR4 Error: 12: Not enough space
              jgarry
              lrp wrote:
              Good thoughts, but we had ruled that out early: Filesystem's not the issue, the disk had plenty of space. Metalink itself (472813.1) points to "the Unix error number (ERRNO) 12 during a Unix write()/open() system call, and this Unix error indicates a lack of *process memory* rather than physical disk space."

              - /var/adm/messages has no memory- or disk-related messages around the time of failure.
              - SAN administrator saw nothing in their logs at the time of failure

              We had already tried raising SGA and raising shared memory in the solaris project, but it seems like we're fishing for an answer by blindly raising a parameter when we don't know what OS limit Oracle had reached. The key numbers I'm looking for are those specified in the 'additional information' section. Oracle's knowledge base has nothing that I can use so far.
              On a different version and platform, I had rare issues when running RMAN. Eventually I came to the conclusion that: RMAN uses large pool; OS eventually fragments I/O buffers due to the way RMAN uses I/O on that platform. It is of course wild speculation that this has anything to do with your problem, but given the lack of real information, I'd say you perhaps want to shrink your SGA some, give more to large pool, and pray a lot. Oh, and sample the large pool SGA statistics when you are running RMAN, and wonder if the parallel automatic tuning is shooting your large_pool_size in the leg if you are using it.
              1 person found this helpful
              • 19. Re: ORA-27102 SVR4 Error: 12: Not enough space
                sb92075
                I just stumble across this

                Subject:      Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device
                     Doc ID:      301830.1

                Let us know if it helped.
                • 20. Re: ORA-27102 SVR4 Error: 12: Not enough space
                  lrp
                  Subject: Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device
                  Doc ID: 301830.1>
                  Thank you. I looked up the article, and it shows a similar message but has a key distinction between error codes -- that OS code was error: 28, while my error was error: 12. The linux and Solaris error code tables are similar, so for OS error 28, the the solaris error code table shows:
                  "28: ENOSPC No space left on device
                  While writing an ordinary file or creating a directory entry, there is no free space
                  left on the device. In the fcntl routine, the setting or removing of record locks
                  on a file cannot be accomplished because there are no more record entries left
                  on the system.{code}
                  By the same token, my OS error code 12 that I'm seeing appears to mean:
                  {code}
                  "12 ENOMEM Not enough space
                  During execution of an exec, brk, or sbrk routine, a program asks for more space
                  than the system is able to supply. This is not a temporary condition; the maximum
                  size is a system parameter. On some architectures, the error may also occur if the
                  arrangement of "text, data, and stack segments requires too many"
                  "segmentation registers, or if there is not enough swap space" during the fork
                  routine. If this error occurs on a resource associated with Remote File
                  Sharing (RFS), it indicates a memory depletion which may be temporary,
                  dependent on system activity at the time the call was invoked.
                  Emphasis on the clause "..*text, data, and stack segments requires too many segmentation registers, or if there is not enough swap space*.."

                  ..So my clues from the OS documentation point to some maximum, like swap, segmentation, stack and other resources.
                  I can only assume the "additional information" of -1 and 8192 are relevant numbers to those resources.

                  The two things that are *8192* in my environment appear to be # of file descriptors and stack size.

                  Therefore, my plan of attack is going to change both in my oracle profile to see if this occurs again:

                  ulimit -n 16834 (raise file descriptors per process from 8kto 16k)
                  ulimit -s 32767 (raise stack from 8mb to 32mb)

                  Hopefully, this will give Oracle more leeway to use OS resources AND give me extra clues if the error shows up again. In other words, if something like stack is truly the issue, then I expect to see another crash with addtional information = *32767* instead of 8192:
                  KCF: write/open error block=0x1571 online=1
                  file=57 /datafile/DB_001.dbf
                  error=27063 txt: 'SVR4 Error: 12: Not enough space
                  Additional information: -1
                  Additional information: "32767"'
                  Will update this thread with any relevant results..
                  • 21. Re: ORA-27102 SVR4 Error: 12: Not enough space
                    orafad
                    lrp wrote:
                    ..So my clues from the OS documentation point to some maximum, like swap, segmentation, stack and other resources.
                    I can only assume the "additional information" of -1 and 8192 are relevant numbers to those resources.

                    The two things that are *8192* in my environment appear to be # of file descriptors and stack size.
                    As I tried to explain earlier: this means no more than "I (Oracle) asked to read/write 8192 bytes from the buffer but got back -1". Syscall returns number of bytes actually read/written, or -1 indicating that an error occurred.

                    error=27063 txt: 'SVR4 Error: 12: Not enough space
                    Is this from a 32-bit Oracle server?

                    As the software owner user, could you verify system parameters? (prctl)

                    Specifically, what are your parameters settings that corresponds to shmmax and shmall?
                    1 person found this helpful
                    • 22. Re: ORA-27102 SVR4 Error: 12: Not enough space
                      lrp
                      >
                      As I tried to explain earlier: this means no more than "I (Oracle) asked to read/write 8192 bytes from the buffer but got back -1". Syscall returns number of bytes actually read/written, or -1 indicating that an error occurred.
                      error=27063 txt: 'SVR4 Error: 12: Not enough space
                      >
                      I do recall the post. Is there a man page describing that error code that I can look into further? I realize that the error number, but I didn't happen to find anywhere which stated the details behind 'errno' or return code. The man page for syscall talks about returning -1 on error, but doesn't say anything about the second return code ( unless you mean the variable errno ).

                      >
                      Is this from a 32-bit Oracle server?
                      As the software owner user, could you verify system parameters? (prctl)
                      Specifically, what are your parameters settings that corresponds to shmmax and shmall?
                      >
                      It's Oracle 64-bit Enterprise on Solaris 10 SPARC 64-bit. I hate pasting the full text of a screendump, but since I cannot really figure which are the important pieces of info, I'll paste the results of prctl on the project that my DB is running under. (if you can narrow which params i'm looking for, i can cull the rest later in an edit). This is only for one of the db's but the idea is the same. Note that project.max-shm-memory = 16 GB, well above my oracle memory_max_size of 8GB. I'm not sure how to obtain the solaris equivalent of shmall.
                      oracle@server1:PROD01:/fs1> prctl -i project oraproj
                      project: 100: oraproj
                      NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
                      project.max-contracts
                              privileged      10.0K       -   deny                                 -
                              system          2.15G     max   deny                                 -
                      project.max-device-locked-memory
                              privileged      1.95GB      -   deny                                 -
                              system          16.0EB    max   deny                                 -
                      project.max-locked-memory
                              system          16.0EB    max   deny                                 -
                      project.max-port-ids
                              privileged      8.19K       -   deny                                 -
                              system          65.5K     max   deny                                 -
                      project.max-shm-memory
                              privileged      16.0GB      -   deny                                 -          <----------
                              system          16.0EB    max   deny                                 -
                      project.max-shm-ids
                              privileged        200       -   deny                                 -
                              system          16.8M     max   deny                                 -
                      project.max-msg-ids
                              privileged        258       -   deny                                 -
                              system          16.8M     max   deny                                 -
                      project.max-sem-ids
                              privileged        200       -   deny                                 -
                              system          16.8M     max   deny                                 -
                      project.max-crypto-memory
                              privileged      7.81GB      -   deny                                 -
                              system          16.0EB    max   deny                                 -
                      project.max-tasks
                              system          2.15G     max   deny                                 -
                      project.max-lwps
                              system          2.15G     max   deny                                 -
                      project.cpu-cap
                              system          4.29G     inf   deny                                 -
                      project.cpu-shares
                              privileged          1       -   none                                 -
                              system          65.5K     max   none                                 -
                      zone.max-swap
                              system          16.0EB    max   deny                                 -
                      zone.max-locked-memory
                              system          16.0EB    max   deny                                 -
                      zone.max-shm-memory
                              system          16.0EB    max   deny                                 -
                      zone.max-shm-ids
                              system          16.8M     max   deny                                 -
                      zone.max-sem-ids
                              system          16.8M     max   deny                                 -
                      zone.max-msg-ids
                              system          16.8M     max   deny                                 -
                      zone.max-lwps
                              system          2.15G     max   deny                                 -
                      zone.cpu-cap
                              system          4.29G     inf   deny                                 -
                      zone.cpu-shares
                              privileged          1       -   none                                 -
                      Edited by: lrp on Jun 4, 2009 4:23 PM
                      • 23. Re: ORA-27102 SVR4 Error: 12: Not enough space
                        lrp1
                        Hello--I'm just updating the post with further information.
                        There is a Nov 9 2009 SUN Blog post (http://blogs.sun.com/hippy/entry/problems_with_solaris_and_a) which mentions similar symptoms to our problem and mentions basically to either:
                        a) upgrade to solaris 10 update 8 (we are at update 4)
                        b) disable oracle DISM

                        The article appears to advise turning off only SGA_MAX_SIZE. We currently have several memory settings:
                        memory_max_target                    big integer 8000M
                        memory_target                        big integer 5056M
                        shared_memory_address                integer     0
                        sga_max_size                         big integer 8000M
                        sga_target                           big integer 0
                        pga_aggregate_target                 big integer 0
                        If we were to disable DISM, does that mean disabling ONLY SGA_MAX_SIZE, or should we also remove the MEMORY_MAX_SIZE? If we wanted to maintain the same memory settings, would we then set memory_target to 8000M and leave SGA_TARGET/PGA_AGGREGATE_TARGET completely alone?

                        All in all, Oracle Support still does not give us many clues beyond saying that it is an OS file-resource error, not recognizing that it is a solaris kernel memory limit.
                        1 2 Previous Next