8 Replies Latest reply on May 25, 2017 9:35 PM by mdtaylor

    oracle account only allow around 1500 processes????

    ef8454

      we have ORACLE/EBS databases on DELL R910 server and OS is Redhat 5.10.

       

      The problem I have is two months ago if ORACLE processes over 1600 (may be a 1625??) It will not allow new connection and give us error messages:

      ORA-27300: OS system dependent operation:fork failed with status: 11

      ORA-27301: OS failure message: Resource temporarily unavailable

      ORA-27302: failure occurred at: skgpspawn3

       

      But after two months (we did not reboot server), it only allow ORACLE account have around 1510 processes.  Otherwise it pop-up error messages.

       

      we have been work with ORACLE support and search on Internet still can not find solution.

       

      Some of my server info

       

      $ ulimit -u

      131072

       

      $more /etc/security/limits.conf (partial)

      # following modify base on ERP requirement

       

      *               soft    nproc          131072

      *               hard    nproc          131072

      *               soft    nofile         4096

      *               hard    nofile         65536

      oracle          soft    nproc          131072

      oracle          hard    nproc          131072

      oracle          soft    nofile         8192

      oracle          hard    nofile         131072

       

      $more /etc/sysctl.conf  (partial)

      ## for fork processes fail and need increase for more processes.  Look document 1546393.1

      kernel.pid_max = 139264

       

      any ideal????

        • 1. Re: oracle account only allow around 1500 processes????
          mdtaylor

          What are your processes and sessions database parameters?  You may need to increase them.

           

          processes       = 2000# Max. no. of users x 2
          sessions        = 4000   # 2 X processes

           

          Also review the following notes for kernel parameters:

           

          Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for Linux x86-64 (Doc ID 761566.1)

          R12 E-Business Suite Database Alert Log File Contains Errors 'ORA-27300: OS system dependent operation:fork failed', 'ORA-27301: OS failure message: Resource temporarily unavailable' And 'ORA-27302: failure occurred at: skgpspawn3' (Doc ID 2220325.1)

           

          Regards,

          Michael

          • 2. Re: oracle account only allow around 1500 processes????
            ef8454

            this is not relate to database "init.ora" file setup.  When total oracle processes over 1500 (something like that) all databases will pop-up for new connection error messages.   Not just one database.

            • 3. Re: oracle account only allow around 1500 processes????
              mdtaylor

              If you have multiple databases running from the same account plan to increase required kernel parameters and unlimi compared to what is documented in the installation guides. Have you run the RDA from the linked MOS note?

              • 4. Re: oracle account only allow around 1500 processes????
                mdtaylor

                Also type ulimit -a as oracle.  Sometimes preinstall limits files in /etc/security/limits.d take precedence over /etc/security/limits.conf.

                • 5. Re: oracle account only allow around 1500 processes????
                  ef8454

                  my /etc/security/limit.d is empty (No file).

                   

                  $ ulimit -a

                  core file size          (blocks, -c) 0

                  data seg size           (kbytes, -d) unlimited

                  scheduling priority             (-e) 0

                  file size               (blocks, -f) unlimited

                  pending signals                 (-i) 8244499

                  max locked memory       (kbytes, -l) 32

                  max memory size         (kbytes, -m) unlimited

                  open files                      (-n) 131072

                  pipe size            (512 bytes, -p) 8

                  POSIX message queues     (bytes, -q) 819200

                  real-time priority              (-r) 0

                  stack size              (kbytes, -s) 10240

                  cpu time               (seconds, -t) unlimited

                  max user processes              (-u) 131072

                  virtual memory          (kbytes, -v) unlimited

                  file locks                      (-x) unlimited

                  • 6. Re: oracle account only allow around 1500 processes????
                    mdtaylor

                    I would also set soft nofile to 131072.

                     

                    Here is what I am using:

                     

                    *    hard    nofile    65536

                    *    soft    nofile    65536

                    *    hard    nproc    65536

                    *    soft    nproc    65536

                    *    hard    stack    262144

                    *    soft    stack    262144

                    *    hard    memlock    60000000

                    *    soft    memlock    60000000

                     

                    Please provide all /etc/sysctl.conf entries added for Oracle database, the total number of databases running under Oracle account, and output of free -m

                    • 7. Re: oracle account only allow around 1500 processes????
                      ef8454

                      $ free -m

                                   total       used       free     shared    buffers     cached

                      Mem:       1030565    1028787       1778          0       2000    1001221

                      -/+ buffers/cache:      25565    1004999

                      Swap:       300002       4336     295666

                       

                      $more /etc/sysctl.conf

                      # Controls IP packet forwarding

                      net.ipv4.ip_forward = 0

                       

                      # Controls source route verification

                      net.ipv4.conf.default.rp_filter = 1

                       

                      # Do not accept source routing

                      net.ipv4.conf.default.accept_source_route = 0

                       

                      # Controls the System Request debugging functionality of the kernel

                      kernel.sysrq = 0

                       

                      # Controls whether core dumps will append the PID to the core filename

                      # Useful for debugging multi-threaded applications

                      kernel.core_uses_pid = 1

                       

                      # Controls the use of TCP syncookies

                      net.ipv4.tcp_syncookies = 1

                       

                      # Controls the default maxmimum size of a mesage queue

                      kernel.msgmnb = 65536

                       

                      # Controls the maximum size of a message, in bytes

                      kernel.msgmax = 65536

                       

                      # Controls the maximum shared segment size, in bytes

                      ### kernel.shmmax = 68719476736

                      kernel.shmmax = 15032385536

                       

                      # Controls the maximum number of shared memory segments, in pages

                      ### kernel.shmall = 4294967296

                      kernel.shmall = 31457280

                       

                      # Oracle 9iAS semaphores, SEMMSL, SEMMNS, SEMOPM, SEMMNI

                      ### kernel.sem = 250 32000 128 128

                       

                      ## for large semaphore use following

                      kernel.sem = 256 256000 128 1024

                       

                      ## for fork processes fail and need increase for more processes.  Look document 1546393.1

                      kernel.pid_max = 139264

                       

                      # Oracle 9iAS set max shared memory

                      ### kernel.shmmax = 2147483648

                      #Oracle 9iAS set max shared memory segments per entire system

                      kernel.shmmni = 4096

                      # SHMALL control maximum amount of memory that ALL shared memory can take

                      # 4194304 is page number it is 16GB total share memory can use under 32 GB RAM

                      ### kernel.shmall = 4194304

                      #Oracle 9iAS set available port range

                      ### net.ipv4.ip_local_port_range = 1024 65000

                      net.ipv4.ip_local_port_range = 9000 65500

                      #Oracle 9iAS set max file handles

                      fs.file-max = 6815744

                      #Oracle 9iAS set async io handles

                      ### fs.aio-max-size = 1048576           ### not require by kernel 2.6

                      ### following parameter and value suggest by oracle support

                      fs.aio-max-nr = 3145728

                       

                      net.core.rmem_default=4194304

                      net.core.wmem_default=262144

                      net.core.rmem_max=4194304

                      ### net.core.wmem_max=262144

                      net.core.wmem_max=1048576

                      net.ipv4.tcp_wmem=262144 262144 262144

                      net.ipv4.tcp_rmem=4194304 4194304 4194304

                      • 8. Re: oracle account only allow around 1500 processes????
                        mdtaylor

                        It appears you have 1024MB of RAM and a kernel.shmmax of 15GB.  Is that intentional?  Also kernel.shmall is 4X smaller than what I use on a DB server with 512GB of RAM.

                         

                        Why are you not using Linux HugePages on a server that large?

                         

                        What version of the DB are you running and how many database are on this server running under the Oracle account?  Also is this a database only node or database plus applications?