3 Replies Latest reply: Sep 23, 2008 9:21 AM by 7537 RSS

    ORA-27102 HP-UX 10.2.0.2

    7537
      We are having a problem with out of memory errors showing up in our alert log. I have found messages about getting it during installs, shutting down the DB but nothing about it showing up in the alert log during run time. It created big problems last weekend because it caused the archive logs to have problems. Here is what I am seeing in the alert log. I could really use some help on this. I was thinking it was an application, but after what I have found on the web, I am not sure it is now.

      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
      ORA-12012: error on auto execute of job 162389
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      Sun Sep 21 15:09:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 520 bytes (Typecheck,frodef : qcpitnm)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      bytes (,)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      Sun Sep 21 15:09:57 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10439.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Additional information: 2506208
      Sun Sep 21 15:10:22 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10439.trc:
      ORA-04030: out of process memory when trying to allocate 753136 bytes (pga heap,kco buffer)
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Additional information: 2506208
      Sun Sep 21 15:10:22 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10439.trc:
      ORA-00081: address range [0x60000000000A8210, 0x60000000000A8214) is not readable
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Additional information: 2506208
      Sun Sep 21 15:10:30 2008
      Process startup failed, error stack:
      Sun Sep 21 15:10:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 11
      ORA-27301: OS failure message: Resource temporarily unavailable
      ORA-27302: failure occurred at: skgpspawn5
      Sun Sep 21 15:10:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
      ORA-12012: error on auto execute of job 162389
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      Sun Sep 21 15:10:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 520 bytes (Typecheck,frodef : qcpitnm)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      bytes (,)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 8224 bytes (sort subheap,sort key)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      Sun Sep 21 15:10:30 2008
      Process J002 died, see its trace file
      Sun Sep 21 15:10:30 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:10:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:11:05 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_4575.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Additional information: -444973056
      Sun Sep 21 15:11:08 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_4575.trc:
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Additional information: -444973056
      Sun Sep 21 15:11:17 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_7438.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 262144
      Additional information: -444973056
      Sun Sep 21 15:11:26 2008
      Process startup failed, error stack:
      Sun Sep 21 15:11:26 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 12
      ORA-27301: OS failure message: Not enough space
      ORA-27302: failure occurred at: skgpspawn3
      Sun Sep 21 15:11:27 2008
      Process J002 died, see its trace file
      Sun Sep 21 15:11:27 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:11:27 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:11:27 2008
      Process startup failed, error stack:
      Sun Sep 21 15:11:27 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 12
      ORA-27301: OS failure message: Not enough space
      ORA-27302: failure occurred at: skgpspawn3
      Sun Sep 21 15:11:28 2008
      Process J002 died, see its trace file
      Sun Sep 21 15:11:28 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:11:28 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:11:32 2008
      Process startup failed, error stack:
      Sun Sep 21 15:11:32 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 12
      ORA-27301: OS failure message: Not enough space
      ORA-27302: failure occurred at: skgpspawn3
      Sun Sep 21 15:11:32 2008
      Process J002 died, see its trace file
      Sun Sep 21 15:11:32 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:11:32 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:11:32 2008
      Process startup failed, error stack:
      Sun Sep 21 15:11:32 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 12
      ORA-27301: OS failure message: Not enough space
      ORA-27302: failure occurred at: skgpspawn3
      Sun Sep 21 15:11:32 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
      ORA-12012: error on auto execute of job 162389
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 123416 bytes (QERGH hash-agg,kllcqas:kllsltba)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      Sun Sep 21 15:11:32 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 520 bytes (Typecheck,frodef : qcpitnm)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 123416 bytes (QERGH hash-agg,kllcqas:kllsltba)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      bytes (,)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 123416 bytes (QERGH hash-agg,kllcqas:kllsltba)
      ORA-06512: at "LPM.LPM_DATAMAINT", line 3961
      ORA-06512: at line 4
      bytes (,)
      Sun Sep 21 15:11:32 2008
      Process J002 died, see its trace file
      Sun Sep 21 15:11:32 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:11:32 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:11:32 2008
      Process startup failed, error stack:
      Sun Sep 21 15:11:32 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 12
      ORA-27301: OS failure message: Not enough space
      ORA-27302: failure occurred at: skgpspawn3
      Sun Sep 21 15:11:33 2008
      Process J002 died, see its trace file
      Sun Sep 21 15:11:33 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:11:33 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:11:42 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_4575.trc:
      ORA-04030: out of process memory when trying to allocate 753136 bytes (pga heap,kco buffer)
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Additional information: -444973056
      Sun Sep 21 15:11:42 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_4575.trc:
      ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Additional information: -444973056
      Sun Sep 21 15:11:43 2008
      Thread 1 advanced to log sequence 34405
      Current log# 3 seq# 34405 mem# 0: +DATA/pcen/onlinelog/group_3.264.636531609
      Current log# 3 seq# 34405 mem# 1: +FLASH/pcen/onlinelog/group_3.256.636531625
      Sun Sep 21 15:11:43 2008
      ERROR: Failed to connect with connect string:(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/db/oraprod01/app/oracle/product/10.2.0/db_1/bin/oracle)(ARGV0=oracle+ASM)(ENVS='ORACLE_HOME=/db/oraprod01/app/oracle/product/10.2.0/db_1,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
      Sun Sep 21 15:11:43 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc1_25220.trc:
      ORA-19504: failed to create file "+FLASH/pcen/1_34404_592672654.dbf"
      ORA-17502: ksfdcre:4 Failed to create file +FLASH/pcen/1_34404_592672654.dbf
      ORA-15055: unable to connect to ASM instance
      ORA-12549: TNS:operating system resource quota exceeded
      ARC1: Error 19504 Creating archive log file to '+FLASH/pcen/1_34404_592672654.dbf'
      ARC1: Failed to archive thread 1 sequence 34404 (19504)
      ARCH: Archival stopped, error occurred. Will continue retrying
      Sun Sep 21 15:11:43 2008
      ORACLE Instance pcen - Archival Error
      Sun Sep 21 15:11:43 2008
      ORA-16038: log 2 sequence# 34404 cannot be archived
      ORA-19504: failed to create file ""
      ORA-00312: online log 2 thread 1: '+DATA/pcen/onlinelog/group_2.263.636531579'
      ORA-00312: online log 2 thread 1: '+FLASH/pcen/onlinelog/group_2.257.636531595'
      Sun Sep 21 15:11:43 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc1_25220.trc:
      ORA-16038: log 2 sequence# 34404 cannot be archived
      ORA-19504: failed to create file ""
      ORA-00312: online log 2 thread 1: '+DATA/pcen/onlinelog/group_2.263.636531579'
      ORA-00312: online log 2 thread 1: '+FLASH/pcen/onlinelog/group_2.257.636531595'
      Sun Sep 21 15:11:44 2008
      LNS: Standby redo logfile selected for thread 1 sequence 34405 for destination LOG_ARCHIVE_DEST_2
      Sun Sep 21 15:11:51 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_3798.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 131072
      Sun Sep 21 15:11:51 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_3798.trc:
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 131072
      Sun Sep 21 15:12:00 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 393216
      Additional information: 2506208
      Sun Sep 21 15:12:01 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j001_14174.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 131072
      Additional information: 2506208
      Sun Sep 21 15:12:02 2008
      ARC0: Archiving not possible: No primary destinations
      ARC0: Failed to archive thread 1 sequence 34404 (4)
      ARCH: Archival stopped, error occurred. Will continue retrying
      Sun Sep 21 15:12:02 2008
      ORACLE Instance pcen - Archival Error
      Sun Sep 21 15:12:02 2008
      ORA-16014: log 2 sequence# 34404 not archived, no available destinations
      ORA-00312: online log 2 thread 1: '+DATA/pcen/onlinelog/group_2.263.636531579'
      ORA-00312: online log 2 thread 1: '+FLASH/pcen/onlinelog/group_2.257.636531595'
      Sun Sep 21 15:12:02 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc0_25200.trc:
      ORA-16014: log 2 sequence# 34404 not archived, no available destinations
      ORA-00312: online log 2 thread 1: '+DATA/pcen/onlinelog/group_2.263.636531579'
      ORA-00312: online log 2 thread 1: '+FLASH/pcen/onlinelog/group_2.257.636531595'
      Sun Sep 21 15:12:07 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_6704.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 524288
      Additional information: -444973056
      Sun Sep 21 15:12:09 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_6706.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 131072
      Additional information: -444973056
      Sun Sep 21 15:12:11 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10066.trc:
      ORA-12012: error on auto execute of job 157369
      ORA-04030: out of process memory when trying to allocate bytes (,)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 589856 bytes (pga heap,drqlist_kfiopg)
      bytes (,)
      Sun Sep 21 15:12:24 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_3798.trc:
      ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 131072
      Sun Sep 21 15:12:26 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j001_14174.trc:
      ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 131072
      Additional information: 2506208
      Sun Sep 21 15:12:26 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j003_7035.trc:
      ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 393216
      Additional information: 2506208
      Sun Sep 21 15:12:30 2008
      Process startup failed, error stack:
      Sun Sep 21 15:12:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 12
      ORA-27301: OS failure message: Not enough space
      ORA-27302: failure occurred at: skgpspawn3
      Sun Sep 21 15:12:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10066.trc:
      ORA-12012: error on auto execute of job 162389
      ORA-04030: out of process memory when trying to allocate bytes (,)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 589856 bytes (pga heap,drqlist_kfiopg)
      bytes (,)
      Sun Sep 21 15:12:30 2008
      Process J001 died, see its trace file
      Sun Sep 21 15:12:30 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:12:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:12:30 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_10066.trc:
      ORA-12012: error on auto execute of job 140817
      ORA-04030: out of process memory when trying to allocate bytes (,)
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 589856 bytes (pga heap,drqlist_kfiopg)
      bytes (,)
      Sun Sep 21 15:13:03 2008
      ARC1: Archiving not possible: No primary destinations
      ARC1: Failed to archive thread 1 sequence 34404 (4)
      Sun Sep 21 15:13:22 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_3798.trc:
      ORA-04030: out of process memory when trying to allocate 753136 bytes (pga heap,kco buffer)
      ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 131072
      Sun Sep 21 15:14:03 2008
      ARC0: Archiving not possible: No primary destinations
      ARC0: Failed to archive thread 1 sequence 34404 (4)
      Sun Sep 21 15:15:03 2008
      ARC1: Archiving not possible: No primary destinations
      ARC1: Failed to archive thread 1 sequence 34404 (4)
      Sun Sep 21 15:15:52 2008
      ARC1: Archiving not possible: No primary destinations
      ARC1: Failed to archive thread 1 sequence 34404 (4)
      Sun Sep 21 15:16:04 2008
      ARC0: Archiving not possible: No primary destinations
      ARC0: Failed to archive thread 1 sequence 34404 (4)
      Sun Sep 21 15:17:47 2008
      Archiver process freed from errors. No longer stopped
      Sun Sep 21 15:24:01 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12565.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 196608
      Additional information: -444973056
      Sun Sep 21 15:24:08 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11334.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 589824
      Additional information: -444973056
      Sun Sep 21 15:24:53 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11926.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 786432
      Sun Sep 21 15:31:33 2008
      Process startup failed, error stack:
      Sun Sep 21 15:31:33 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 11
      ORA-27301: OS failure message: Resource temporarily unavailable
      ORA-27302: failure occurred at: skgpspawn5
      Sun Sep 21 15:31:34 2008
      Process J003 died, see its trace file
      Sun Sep 21 15:31:34 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:31:34 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:31:39 2008
      Process startup failed, error stack:
      Sun Sep 21 15:31:39 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_psp0_21007.trc:
      ORA-27300: OS system dependent operation:fork failed with status: 11
      ORA-27301: OS failure message: Resource temporarily unavailable
      ORA-27302: failure occurred at: skgpspawn5
      Sun Sep 21 15:31:40 2008
      Process J003 died, see its trace file
      Sun Sep 21 15:31:40 2008
      kkjcre1p: unable to spawn jobq slave process
      Sun Sep 21 15:31:40 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_cjq0_21084.trc:

      Sun Sep 21 15:31:59 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12495.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 917504
      Additional information: -444973056
      Sun Sep 21 15:31:59 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_21389.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 1441792
      Sun Sep 21 15:32:00 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j000_13595.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 458752
      Additional information: 2506208
      Sun Sep 21 15:32:01 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j002_17863.trc:
      ORA-12012: error on auto execute of job 11
      ORA-04030: out of process memory when trying to allocate 589856 bytes (pga heap,drqlist_kfiopg)
      Sun Sep 21 15:32:01 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j001_17610.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Sun Sep 21 15:32:11 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11248.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 589824
      Additional information: -444973056
      Sun Sep 21 15:32:11 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11248.trc:
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 589824
      Additional information: -444973056
      Sun Sep 21 15:32:26 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j000_13595.trc:
      ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 458752
      Additional information: 2506208
      Sun Sep 21 15:32:27 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j001_17610.trc:
      ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 655360
      Sun Sep 21 15:32:28 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_21389.trc:
      ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 1441792
      Sun Sep 21 15:32:45 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_11248.trc:
      ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 589824
      Additional information: -444973056
      Sun Sep 21 15:33:03 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17422.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 262144
      Additional information: 21647360
      Sun Sep 21 15:33:04 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_j000_17989.trc:
      ORA-12012: error on auto execute of job 166160
      ORA-04030: out of process memory when trying to allocate ORA-04030: out of process memory when trying to allocate 123416 bytes (QERHJ hash-joi,kllcqas:kllsltba)
      ORA-06512: at "TERMINAL.LPM_TCH_EXTRACT", line 1424
      ORA-06512: at line 5
      bytes (,)
      Sun Sep 21 15:33:21 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17928.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 1179648
      Additional information: -444973056
      Sun Sep 21 15:33:21 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17928.trc:
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 1179648
      Additional information: -444973056
      Sun Sep 21 15:33:22 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_21389.trc:
      ORA-00081: address range [0x60000000000A81E0, 0x60000000000A81E4) is not readable
      ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 1441792
      Sun Sep 21 15:33:51 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17928.trc:
      ORA-07445: exception encountered: core dump [kghalp()+496] [SIGSEGV] [Invalid permissions for mapped object] [0x00000003B] [] []
      ORA-07445: exception encountered: core dump [kghalf()+624] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 1179648
      Additional information: -444973056
      Sun Sep 21 15:34:08 2008
      Thread 1 advanced to log sequence 34406
      Current log# 1 seq# 34406 mem# 0: +DATA/pcen/onlinelog/group_1.262.636531547
      Current log# 1 seq# 34406 mem# 1: +FLASH/pcen/onlinelog/group_1.258.636531565
      Sun Sep 21 15:34:08 2008
      ERROR: Failed to connect with connect string:(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/db/oraprod01/app/oracle/product/10.2.0/db_1/bin/oracle)(ARGV0=oracle+ASM)(ENVS='ORACLE_HOME=/db/oraprod01/app/oracle/product/10.2.0/db_1,ORACLE_SID=+ASM')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(enable=setuser))
      Sun Sep 21 15:34:08 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc0_25200.trc:
      ORA-19504: failed to create file "+FLASH/pcen/1_34405_592672654.dbf"
      ORA-17502: ksfdcre:4 Failed to create file +FLASH/pcen/1_34405_592672654.dbf
      ORA-15055: unable to connect to ASM instance
      ORA-12549: TNS:operating system resource quota exceeded
      ARC0: Error 19504 Creating archive log file to '+FLASH/pcen/1_34405_592672654.dbf'
      ARC0: Failed to archive thread 1 sequence 34405 (19504)
      ARCH: Archival stopped, error occurred. Will continue retrying
      Sun Sep 21 15:34:08 2008
      ORACLE Instance pcen - Archival Error
      Sun Sep 21 15:34:08 2008
      ORA-16038: log 3 sequence# 34405 cannot be archived
      ORA-19504: failed to create file ""
      ORA-00312: online log 3 thread 1: '+DATA/pcen/onlinelog/group_3.264.636531609'
      ORA-00312: online log 3 thread 1: '+FLASH/pcen/onlinelog/group_3.256.636531625'
      Sun Sep 21 15:34:08 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc0_25200.trc:
      ORA-16038: log 3 sequence# 34405 cannot be archived
      ORA-19504: failed to create file ""
      ORA-00312: online log 3 thread 1: '+DATA/pcen/onlinelog/group_3.264.636531609'
      ORA-00312: online log 3 thread 1: '+FLASH/pcen/onlinelog/group_3.256.636531625'
      Sun Sep 21 15:34:09 2008
      ARC1: Archiving not possible: No primary destinations
      ARC1: Failed to archive thread 1 sequence 34405 (4)
      ARCH: Archival stopped, error occurred. Will continue retrying
      Sun Sep 21 15:34:09 2008
      ORACLE Instance pcen - Archival Error
      Sun Sep 21 15:34:09 2008
      ORA-16014: log 3 sequence# 34405 not archived, no available destinations
      ORA-00312: online log 3 thread 1: '+DATA/pcen/onlinelog/group_3.264.636531609'
      ORA-00312: online log 3 thread 1: '+FLASH/pcen/onlinelog/group_3.256.636531625'
      Sun Sep 21 15:34:09 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/bdump/pcen_arc1_25220.trc:
      ORA-16014: log 3 sequence# 34405 not archived, no available destinations
      ORA-00312: online log 3 thread 1: '+DATA/pcen/onlinelog/group_3.264.636531609'
      ORA-00312: online log 3 thread 1: '+FLASH/pcen/onlinelog/group_3.256.636531625'
      Sun Sep 21 15:34:10 2008
      LNS: Standby redo logfile selected for thread 1 sequence 34406 for destination LOG_ARCHIVE_DEST_2
      Sun Sep 21 15:35:03 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_17901.trc:
      ORA-27102: out of memory
      HPUX-ia64 Error: 12: Not enough space
      Additional information: 108
      Additional information: 327680
      Additional information: -1346033664
      Sun Sep 21 15:35:10 2008
      ARC0: Archiving not possible: No primary destinations
      ARC0: Failed to archive thread 1 sequence 34405 (4)
      Sun Sep 21 15:35:55 2008
      ARC1: Archiving not possible: No primary destinations
      ARC1: Failed to archive thread 1 sequence 34405 (4)
      Sun Sep 21 15:36:10 2008
      ARC1: Archiving not possible: No primary destinations
      ARC1: Failed to archive thread 1 sequence 34405 (4)
      Sun Sep 21 15:36:41 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12750.trc:
      ORA-00600: internal error code, arguments: [12240], [], [], [], [], [], [], []
      Sun Sep 21 15:37:02 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12750.trc:
      ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions for mapped object] [0x0000006EC] [] []
      ORA-00600: internal error code, arguments: [12240], [], [], [], [], [], [], []
      Sun Sep 21 15:37:10 2008
      ARC0: Archiving not possible: No primary destinations
      ARC0: Failed to archive thread 1 sequence 34405 (4)
      Sun Sep 21 15:37:35 2008
      Errors in file /db/oraprod01/app/oracle/admin/pcen/udump/pcen_ora_12750.trc:
      ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions for mapped object] [0x0000006EC] [] []
      ORA-07445: exception encountered: core dump [kcsgrsn()+400] [SIGSEGV] [Invalid permissions for mapped object] [0x0000006EC] [] []
      ORA-00600: internal error code, arguments: [12240], [], [], [], [], [], [], []
      Sun Sep 21 15:38:11 2008
      ARC1: Archiving not possible: No primary destinations
      ARC1: Failed to archive thread 1 sequence 34405 (4)
        • 1. Re: ORA-27102 HP-UX 10.2.0.2
          Anand...
          Hi....

          Its better to open SR with oracle support.Well going through metalink for errors..most of it is related to bugs which is fixed in 10.2.0.4 version.So, you will need to migrate to 10.2.0.4.But,open aSR with oracle.


          Regards,
          Anand
          • 2. Re: ORA-27102 HP-UX 10.2.0.2
            247514
            Look into error
            ORA-04030 and ORA-27102

            It looks like your OS out of memory to process your Job request.

            First check how much memory left on OS and check how many sessions you have.

            also check metalink doc

            Diagnosing and Resolving ORA-4030 errors
            Doc ID: NOTE:233869.1
            • 3. Re: ORA-27102 HP-UX 10.2.0.2
              7537
              One of the things we see is the swap on the machine keeps creaping up like it does not get released. We end up with zombies on the machine also. I found a SR that sounded like it could be our problem but it is a windows and we are HP-UX Unix. We have errors in the psp0 type trace files:

              *** 2008-09-15 16:00:30.117
              Process startup failed, error stack:
              ORA-27300: OS system dependent operation:fork failed with status: 11
              ORA-27301: OS failure message: Resource temporarily unavailable
              ORA-27302: failure occurred at: skgpspawn5

              I am pretty sure this is where the zombies come from because the number or entries in the file matched the number of zombies.

              Has anyone see this type of behavior on a Unix 10.2.0.2?