11 Replies Latest reply on Sep 7, 2010 2:33 PM by Chinar

    ORA-00470: LGWR process terminated with error

    kkrm333
      Hi,

      Often we are getting the following error on our 10g instance.

      ORA-00470: LGWR process terminated with error
      Mon Sep 6 22:47:53 2010
      PMON: terminating instance due to error 470
      Instance terminated by PMON, pid = 16986

      And checked in /var/log/messages showing the following

      kernel: Out of Memory: Killed process 16995 (oracle)

      How can I solve this?

      Thanks,
      KAvitha
        • 1. Re: ORA-00470: LGWR process terminated with error
          AliD
          If you have enough physical memory, try tuning shm parameters in Linux (shmmax,...) according to the oracle installation document for your platform.
          1 person found this helpful
          • 2. Re: ORA-00470: LGWR process terminated with error
            Srini Chavali-Oracle
            Pl post details of OS and database versions. Are there any other errors in the alert.log ?

            HTH
            Srini
            • 3. Re: ORA-00470: LGWR process terminated with error
              kkrm333
              Hi,

              I am using 10g database on Red Hat Enterprise Linux AS release 4 (Nahant Update 4).

              The alert log showing the following information.

              Mon Sep 6 23:09:45 2010
              Completed: ALTER DATABASE OPEN
              Mon Sep 6 23:17:30 2010
              Errors in file /d04/u02/oracle/admin/cidb/bdump/cidb_pmon_19969.trc:
              ORA-00470: LGWR process terminated with error
              Mon Sep 6 23:17:30 2010
              PMON: terminating instance due to error 470
              Termination issued to instance processes. Waiting for the processes to exit
              Mon Sep 6 23:17:44 2010
              Instance termination failed to kill one or more processes
              Instance terminated by PMON, pid = 19969


              And also check cidb_pmon_19969.trc file

              /d04/u02/oracle/admin/cidb/bdump/cidb_pmon_19969.trc
              Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - Production
              With the Partitioning, OLAP and Data Mining options
              ORACLE_HOME = /d04/u02/oracle/cidb
              System name: Linux
              Node name: sflxdb07.donerus.com
              Release: 2.6.9-34.ELsmp
              Version: #1 SMP Fri Feb 24 16:54:53 EST 2006
              Machine: i686
              Instance name: cidb
              Redo thread mounted by this instance: 1
              Oracle process number: 2
              Unix process pid: 19969, image: oracle@sflxdb07.donerus.com (PMON)

              *** 2010-09-06 23:17:30.676
              *** SERVICE NAME:(SYS$BACKGROUND) 2010-09-06 23:17:30.530
              *** SESSION ID:(170.1) 2010-09-06 23:17:30.530
              Background process LGWR found dead
              Oracle pid = 6
              OS pid (from detached process) = 19977
              OS pid (from process state) = 19977
              dtp = 0x2000bab8, proc = 0x8b2b13fc
              Dump of memory from 0x2000BAB8 to 0x2000BAE4
              2000BAB0 00000074 8B2B13FC [t.....+.]
              2000BAC0 00000000 00000000 5257474C 00000200 [........LGWR....]
              2000BAD0 00004E09 113B6215 00000001 2AD428DC [.N...b;......(.*]
              2000BAE0 00040001 [....]
              Dump of memory from 0x8B2B13FC to 0x8B2B19B4
              8B2B13F0 00000102 [....]
              8B2B1400 00000000 00000000 00000000 00000000 [................]
              8B2B1410 89555C60 8B82DB2C 8B3C2078 8B82CBEC [`\U.,...x <.....]
              8B2B1420 00000000 8B82CC50 8B82CC50 8B82DB20 [....P...P... ...]
              8B2B1430 00000601 8B3AE848 8B3C2078 00000006 [....H.:.x <.....]
              8B2B1440 00000000 00000000 00000000 8B4D2F48 [............H/M.]
              8B2B1450 8B4D2FE4 00000000 00000000 00000000 [./M.............]
              8B2B1460 00000000 00000000 00000000 00000000 [................]
              Repeat 2 times
              8B2B1490 00000000 00000000 00030000 00000000 [................]
              8B2B14A0 00030000 00000000 00000000 00000000 [................]
              Repeat 1 times
              8B2B17B0 00000000 00000000 00000014 4E4B4E55 [............UNKN]
              8B2B17C0 004E574F 00000000 00000000 00000000 [OWN.............]
              8B2B17D0 00000000 00000000 00000000 00000008 [................]
              8B2B17E0 37393931 00000037 00000000 00000000 [19977...........]
              8B2B17F0 00000000 00000000 00000005 6361726F [............orac]
              .......
              .......
              8B2B19B0 00000006 [....]
              error 470 detected in background process
              ORA-00470: LGWR process terminated with error
              *** 2010-09-06 23:17:44.081
              Instance termination failed to kill one or more processes
              ksuitm_check: OS PID=20017 is still alive
              *** 2010-09-06 23:17:44.083
              Dumping diagnostic information for oracle@sflxdb07.donerus.com (J002):
              OS pid = 20017
              loadavg : 63.67 22.12 10.45
              memory info: free memory = 0.00M
              swap info: free = 0.00M alloc = 0.00M total = 0.00M
              F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
              ERROR: process 20017 is not alive
              *** 2010-09-06 23:17:44.814

              Thanks,
              Kavitha
              • 4. Re: ORA-00470: LGWR process terminated with error
                kkrm333
                And also at the same time /var/log/messages showing

                Sep 6 23:17:43 sflxdb07 kernel: Out of Memory: Killed process 19977 (oracle)

                Thanks,
                Kavitha
                • 5. Re: ORA-00470: LGWR process terminated with error
                  kkrm333
                  Hi,

                  From metalink note [ID 1022782.6] and According to Alid reply also

                  we may need to change maxfiles setting.
                  Is it changing /proc/sys/kernel/shmmax file?

                  Current it is 4294967295.

                  Changing kernal parameters need a server restart?
                  What is the max limit for this parameter?

                  Thanks,
                  Kavitha

                  Edited by: kavi on Sep 7, 2010 6:03 AM
                  • 6. Re: ORA-00470: LGWR process terminated with error
                    gjilevski1
                    Hi,

                    Double check the minimal requirements

                    http://download.oracle.com/docs/cd/B19306_01/install.102/b15660/pre_install.htm#sthref264

                    You do not need a reboot for the kernel parameters to take effect.

                    Regards,
                    1 person found this helpful
                    • 7. Re: ORA-00470: LGWR process terminated with error
                      kkrm333
                      The shmmax value is correctly set in my server. It is approximately half of the
                      physical memory in bytes.
                      Now how can I solve my issue?

                      From metalink note Database Fails With Message User Terminating Instance Due To Error 472 [ID 605280.1]

                      Some 3rd party product or user or user process is killing the PMON process or other database background process.

                      as /var/log/messages showing kernel: Out of Memory: Killed process 29109 (oracle).

                      From metalink note Linux: Out-of-Memory (OOM) Killer [ID 452000.1]
                      add more extra memory or increase swap if physical memory/swap space.

                      Thanks,
                      Kavitha
                      • 8. Re: ORA-00470: LGWR process terminated with error
                        Pavan Kumar
                        Hi,
                        loadavg : 63.67 22.12 10.45
                        memory info: free memory = 0.00M
                        swap info: free = 0.00M alloc = 0.00M total = 0.00M
                        Try to check on the Server to have enough Swap space - it is nil at the particualr peak moment.

                        - Pavan Kumar N
                        • 9. Re: ORA-00470: LGWR process terminated with error
                          kkrm333
                          Presently it is

                          Mem: 8053784k total, 8009060k used, 44724k free, 18284k buffers
                          Swap: 2096472k total, 2021772k used, 74700k free, 2486120k cached

                          But at that time swap free may be 0.

                          This error is from past 10 days. No other applications were added to the server during this time.
                          Why suddenly the swap increased?
                          • 10. Re: ORA-00470: LGWR process terminated with error
                            orawiss
                            kavi wrote:
                            Presently it is

                            Mem: 8053784k total, 8009060k used, 44724k free, 18284k buffers
                            Swap: 2096472k total, 2021772k used, 74700k free, 2486120k cached

                            But at that time swap free may be 0.

                            This error is from past 10 days. No other applications were added to the server during this time.
                            Why suddenly the swap increased?
                            Run an AWR report an analyze it ;)
                            • 11. Re: ORA-00470: LGWR process terminated with error
                              Chinar
                              Refer below notes
                              *Pmon terminated instance due to LGWR termination ORA-470 [ID 431246.1]*
                              *LGWR DIES WITH ORA-27069 [ID 1011343.102]*