4 Replies Latest reply: May 8, 2013 8:55 AM by Fran RSS

    Standby DB Instance fails to start and crashes on try to connect internally

    1007768
      Hello All!

      I have PROD + Physical Standby Oracle 11.2.0.2 Enterprise on Oracle Solaris 10
      PROD is a separate Machine and works fine
      Standby instance lives on the similar machine, but shares OS with development instances.

      Once there was a delay in supplying logs to the standby for some time, and after that I restored some of them and applied them manually. During that process on the one of the archived logs I faced a crash of Standby Oracle Instance. It crashed and wrote to the alert.log that there were many damaged blocks in datafiles and started to write a "core" file with a dump of memory trying to save on disk 35G of something. Unfortunately there was not enough space, so the core writing was not completed.

      The strange things began after that.

      Now when I try to connect to standby instnace to make it somehow alive:
      +> sqlplus / as sysdba+
      it writes:
      SQLPlus: Release 11.2.0.2.0 Production on Wed May 8 16:13:39 2013
      Copyright (c) 1982, 2010, Oracle.  All rights reserved.
      but after that hangs for some time.
      During this I may see thet there are 2 server processes
      consuming 35G of memory, sleeping, and one of them tries to create a core file on disk.
      PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
      *12943 oracle 35G 34G sleep 60 0 0:00:14 0.5% oracle/1*
      *12942 oracle 35G 34G sleep 59 0 0:00:01 0.0% oracle/1*

      and after that the client writes:
      ERROR:
      ORA-12547: TNS:lost contact
      Enter user-name: sys as sysdba
      Enter password:
      ERROR:
      ORA-12547: TNS:lost contact

      Enter user-name: sys
      Enter password:
      ERROR:
      ORA-12547: TNS:lost contact
      SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQLPlus

      After all this I only have a core file and nothing written to the alert log or anywhere else.

      I must also notice that during all this all other instances of Dev oracle environments continue to live normally.
      I tried to remove pfile and spfile from the dbs directory in order to connect to the instance just as if it is all new, but the result was same error and core dump..

      Please advise me how to reanimate and start that instance.
        • 1. Re: Standby DB Instance fails to start and crashes on try to connect internally
          Fran
          welcome to otn forums.

          ORA-12547: TNS:lost contact
          post tnsnames, last 50 lines of alert.log, listener.ora, sqlnet.ora and result of $ listener status.

          Can you tnsping to that standbydatabase?
          • 2. Re: Standby DB Instance fails to start and crashes on try to connect internally
            1007768
            tnsnames.ora:

            .....
            LISTENER_IBS9 =
            (ADDRESS_LIST =
            (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
            (ADDRESS = (PROTOCOL = TCP)(HOST = 10.211.2.16)(PORT = 1521))
            )

            IBS9 = # that is the description of primary instance itself
            (DESCRIPTION =
            (ADDRESS_LIST =
            (ADDRESS = (PROTOCOL = TCP)(HOST = mosibs01)(PORT = 1521))
            )
            (CONNECT_DATA =
            (SERVER = DEDICATED)
            (SERVICE_NAME = ibs9)
            )
            )
            ........
            ........
            IBS9STB =
            (DESCRIPTION =
            (ADDRESS_LIST =
            (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
            (ADDRESS = (PROTOCOL = TCP)(HOST = 10.211.2.16)(PORT = 1521))
            )
            (CONNECT_DATA =
            (SERVER = DEDICATED)
            (SERVICE_NAME = ibs9)
            )
            )
            ......

            listener.ora

            LISTENER_EXT =
            (DESCRIPTION =
            (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
            )

            ADR_BASE_LISTENER_EXT = /oracle

            SID_LIST_LISTENER =
            (SID_LIST =
            (SID_DESC =
            (GLOBAL_DBNAME = ibs9)
            (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
            (SID_NAME = ibs9)
            )
            (SID_DESC =
            (GLOBAL_DBNAME = ibsdrv2)
            (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
            (SID_NAME = ibsdrv2)
            )
            (SID_DESC =
            (GLOBAL_DBNAME = ibspatch)
            (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
            (SID_NAME = ibspatch)
            )
            (SID_DESC =
            (GLOBAL_DBNAME = ibsdrv)
            (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
            (SID_NAME = ibsdrv)
            )
            (SID_DESC =
            (GLOBAL_DBNAME = ibsdev)
            (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
            (SID_NAME = ibsdev)
            )
            )

            LISTENER =
            (DESCRIPTION =
            (ADDRESS = (PROTOCOL = TCP)(HOST = 10.211.2.16)(PORT = 1521))
            )

            ADR_BASE_LISTENER = /oracle

            SID_LIST_LISTENER_EXT =
            (SID_LIST =
            (SID_DESC =
            (SID_NAME = PLSExtProc)
            (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
            (PROGRAM = extproc)
            (ENVS = "EXTPROC_DLLS=ANY,LD_LIBRARY_PATH=/u00/tools/xml/lib:/u00/tools:$LD_LIBRARY_PATH")
            )
            )


            sqlnet.ora - we do not have it

            *>lsnrctl status*

            LSNRCTL for Solaris: Version 11.2.0.2.0 - Production on 08-MAY-2013 17:33:07

            Copyright (c) 1991, 2010, Oracle. All rights reserved.

            Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.211.2.16)(PORT=1521)))
            STATUS of the LISTENER
            ------------------------
            Alias listener
            Version TNSLSNR for Solaris: Version 11.2.0.2.0 - Production
            Start Date 24-APR-2013 18:49:53
            Uptime 13 days 22 hr. 43 min. 15 sec
            Trace Level off
            Security ON: Local OS Authentication
            SNMP OFF
            Listener Parameter File /oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
            Listener Log File /oracle/diag/tnslsnr/mosibs03/listener/alert/log.xml
            Listening Endpoints Summary...
            (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.211.2.16)(PORT=1521)))
            Services Summary...
            Service "ibs9" has 1 instance(s).
            Instance "ibs9", status UNKNOWN, has 1 handler(s) for this service...
            Service "ibsdev" has 2 instance(s).
            Instance "ibsdev", status UNKNOWN, has 1 handler(s) for this service...
            Instance "ibsdev", status READY, has 1 handler(s) for this service...
            Service "ibsdrv" has 2 instance(s).
            Instance "ibsdrv", status UNKNOWN, has 1 handler(s) for this service...
            Instance "ibsdrv", status READY, has 1 handler(s) for this service...
            Service "ibsdrv2" has 1 instance(s).
            Instance "ibsdrv2", status UNKNOWN, has 1 handler(s) for this service...
            Service "ibspatch" has 2 instance(s).
            Instance "ibspatch", status UNKNOWN, has 1 handler(s) for this service...
            Instance "ibspatch", status READY, has 1 handler(s) for this service...
            Service "rmancat" has 1 instance(s).
            Instance "rmancat", status READY, has 1 handler(s) for this service...
            Service "rmancatXDB" has 1 instance(s).
            Instance "rmancat", status READY, has 1 handler(s) for this service...
            The command completed successfully

            tail of alert.log:

            Writing to the above trace file is disabled for now on...
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x100991E70, kspgvc()+720] [flags: 0x0, count: 1]
            Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_pr0e_7393.trc (incident=89942):
            ORA-07445: exception encountered: core dump [kspgvc()+720] [SIGSEGV] [ADDR:0x0] [PC:0x100991E70] [Address not mapped to object] []
            Incident details in: /oracle/diag/rdbms/ibs9stb/ibs9/incident/incdir_89942/ibs9_pr0e_7393_i89942.trc
            Use ADRCI or Support Workbench to package the incident.
            See Note 411.1 at My Oracle Support for error and packaging details.
            Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xC2891] [PC:0x104758740, ssexhd()+2176] [flags: 0x0, count: 2]
            Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_pr0e_7393.trc (incident=89943):
            ORA-07445: exception encountered: core dump [ssexhd()+2176] [SIGBUS] [ADDR:0xC2891] [PC:0x104758740] [Invalid address alignment] []
            ORA-07445: exception encountered: core dump [kspgvc()+720] [SIGSEGV] [ADDR:0x0] [PC:0x100991E70] [Address not mapped to object] []
            Incident details in: /oracle/diag/rdbms/ibs9stb/ibs9/incident/incdir_89943/ibs9_pr0e_7393_i89943.trc
            Use ADRCI or Support Workbench to package the incident.
            See Note 411.1 at My Oracle Support for error and packaging details.
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9AC80000000C28C5] [PC:0x1009BFBE4, kspGetInstType()+196] [flags: 0x0, count: 3]
            Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/incident/incdir_89943/ibs9_pr0e_7393_i89943.trc:
            ORA-49203: Too many recursive DDE invocations
            ORA-07445: exception encountered: core dump [kspGetInstType()+196] [SIGSEGV] [ADDR:0x9AC80000000C28C5] [PC:0x1009BFBE4] [Address not mapped to object] []
            ORA-07445: exception encountered: core dump [ssexhd()+2176] [SIGBUS] [ADDR:0xC2891] [PC:0x104758740] [Invalid address alignment] []
            ORA-07445: exception encountered: core dump [kspgvc()+720] [SIGSEGV] [ADDR:0x0] [PC:0x100991E70] [Address not mapped to object] []
            Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F0000000819] [PC:0x10099126C, kspgsp()+236] [flags: 0x2, count: 4]
            Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F0000000819] [PC:0x10099126C, kspgsp()+236] [flags: 0x2, count: 5]
            Tue Apr 23 11:10:57 2013
            Non critical error ORA-07202 caught while writing to trace file "/oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_lgwr_7123.trc"
            Error message: ORA-07202: sltln: invalid parameter to sltln.

            Writing to the above trace file is disabled for now on...
            Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F0000000021] [PC:0x100991C88, kspgvc()+232] [flags: 0x0, count: 1]
            Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_lgwr_7123.trc (incident=89702):
            ORA-07445: exception encountered: core dump [kspgvc()+232] [SIGBUS] [ADDR:0xB38F0000000021] [PC:0x100991C88] [Invalid address alignment] []
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xA] [PC:0x10098F6EC, kspgip()+236] [flags: 0x2, count: 2]
            Tue Apr 23 11:10:58 2013
            Non critical error ORA-07202 caught while writing to trace file "/oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_smon_7127.trc"
            Error message: ORA-07202: sltln: invalid parameter to sltln.

            Writing to the above trace file is disabled for now on...
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xFFFFFFFF0000000A] [PC:0x10098F6EC, kspgip()+236] [flags: 0x0, count: 1]
            Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_smon_7127.trc (incident=89718):
            ORA-07445: exception encountered: core dump [kspgip()+236] [SIGSEGV] [ADDR:0xFFFFFFFF0000000A] [PC:0x10098F6EC] [Address not mapped to object] []
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xA] [PC:0x10098F6EC, kspgip()+236] [flags: 0x2, count: 2]
            Tue Apr 23 11:10:58 2013
            Non critical error ORA-07202 caught while writing to trace file "/oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_mmnl_7133.trc"
            Error message: ORA-07202: sltln: invalid parameter to sltln.

            Writing to the above trace file is disabled for now on...
            Exception [type: SIGBUS, Invalid address alignment] [ADDR:0x6B676C7365736874] [PC:0x100991E70, kspgvc()+720] [flags: 0x0, count: 1]
            Tue Apr 23 11:10:58 2013
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x2073697A6520746F] [PC:0xFFFFFFFF7A93C6F8, strlen()+24] [flags: 0x0, count: 1]
            Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_mmnl_7133.trc (incident=89742):
            ORA-07445: exception encountered: core dump [kspgvc()+720] [SIGBUS] [ADDR:0x6B676C7365736874] [PC:0x100991E70] [Invalid address alignment] []
            Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F000000002B] [PC:0x10098F6EC, kspgip()+236] [flags: 0x2, count: 2]
            Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_mman_7117.trc (incident=89678):
            ORA-07445: exception encountered: core dump [strlen()+24] [SIGSEGV] [ADDR:0x2073697A6520746F] [PC:0xFFFFFFFF7A93C6F8] [Address not mapped to object] []
            Incident details in: /oracle/diag/rdbms/ibs9stb/ibs9/incident/incdir_89678/ibs9_mman_7117_i89678.trc
            Use ADRCI or Support Workbench to package the incident.
            See Note 411.1 at My Oracle Support for error and packaging details.
            Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xC2891] [PC:0x104758740, ssexhd()+2176] [flags: 0x0, count: 2]
            Tue Apr 23 11:11:00 2013
            (ospid: 7148): terminating the instance due to error -557318144
            Tue Apr 23 11:11:00 2013
            Non critical error ORA-07202 caught while writing to trace file "/oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_dbrm_7113.trc"
            Error message: ORA-07202: sltln: invalid parameter to sltln.

            Writing to the above trace file is disabled for now on...
            Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F0000000021] [PC:0x100991C88, kspgvc()+232] [flags: 0x0, count: 1]
            Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_dbrm_7113.trc (incident=89662):
            ORA-07445: exception encountered: core dump [kspgvc()+232] [SIGBUS] [ADDR:0xB38F0000000021] [PC:0x100991C88] [Invalid address alignment] []
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xA] [PC:0x10098F6EC, kspgip()+236] [flags: 0x2, count: 2]
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9AC80000000C28C5] [PC:0x1009BFBE4, kspGetInstType()+196] [flags: 0x2, count: 3]
            Tue Apr 23 11:11:18 2013
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9AC80000000C28C5] [PC:0x1009BFBE4, kspGetInstType()+196] [flags: 0x2, count: 3]
            Tue Apr 23 11:11:40 2013
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10099125C, kspgsp()+220] [flags: 0x2, count: 4]
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10099125C, kspgsp()+220] [flags: 0x2, count: 5]
            Tue Apr 23 11:12:18 2013
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10099126C, kspgsp()+236] [flags: 0x2, count: 4]
            Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10099126C, kspgsp()+236] [flags: 0x2, count: 5]
            • 3. Re: Standby DB Instance fails to start and crashes on try to connect internally
              sb92075
              open Service Request & resolve source of all the ORA-07445 errors.
              • 4. Re: Standby DB Instance fails to start and crashes on try to connect internally
                Fran
                like sb told you must check and open a SR in MOS (My oracle support).

                Also check this thread for the error "ORA-07202 sltln: invalid parameter to sltln"

                ORA-07202 sltln: invalid parameter to sltln.using