This discussion is archived
4 Replies Latest reply: May 8, 2013 6:55 AM by Fran RSS

Standby DB Instance fails to start and crashes on try to connect internally

1007768 Newbie
Currently Being Moderated
Hello All!

I have PROD + Physical Standby Oracle 11.2.0.2 Enterprise on Oracle Solaris 10
PROD is a separate Machine and works fine
Standby instance lives on the similar machine, but shares OS with development instances.

Once there was a delay in supplying logs to the standby for some time, and after that I restored some of them and applied them manually. During that process on the one of the archived logs I faced a crash of Standby Oracle Instance. It crashed and wrote to the alert.log that there were many damaged blocks in datafiles and started to write a "core" file with a dump of memory trying to save on disk 35G of something. Unfortunately there was not enough space, so the core writing was not completed.

The strange things began after that.

Now when I try to connect to standby instnace to make it somehow alive:
+> sqlplus / as sysdba+
it writes:
SQLPlus: Release 11.2.0.2.0 Production on Wed May 8 16:13:39 2013
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
but after that hangs for some time.
During this I may see thet there are 2 server processes
consuming 35G of memory, sleeping, and one of them tries to create a core file on disk.
PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
*12943 oracle 35G 34G sleep 60 0 0:00:14 0.5% oracle/1*
*12942 oracle 35G 34G sleep 59 0 0:00:01 0.0% oracle/1*

and after that the client writes:
ERROR:
ORA-12547: TNS:lost contact
Enter user-name: sys as sysdba
Enter password:
ERROR:
ORA-12547: TNS:lost contact

Enter user-name: sys
Enter password:
ERROR:
ORA-12547: TNS:lost contact
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQLPlus

After all this I only have a core file and nothing written to the alert log or anywhere else.

I must also notice that during all this all other instances of Dev oracle environments continue to live normally.
I tried to remove pfile and spfile from the dbs directory in order to connect to the instance just as if it is all new, but the result was same error and core dump..

Please advise me how to reanimate and start that instance.
  • 1. Re: Standby DB Instance fails to start and crashes on try to connect internally
    Fran Guru
    Currently Being Moderated
    welcome to otn forums.

    ORA-12547: TNS:lost contact
    post tnsnames, last 50 lines of alert.log, listener.ora, sqlnet.ora and result of $ listener status.

    Can you tnsping to that standbydatabase?
  • 2. Re: Standby DB Instance fails to start and crashes on try to connect internally
    1007768 Newbie
    Currently Being Moderated
    tnsnames.ora:

    .....
    LISTENER_IBS9 =
    (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.211.2.16)(PORT = 1521))
    )

    IBS9 = # that is the description of primary instance itself
    (DESCRIPTION =
    (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = TCP)(HOST = mosibs01)(PORT = 1521))
    )
    (CONNECT_DATA =
    (SERVER = DEDICATED)
    (SERVICE_NAME = ibs9)
    )
    )
    ........
    ........
    IBS9STB =
    (DESCRIPTION =
    (ADDRESS_LIST =
    (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.211.2.16)(PORT = 1521))
    )
    (CONNECT_DATA =
    (SERVER = DEDICATED)
    (SERVICE_NAME = ibs9)
    )
    )
    ......

    listener.ora

    LISTENER_EXT =
    (DESCRIPTION =
    (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
    )

    ADR_BASE_LISTENER_EXT = /oracle

    SID_LIST_LISTENER =
    (SID_LIST =
    (SID_DESC =
    (GLOBAL_DBNAME = ibs9)
    (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
    (SID_NAME = ibs9)
    )
    (SID_DESC =
    (GLOBAL_DBNAME = ibsdrv2)
    (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
    (SID_NAME = ibsdrv2)
    )
    (SID_DESC =
    (GLOBAL_DBNAME = ibspatch)
    (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
    (SID_NAME = ibspatch)
    )
    (SID_DESC =
    (GLOBAL_DBNAME = ibsdrv)
    (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
    (SID_NAME = ibsdrv)
    )
    (SID_DESC =
    (GLOBAL_DBNAME = ibsdev)
    (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
    (SID_NAME = ibsdev)
    )
    )

    LISTENER =
    (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 10.211.2.16)(PORT = 1521))
    )

    ADR_BASE_LISTENER = /oracle

    SID_LIST_LISTENER_EXT =
    (SID_LIST =
    (SID_DESC =
    (SID_NAME = PLSExtProc)
    (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)
    (PROGRAM = extproc)
    (ENVS = "EXTPROC_DLLS=ANY,LD_LIBRARY_PATH=/u00/tools/xml/lib:/u00/tools:$LD_LIBRARY_PATH")
    )
    )


    sqlnet.ora - we do not have it

    *>lsnrctl status*

    LSNRCTL for Solaris: Version 11.2.0.2.0 - Production on 08-MAY-2013 17:33:07

    Copyright (c) 1991, 2010, Oracle. All rights reserved.

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.211.2.16)(PORT=1521)))
    STATUS of the LISTENER
    ------------------------
    Alias listener
    Version TNSLSNR for Solaris: Version 11.2.0.2.0 - Production
    Start Date 24-APR-2013 18:49:53
    Uptime 13 days 22 hr. 43 min. 15 sec
    Trace Level off
    Security ON: Local OS Authentication
    SNMP OFF
    Listener Parameter File /oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
    Listener Log File /oracle/diag/tnslsnr/mosibs03/listener/alert/log.xml
    Listening Endpoints Summary...
    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.211.2.16)(PORT=1521)))
    Services Summary...
    Service "ibs9" has 1 instance(s).
    Instance "ibs9", status UNKNOWN, has 1 handler(s) for this service...
    Service "ibsdev" has 2 instance(s).
    Instance "ibsdev", status UNKNOWN, has 1 handler(s) for this service...
    Instance "ibsdev", status READY, has 1 handler(s) for this service...
    Service "ibsdrv" has 2 instance(s).
    Instance "ibsdrv", status UNKNOWN, has 1 handler(s) for this service...
    Instance "ibsdrv", status READY, has 1 handler(s) for this service...
    Service "ibsdrv2" has 1 instance(s).
    Instance "ibsdrv2", status UNKNOWN, has 1 handler(s) for this service...
    Service "ibspatch" has 2 instance(s).
    Instance "ibspatch", status UNKNOWN, has 1 handler(s) for this service...
    Instance "ibspatch", status READY, has 1 handler(s) for this service...
    Service "rmancat" has 1 instance(s).
    Instance "rmancat", status READY, has 1 handler(s) for this service...
    Service "rmancatXDB" has 1 instance(s).
    Instance "rmancat", status READY, has 1 handler(s) for this service...
    The command completed successfully

    tail of alert.log:

    Writing to the above trace file is disabled for now on...
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x100991E70, kspgvc()+720] [flags: 0x0, count: 1]
    Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_pr0e_7393.trc (incident=89942):
    ORA-07445: exception encountered: core dump [kspgvc()+720] [SIGSEGV] [ADDR:0x0] [PC:0x100991E70] [Address not mapped to object] []
    Incident details in: /oracle/diag/rdbms/ibs9stb/ibs9/incident/incdir_89942/ibs9_pr0e_7393_i89942.trc
    Use ADRCI or Support Workbench to package the incident.
    See Note 411.1 at My Oracle Support for error and packaging details.
    Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xC2891] [PC:0x104758740, ssexhd()+2176] [flags: 0x0, count: 2]
    Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_pr0e_7393.trc (incident=89943):
    ORA-07445: exception encountered: core dump [ssexhd()+2176] [SIGBUS] [ADDR:0xC2891] [PC:0x104758740] [Invalid address alignment] []
    ORA-07445: exception encountered: core dump [kspgvc()+720] [SIGSEGV] [ADDR:0x0] [PC:0x100991E70] [Address not mapped to object] []
    Incident details in: /oracle/diag/rdbms/ibs9stb/ibs9/incident/incdir_89943/ibs9_pr0e_7393_i89943.trc
    Use ADRCI or Support Workbench to package the incident.
    See Note 411.1 at My Oracle Support for error and packaging details.
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9AC80000000C28C5] [PC:0x1009BFBE4, kspGetInstType()+196] [flags: 0x0, count: 3]
    Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/incident/incdir_89943/ibs9_pr0e_7393_i89943.trc:
    ORA-49203: Too many recursive DDE invocations
    ORA-07445: exception encountered: core dump [kspGetInstType()+196] [SIGSEGV] [ADDR:0x9AC80000000C28C5] [PC:0x1009BFBE4] [Address not mapped to object] []
    ORA-07445: exception encountered: core dump [ssexhd()+2176] [SIGBUS] [ADDR:0xC2891] [PC:0x104758740] [Invalid address alignment] []
    ORA-07445: exception encountered: core dump [kspgvc()+720] [SIGSEGV] [ADDR:0x0] [PC:0x100991E70] [Address not mapped to object] []
    Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F0000000819] [PC:0x10099126C, kspgsp()+236] [flags: 0x2, count: 4]
    Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F0000000819] [PC:0x10099126C, kspgsp()+236] [flags: 0x2, count: 5]
    Tue Apr 23 11:10:57 2013
    Non critical error ORA-07202 caught while writing to trace file "/oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_lgwr_7123.trc"
    Error message: ORA-07202: sltln: invalid parameter to sltln.

    Writing to the above trace file is disabled for now on...
    Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F0000000021] [PC:0x100991C88, kspgvc()+232] [flags: 0x0, count: 1]
    Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_lgwr_7123.trc (incident=89702):
    ORA-07445: exception encountered: core dump [kspgvc()+232] [SIGBUS] [ADDR:0xB38F0000000021] [PC:0x100991C88] [Invalid address alignment] []
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xA] [PC:0x10098F6EC, kspgip()+236] [flags: 0x2, count: 2]
    Tue Apr 23 11:10:58 2013
    Non critical error ORA-07202 caught while writing to trace file "/oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_smon_7127.trc"
    Error message: ORA-07202: sltln: invalid parameter to sltln.

    Writing to the above trace file is disabled for now on...
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xFFFFFFFF0000000A] [PC:0x10098F6EC, kspgip()+236] [flags: 0x0, count: 1]
    Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_smon_7127.trc (incident=89718):
    ORA-07445: exception encountered: core dump [kspgip()+236] [SIGSEGV] [ADDR:0xFFFFFFFF0000000A] [PC:0x10098F6EC] [Address not mapped to object] []
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xA] [PC:0x10098F6EC, kspgip()+236] [flags: 0x2, count: 2]
    Tue Apr 23 11:10:58 2013
    Non critical error ORA-07202 caught while writing to trace file "/oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_mmnl_7133.trc"
    Error message: ORA-07202: sltln: invalid parameter to sltln.

    Writing to the above trace file is disabled for now on...
    Exception [type: SIGBUS, Invalid address alignment] [ADDR:0x6B676C7365736874] [PC:0x100991E70, kspgvc()+720] [flags: 0x0, count: 1]
    Tue Apr 23 11:10:58 2013
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x2073697A6520746F] [PC:0xFFFFFFFF7A93C6F8, strlen()+24] [flags: 0x0, count: 1]
    Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_mmnl_7133.trc (incident=89742):
    ORA-07445: exception encountered: core dump [kspgvc()+720] [SIGBUS] [ADDR:0x6B676C7365736874] [PC:0x100991E70] [Invalid address alignment] []
    Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F000000002B] [PC:0x10098F6EC, kspgip()+236] [flags: 0x2, count: 2]
    Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_mman_7117.trc (incident=89678):
    ORA-07445: exception encountered: core dump [strlen()+24] [SIGSEGV] [ADDR:0x2073697A6520746F] [PC:0xFFFFFFFF7A93C6F8] [Address not mapped to object] []
    Incident details in: /oracle/diag/rdbms/ibs9stb/ibs9/incident/incdir_89678/ibs9_mman_7117_i89678.trc
    Use ADRCI or Support Workbench to package the incident.
    See Note 411.1 at My Oracle Support for error and packaging details.
    Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xC2891] [PC:0x104758740, ssexhd()+2176] [flags: 0x0, count: 2]
    Tue Apr 23 11:11:00 2013
    (ospid: 7148): terminating the instance due to error -557318144
    Tue Apr 23 11:11:00 2013
    Non critical error ORA-07202 caught while writing to trace file "/oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_dbrm_7113.trc"
    Error message: ORA-07202: sltln: invalid parameter to sltln.

    Writing to the above trace file is disabled for now on...
    Exception [type: SIGBUS, Invalid address alignment] [ADDR:0xB38F0000000021] [PC:0x100991C88, kspgvc()+232] [flags: 0x0, count: 1]
    Errors in file /oracle/diag/rdbms/ibs9stb/ibs9/trace/ibs9_dbrm_7113.trc (incident=89662):
    ORA-07445: exception encountered: core dump [kspgvc()+232] [SIGBUS] [ADDR:0xB38F0000000021] [PC:0x100991C88] [Invalid address alignment] []
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xA] [PC:0x10098F6EC, kspgip()+236] [flags: 0x2, count: 2]
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9AC80000000C28C5] [PC:0x1009BFBE4, kspGetInstType()+196] [flags: 0x2, count: 3]
    Tue Apr 23 11:11:18 2013
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x9AC80000000C28C5] [PC:0x1009BFBE4, kspGetInstType()+196] [flags: 0x2, count: 3]
    Tue Apr 23 11:11:40 2013
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10099125C, kspgsp()+220] [flags: 0x2, count: 4]
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10099125C, kspgsp()+220] [flags: 0x2, count: 5]
    Tue Apr 23 11:12:18 2013
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10099126C, kspgsp()+236] [flags: 0x2, count: 4]
    Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10099126C, kspgsp()+236] [flags: 0x2, count: 5]
  • 3. Re: Standby DB Instance fails to start and crashes on try to connect internally
    sb92075 Guru
    Currently Being Moderated
    open Service Request & resolve source of all the ORA-07445 errors.
  • 4. Re: Standby DB Instance fails to start and crashes on try to connect internally
    Fran Guru
    Currently Being Moderated
    like sb told you must check and open a SR in MOS (My oracle support).

    Also check this thread for the error "ORA-07202 sltln: invalid parameter to sltln"

    ORA-07202 sltln: invalid parameter to sltln.using

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points