This discussion is archived
1 2 Previous Next 16 Replies Latest reply: Oct 9, 2012 1:29 PM by EduardB RSS

Oracle Database unavailable after machine restart

EduardB Newbie
Currently Being Moderated
I have a dedicated machine where I have installed Oracle Database 11.2.0.1.0 under Oracle Linux 6. It worked fine until today, when I had to shutdown the machine for physical relocation. Much to my surprise, when I have restarted the machine, EM wasn't available. Then I realized I could no longer remotely connect to the Database. Then I found out that the listener was down. Then I tried starting the listener, it said that no clients are supported. Then I tried to manually start the DB from sqlplus, I could not even log in to it. Then I ran dbca, which cried about EM not being secure. Then I found out that the emkey.ora was missing (and as a newbie, I hadn't made a backup).

I spent 3 hours trying to figure out how to bring the DB online; I'm afraid I ran out of solutions. Right now I am rebuilding the whole EM Repository using dbca and hope for the best.

Update: dbca finished with no errors. I can now connect with sqlplus and see that all data is there. However, EM still doesn't properly connect. I am getting "Enterprise Manager is not able to connect to the database instance". However, state of components lists:

- Database Instance: Open
- Listener status: UP
- Agent connection to Instance: Succeeded

I can't figure out why EM won't connect. I have this feeling it's a listener misconfiguration, however netca is very limited, just a next-next-next thing.

My listener.ora is below:

SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = eburlacu)
(ORACLE_HOME = /oracledb/product/11.2.0/dbhome_1)
(SID_NAME = eburlacu)
)
)

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = eburlacu-server.ro.oracle.com)(PORT = 1521))
)
)

ADR_BASE_LISTENER = /oracledb

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER = OFF


...And why is EM using port 5500?

LE: I can now remotely connect to the Database from a different machine, using Oracle SQL Developer, so the listener seems to be working well, after all. I'm confused now...

Edited by: EduardB on Sep 28, 2012 3:24 PM
  • 1. Re: Oracle Database unavailable after machine restart
    sb92075 Guru
    Currently Being Moderated
    what clues exist within EM log file?
  • 2. Re: Oracle Database unavailable after machine restart
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
     I have installed Oracle Database 11.2.0.1.0 under Oracle Linux 6
    First
    Oracle Database 11.2.0.1.0 is not certified on Linux x86 Oracle Linux 6

    the only version certified with linux 6 is 11.2.0.3


    Did you check the agent for EM ?
  • 3. Re: Oracle Database unavailable after machine restart
    EduardB Newbie
    Currently Being Moderated
    Hi Osama,

    It worked for a full month until a shutdown and machine restart. It also used to work through a set of over 10 restarts right after I installed it, with no issues.

    I have checked the log file emrep_config.log.1 and it has no errors. This is the only file that's appearing there. Should I check a different file?
  • 4. Re: Oracle Database unavailable after machine restart
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    maybe its installed successfully , and you use it normally but its NOT CERTIFIED which mean its wrong .
  • 5. Re: Oracle Database unavailable after machine restart
    EduardB Newbie
    Currently Being Moderated
    If it DID work for a full month, then stopped working, it means it CAN work, doesn't it?
    Anyway, I'll probably go ahead and install latest version (which I downloaded) and see how it goes from there.
  • 6. Re: Oracle Database unavailable after machine restart
    jgarry Guru
    Currently Being Moderated
    EduardB wrote:
    If it DID work for a full month, then stopped working, it means it CAN work, doesn't it?
    Not really. I can drive a car for a full month, but if all the oil leaks out, it can't anymore.

    Probably what happened is you did something for those ten restarts to modify the em environment that you forgot about, or your network environment changed with the move so it looks in the wrong place for configuration. It's also possible that something em itself did caused it to self-immolate. Most likely there is something in the logs which will tell you, find all the occurrences of emagent.trc and look at all the recently written files in those directories. You might notice there are several directories in different places that reflect hostname and port changes or some such.
    Anyway, I'll probably go ahead and install latest version (which I downloaded) and see how it goes from there.
    It's tough even for the experienced, but you have to learn where the logs and traces are and what they are trying to tell you. It's hard enough with a supported configuration, you're on your own with unsupported.
  • 7. Re: Oracle Database unavailable after machine restart
    EdStevens Guru
    Currently Being Moderated
    EduardB wrote:
    If it DID work for a full month, then stopped working, it means it CAN work, doesn't it?
    It's like IBM told us when they were making OS changes to implement the new support libraries for COBOL-85. "Previously compiled and linked code will continue to work until such time as it doesn't". It wasn't a threat. It was simply a statement of how things are on an unsupported/uncertified platform.

    Anyway, I'll probably go ahead and install latest version (which I downloaded) and see how it goes from there.
  • 8. Re: Oracle Database unavailable after machine restart
    EduardB Newbie
    Currently Being Moderated
    Thank you for the response. It makes some sense, however I wouldn't have expected the Oracle DB to be so se sensitive to a simple machine move.
    The network configuration hasn't changed a bit. I am in the same network, I have the same IP (it's a static IP and a static DNS hostname which was granted to this machine specifically). I just changed floors.

    I never touched the EM environment. All I did was:
    1. Installed Oracle Database.
    2. Added a schema (basically created a new user).
    3. Allowed SQL Developer through the firewall.
    4. Remotely connected with SQL Developer and created tables and functions, all basic SQL work, nothing fancy.

    The restarts were simply for testing purposes (does it still start as expected? Are there warnings, error messages?). Two restarts were performed to re-arange some wiring, again nothing fancy.

    I am just a newbie doing some learning and gaining some hands-on experience (after taking Linux and SQL classes). The Oracle Database 11g Administration Workshops are scheduled for 2013 but in the meanwhile I am trying to understand why could my installation mess up so badly with no outside interaction.

    What would your advice be in this situation? Reinstall everything under OEL5 + 11.2.0.1.0? Reinstall OEL6 with clean 11.2.0.3.0 installation?

    Note: my first priority at this moment is hands-on experience with SQL. Since both sql*plus and Oracle SQL Developer can connect and work fine, should I simply stop bothering with EM for the moment and let it limp until I attend the Administration Workshops? I have almost half a million records in the DB but my SQL statements (saved on a different machine) can re-generate all the data in about 1h or so, so it's not a big deal if I wipe everything.
  • 9. Re: Oracle Database unavailable after machine restart
    Mahmoud_Rabie Journeyer
    Currently Being Moderated
    Dear Eduard,
    What would your advice be in this situation? Reinstall everything under OEL5 + 11.2.0.1.0? Reinstall OEL6 with clean 11.2.0.3.0 installation?
    My advice:
    Do not reinstall. Learn and Try to Fix and overcome problems.
    There is a solution and if not, there is a workaround.

    Let us focus with you on the most two important quotes

    First of sb92075
    what clues exist within EM log file?
    Second of jgarry      
    Most likely there is something in the logs which will tell you, find all the occurrences of emagent.trc and look at all the recently written files in those directories.
    Did you check the emagent.trc?

    Best Regards
    Mahmoud
  • 10. Re: Oracle Database unavailable after machine restart
    jgarry Guru
    Currently Being Moderated
    When all else fails, follow the directions. https://blogs.oracle.com/linux/entry/oracle_rdbms_server_11gr2_pre
  • 11. Re: Oracle Database unavailable after machine restart
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Check the error in log ?

    And What you mean by No connected since you already said you check Database is Up , Listener is up , Agent is Up
  • 12. Re: Oracle Database unavailable after machine restart
    EduardB Newbie
    Currently Being Moderated
    @jgarry: yes, I had already done that when I installed the Database.
    @Osama: this is exactly what I am getting: see screenshot here: http://imageshack.us/a/img834/7183/11gemerror.png
  • 13. Re: Oracle Database unavailable after machine restart
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    Seems EM agent Stuck !!!

    after make sure of $ORACLE_HOME, $ORACLE_SID
    emctl stop agent
    emctl stop dbconsole 
    emctl start dbconsole 
    emctl status agent 
    reconnect again , if this not works then recreate EM with emca
  • 14. Re: Oracle Database unavailable after machine restart
    EduardB Newbie
    Currently Being Moderated
    There seems to be some problem displayed in the emagent.trc log:
    2012-10-09 22:34:37,752 Thread-2277304064 WARN  http: nmehl_connect_internal: connect failed to (eburlacu-server.ro.oracle.com:5500): Connection refused (error = 111)
    2012-10-09 22:34:37,752 Thread-2277304064 ERROR pingManager: nmepm_pingReposURL: Cannot connect to http://eburlacu-server.ro.oracle.com:5500/em/upload/: retStatus=-1
    2012-10-09 22:34:37,774 Thread-2277304064 WARN  command: Job Subsystem Timeout set at 600 seconds
    2012-10-09 22:34:37,848 Thread-2277304064 WARN  upload: Upload manager has no Failure script: disabled
    2012-10-09 22:34:37,975 Thread-2277304064 WARN  metadata: Metric VirtualHosts does not have any data columns
    2012-10-09 22:34:38,227 Thread-2277304064 WARN  metadata: Metric hung_system_traces does not have any data columns
    2012-10-09 22:34:38,569 Thread-1946154752 ERROR : (nmecmgr.c,3230):Memory 0x0 encountered, expect struct_id=11011
    2012-10-09 22:34:38,570 Thread-1943004928 ERROR : (nmecmgr.c,3230):Memory 0x0 encountered, expect struct_id=11011
    2012-10-09 22:34:38,571 Thread-1936705280 ERROR : (nmecmgr.c,3230):Memory 0x0 encountered, expect struct_id=11011
    2012-10-09 22:34:38,571 Thread-1939855104 ERROR : (nmecmgr.c,3230):Memory 0x0 encountered, expect struct_id=11011
    2012-10-09 22:34:38,692 Thread-1936705280 WARN  TargetManager: Query returned 0 rows (only one expected) for the dynamic property from_cluster
    2012-10-09 22:34:38,692 Thread-1936705280 ERROR : (nmecmgr.c,3230):Memory 0x0 encountered, expect struct_id=11011
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points