This discussion is archived
5 Replies Latest reply: Sep 11, 2013 6:01 AM by Akanksha Sheoran Kaler RSS

OEM 12c "Error Message: Not Available" while installing agent in Solaris SPARC

Nock Newbie
Currently Being Moderated

Hello,

 

I'm installing an agent of cloud control 12c in Solaris SPARC, and everything runs fine until I reach the "secure agent- start agent" step. In the em console it says:

 

"Execution of command /oracle/app/oraagent/agent_inst/bin/emctl start agent on host izanagi_oracleoltp Failed. Error Message:Not Available."


The recommendation tells me to run secure agent, start agent, etc manually but it doesn't work either as you imagine. Actually I found that secure agent is finishing well but the problem is with start agent. When I try to run it manually it just tells me to Consult emctl.log and emagent.nohup in: /oracle/app/oraagent/agent_inst/sysman/log. If I try to run it again it tells me theres a possible conflict with the port (3872), I imagine is because it left it open when I run it for the first time and falied. Anyways, here is some output from those logs and from the log in the em host.


From the em host in gc_inst/em/EMGC_OMS1/sysman/agentpush/2013-09-05_09-36-12-AM/applogs/oracleoltp_deploy.log:


2013-09-05_09-55-14:INFO:Action description Execution of command /oracle/app/oraagent/agent_inst/bin/emctl start agent  on host oracleoltp

2013-09-05_09-55-14:INFO:Attempt :1 pty required false  with no inputs

2013-09-05_10-03-53:INFO:/oracle/app/oraagent/agent_inst/bin/emctl start agent execution failed on host oracleoltp

2013-09-05_10-03-54:INFO: ACTION Execution of command /oracle/app/oraagent/agent_inst/bin/emctl start agent  on host oracleoltp

2013-09-05_10-03-54:INFO: OUT Oracle Enterprise Manager Cloud Control 12c Release 3 

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.Starting agent ...........................................................................................................................Consult emctl.log and emagent.nohup in: /oracle/app/oraagent/agent_inst/sysman/log

 

 

2013-09-05_10-03-54:INFO: ERR null

2013-09-05_10-03-54:INFO: EXIT CODE1

2013-09-05_10-03-54:INFO:InvocationTargetException Exception

2013-09-05_10-03-54:INFO:Printing Exception :java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at oracle.sysman.core.agentpush.ui.deployer.BaseDeployerOps.executeActions(BaseDeployerOps.java:1862)

        at oracle.sysman.core.agentpush.ui.deployer.NewAgentDeployer.deploy(NewAgentDeployer.java:59)

        at oracle.sysman.core.agentpush.ui.deployfwk.DeploymentWorker.run(DeploymentWorker.java:26)

        at oracle.sysman.util.threadPoolManager.WorkerThread.run(Worker.java:311)

Caused by: CommandException: err: null out: Oracle Enterprise Manager Cloud Control 12c Release 3 

Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.Starting agent ...........................................................................................................................Consult emctl.log and emagent.nohup in: /oracle/app/oraagent/agent_inst/sysman/log

exitcode: 1

stacktrace:

null

        at oracle.sysman.core.agentpush.ui.deployer.DeployerOps.executeCommandOnNodeInteractive(DeployerOps.java:1199)

        at oracle.sysman.core.agentpush.ui.deployer.DeployerOps.executeCommandOnNodeInteractive(DeployerOps.java:1039)

        at oracle.sysman.core.agentpush.ui.deployer.BaseDeployerOps.doSecureAgent(BaseDeployerOps.java:664)

        ... 8 more

 

 

2013-09-05_10-03-54:INFO:=========Command Exception :Error Message:Not Available<br><br> Exit Code :1

2013-09-05_10-03-54:INFO:Updating Action SecureAgentwith Status FAILED and error Message :Error Message:Not Available<br><br> Exit Code :1 and problem Execution of command /oracle/app/oraagent/agent_inst/bin/emctl start agent  on host oracleoltp Failed and recommendation Fix the cause of the error and retry the operation (or) manually run the following commands on the remote host <ul> <li>/oracle/app/oraagent/agent_inst/bin/emctl secure agent</li><li>/oracle/app/oraagent/agent_inst/bin/emctl start agent</li><li>/oracle/app/oraagent/agent_inst/bin/emctl config agent addinternaltargets</li></ul>

2013-09-05_10-03-54:INFO:=================action status is not empty FAILED

2013-09-05_10-03-54:INFO:Breaking since the action has failed

2013-09-05_10-03-54:INFO:Skipping action RunRootSH since some previous step has failed

 

From agent_inst/sysman/log/emctl.log:

 

3953 :: Thu Sep  5 09:54:43 2013::Initializing the agent command locking system

3953 :: Thu Sep  5 09:54:48 2013::AgentStatus.pm:emdctl status agent returned 1

3953 :: Thu Sep  5 09:54:48 2013::Status Output:Status agent Failure:Unable to connect to the agent at http://izanagi_oracleoltp:3872/emd/lifecycle/main/ [oracleoltp]

3953 :: Thu Sep  5 09:54:48 2013::AgentPlugin.pm:istatusCEMD returned 1

3953 :: Thu Sep  5 09:54:48 2013::AgentPlugin.pm: Agent is not running

3953 :: Thu Sep  5 09:54:48 2013::AgentPlugin.pm: Wrote UpdateInventory=true in /oracle/app/oraagent/agent_inst/sysman/emd/agntstmp.txt

3953 :: Thu Sep  5 09:54:48 2013::AgentPlugin.pm:EMD update_inventory plugin succeeded

3953 :: Thu Sep  5 09:54:48 2013::Cleaning up agent command lock

3953 :: Thu Sep  5 09:54:48 2013::AgentCommandLock:closed file handle of emctl lockfile

4070 :: Thu Sep  5 09:54:51 2013::Initializing the agent command locking system

4070 :: Thu Sep  5 09:54:51 2013::AgentLifeCycle.pm: Processing secure agent

4070 :: Thu Sep  5 09:54:51 2013::AgentStatus.pm:Processing secure agent

4070 :: Thu Sep  5 09:54:58 2013::AgentStatus.pm:emdctl gensudoprops returned 0

4070 :: Thu Sep  5 09:54:58 2013::Cleaning up agent command lock

4070 :: Thu Sep  5 09:54:58 2013::AgentCommandLock:closed file handle of emctl lockfile

4165 :: Thu Sep  5 09:55:00 2013::Initializing the agent command locking system

4165 :: Thu Sep  5 09:55:00 2013::AgentLifeCycle.pm: Processing config agent

4165 :: Thu Sep  5 09:55:00 2013::AgentStatus.pm:Processing config agent

4165 :: Thu Sep  5 09:55:00 2013::AgentStatus.pm:EMHOME is /oracle/app/oraagent/agent_inst

4178 :: Thu Sep  5 09:55:02 2013::Initializing the agent command locking system

4178 :: Thu Sep  5 09:55:02 2013::AgentLifeCycle.pm: Processing list_type_metaversions agent

4178 :: Thu Sep  5 09:55:02 2013::AgentStatus.pm:Processing list_type_metaversions agent

4178 :: Thu Sep  5 09:55:13 2013::AgentStatus.pm:/oracle/app/oraagent/core/12.1.0.3.0/bin/emdctl list_type_metaversions agent -type plugins returned 0

4178 :: Thu Sep  5 09:55:13 2013::Cleaning up agent command lock

4178 :: Thu Sep  5 09:55:13 2013::AgentCommandLock:closed file handle of emctl lockfile

4165 :: Thu Sep  5 09:55:13 2013::AgentStatus.pm: /oracle/app/oraagent/core/12.1.0.3.0/jdk/jre/bin/java -DORACLE_HOME=/oracle/app/oraagent/core/12.1.0.3.0 -DEMHOME=/oracle/app/oraagent/agent_inst -cp /oracle/app/oraagent/core/12.1.0.3.0/jlib/emConfigInstall.jar oracle.sysman.emSDK.conf.TargetInstaller addinternaltargets -dump /oracle/app/oraagent/ADATMP_2013-09-05_09-36-12-AM/targets.xml returned 0

4165 :: Thu Sep  5 09:55:13 2013::Cleaning up agent command lock

4165 :: Thu Sep  5 09:55:13 2013::AgentCommandLock:closed file handle of emctl lockfile

4295 :: Thu Sep  5 09:55:17 2013::Initializing the agent command locking system

4295 :: Thu Sep  5 09:55:17 2013::AgentLifeCycle.pm: Processing start agent

4295 :: Thu Sep  5 09:55:17 2013::AgentLifeCycle.pm: EMHOME is /oracle/app/oraagent/agent_inst

4295 :: Thu Sep  5 09:55:17 2013::AgentLifeCycle.pm: service name is 

4295 :: Thu Sep  5 09:55:17 2013::AgentLifeCycle.pm: Agent will be listening on port: 3872

4295 :: Thu Sep  5 09:55:17 2013::AgentLifeCycle.pm: ParentProcess id=4288

4295 :: Thu Sep  5 09:55:17 2013::Calling acquireGlobalLock (1)

4295 :: Thu Sep  5 09:55:17 2013::AgentCommandLock:Acquiring a lock(type=1)on emctl lockfile

4295 :: Thu Sep  5 09:55:17 2013::AgentCommandLock:lockFile returns 1

4295 :: Thu Sep  5 09:55:17 2013::acquireGlobalLock returned: 1

4295 :: Thu Sep  5 09:55:17 2013::EM_SECURE_HOSTNME: oracleoltp

4295 :: Thu Sep  5 09:55:17 2013::EM_SECURE_PORT: 3872

4295 :: Thu Sep  5 09:55:17 2013::EM_LISTEN_ON_ALL_NICS: true

4295 :: Thu Sep  5 09:55:25 2013::AgentLifeCycle.pm:status agent returned with retCode=1

4413 :: Thu Sep  5 09:55:25 2013::AgentLifeCycle.pm: Launching the watchdog process.

4413 :: Thu Sep  5 09:55:27 2013::TZ = , Read = . Found = 1.

4413 :: Thu Sep  5 09:55:27 2013::validateTZAgainstAgent:

4413 :: Thu Sep  5 09:55:27 2013::agentTZRegion successfully validated.

4413 :: Thu Sep  5 09:55:28 2013::AgentTuning.pm: Launching emdctl with -Xmx1024m

4413 :: Thu Sep  5 09:55:42 2013::AgentStatus.pm:emdctl run_autotune returned 0

4295 :: Thu Sep  5 09:55:25 2013::AgentLifeCycle.pm: Launched the watchdog process pid=4413

4295 :: Thu Sep  5 09:55:25 2013::AgentLifeCycle.pm: StartCEMD start retryCount=120

4295 :: Thu Sep  5 09:56:02 2013::AgentLifeCycle.pm: StartCEMD status message file detected

4295 :: Thu Sep  5 09:56:02 2013::Found Msg file with: {1378391161885,running}

4295 :: Thu Sep  5 09:56:02 2013::AgentLifeCycle.pm: StartCEMD Querying for the real status of the agent

4295 :: Thu Sep  5 09:56:11 2013::AgentLifeCycle.pm: StartCEMD status message file detected

 

From agent_inst/sysman/log/emagent.nohup:

 

----- Thu Sep  5 09:55:42 2013::4413::Agent Launched with PID 4508 at time Thu Sep  5 09:55:42 2013 -----

----- Thu Sep  5 09:55:42 2013::4508::Time elapsed between Launch of Watchdog process and execing EMAgent is 17 secs -----

2013-09-05 09:55:43,740 [1:main] WARN - Missing filename for log handler 'wsm'

2013-09-05 09:55:43,766 [1:main] WARN - Missing filename for log handler 'opss'

2013-09-05 09:55:43,768 [1:main] WARN - Missing filename for log handler 'opsscfg'

----- Thu Sep  5 09:56:23 2013::4413::Checking status of EMAgent : 4508 -----

----- Thu Sep  5 10:01:23 2013::4413::Checking status of EMAgent : 4508 -----

----- Thu Sep  5 10:01:23 2013::4413::Abnormality reported for EMAgent : 4508 -----

----- Thu Sep  5 10:01:23 2013::4413::Debugging component EMAgent -----

----- Thu Sep  5 10:01:23 2013::generate first thread dump file for diagnosis -----

----- Thu Sep  5 10:01:37 2013::generate second thread dump file for diagnosis -----

----- Thu Sep  5 10:01:41 2013::generate Threads.4508lsof.1 for diagnosis -----

----- Attempting to kill EMAgent : 4508 -----

----- Thu Sep  5 10:01:46 2013::4413::EMAgent exited at Thu Sep  5 10:01:46 2013 with signal 9 -----

----- Thu Sep  5 10:01:46 2013::4413::EMAgent either hung or in abnormal state. -----

----- Thu Sep  5 10:01:46 2013::4413::EMAgent will be restarted/thrashed. -----

----- Thu Sep  5 10:01:46 2013::4413::Restarting EMAgent. -----

----- Thu Sep  5 10:01:49 2013::4413::Auto tuning the agent at time Thu Sep  5 10:01:49 2013 -----

----- Thu Sep  5 10:01:56 2013::4413::Finished auto tuning the agent at time Thu Sep  5 10:01:56 2013 -----

----- Thu Sep  5 10:01:56 2013::4413::Launching the JVM with following options: -Xmx128M -server -d64 -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:-DoEscapeAnalysis -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -XX:-UseLWPSynchronization -Dsun.security.pkcs11.enable-solaris=false -----

----- Thu Sep  5 10:01:56 2013::8135::Execing EMAgent process is taking longer than expected 120 secs -----

----- Thu Sep  5 10:01:56 2013::8135::Time elapsed between Launch of Watchdog process and execing EMAgent is 391 secs -----

----- Thu Sep  5 10:01:56 2013::4413::Agent Launched with PID 8135 at time Thu Sep  5 10:01:56 2013 -----

----- Thu Sep  5 10:02:40 2013::4413::Checking status of EMAgent : 8135 -----

 

I double checked the IPs/hostname configurations, and double checked the ports where available, I really don't know why this is happening.

 

Thanks for your help,

 

Néstor Manrique

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points