-
-
2. Re: OMS/sysman/recv directory filled up with xml-files
Hi Ugonic,
thank you for the tip, but it don't work.
As far as I know, the upload-command causes the agent to move the xml-files to OMS. But they are already there.
oracle@:/oracle/product/102grid/oms10g/sysman/recv # ls -la|wc -l
130127
I need to force the OMS to put them to the Repository and delete them subsequently. -
3. Re: OMS/sysman/recv directory filled up with xml-files
When you tried this did you check whether the number of files reduced?
You can go to the Agent in Grid Control by select Targets > All Targets, click on the Agent, then under Upload on the screen, check the value of Last Successful Upload and Data Pending Upload to see if they corresponds to the waiting files or the last time you tried the upload.
Click on Upload Metric Data button. If you get any error that relate to maximum upload files or size, keep clicking it until it uploads all. If it does not work, you may have to increase the maximum allowed upload and size in emd.properties to accomodate that size.
Also check the log files for relaed errors that will indicate what is happening -
4. Re: OMS/sysman/recv directory filled up with xml-files
Please paste the emoms.trc and emagent..trc here -
5. Re: OMS/sysman/recv directory filled up with xml-files
Hello Mnazim,
no, the number of files did not decrease after the emctl upload.
I'm quite shure that the infomation of the xml-files is proceeded to the repository because the grid control surface shows information which come with the xml-files. Maybe the just are not deleted.
How can i attach the trace-files? I think cut and paste is not a good idea because the emagent.trc is 123kB and the emoms.trc is 609kB.
Could i send them via e-mail?
best regards
Zaphod -
6. Re: OMS/sysman/recv directory filled up with xml-files
Could you please paste the last 50 lines of both the trc file? -
7. Re: OMS/sysman/recv directory filled up with xml-files
tail -50 emoms.trc:
2006-11-21 13:04:49,950 [MetadataLoad http://isora74.audi.de:15746/emd/main/:A0000000002.xml] ERROR XMLLoader.Splitter submitDiscardStateJob.2232 - Exception occurred during DiscardStateJob submission.
java.sql.SQLException: ORA-20224: The following set of target parameters reference an invalid target: (targetNames,targetTypes)
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 14883
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 6877
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7009
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7090
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7543
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 8387
ORA-06512: at "SYSMAN.MGMT_JOBS", line 259
ORA-06512: at "SYSMAN.MGMT_JOBS", line 86
ORA-06512: at line 1
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:304)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:271)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:625)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:183)
at oracle.jdbc.driver.T4CCallableStatement.execute_for_rows(T4CCallableStatement.java:872)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1162)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2932)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3023)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4132)
at oracle.sysman.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:645)
at oracle.sysman.emdrep.jobs.JobEngine.scheduleJob(JobEngine.java:870)
at oracle.sysman.emdrep.jobs.JobEngine.submitJob(JobEngine.java:778)
at oracle.sysman.emSDK.job.JobManager.submitJob(JobManager.java:157)
at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.submitDiscardStateJob(XMLDocumentSplitter.java:2221)
at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.onEndUpload(XMLDocumentSplitter.java:1748)
at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.endElement(XMLDocumentSplitter.java:979)
at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1257)
at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:314)
at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:281)
at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:196)
at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.split(XMLDocumentSplitter.java:451)
at oracle.sysman.emdrep.dbjava.loader.XMLLoaderContext.loadFromStream(XMLLoaderContext.java:1672)
at oracle.sysman.emdrep.dbjava.loader.XMLLoader.DirectLoad(XMLLoader.java:491)
at oracle.sysman.emdrep.receiver.FxferRecv.DirectLoad(FxferRecv.java:993)
at oracle.sysman.emdrep.receiver.FxferRecv.processFile(FxferRecv.java:1368)
at oracle.sysman.emdrep.receiver.FxferRecv.receiveFile(FxferRecv.java:1614)
at oracle.sysman.emdrep.receiver.FxferRecv.doPost(FxferRecv.java:764)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:336)
at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:663)
at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:330)
at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:830)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:224)
at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:133)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
at java.lang.Thread.run(Thread.java:534)
2006-11-21 13:14:44,217 [AJPRequestHandler-ApplicationServerThread-12] ERROR audit.AuditManager isAuditOn.157 - Repository may not be up.
tail -50 emagent.trc:
2006-11-18 05:52:10 Thread-28575 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated
2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 05:52:10 Thread-28575 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 05:52:10 Thread-28575 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 06:22:15 Thread-28901 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated
2006-11-18 06:22:15 Thread-28901 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 06:22:15 Thread-28901 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 06:22:15 Thread-28901 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 06:22:15 Thread-28901 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 07:52:06 Thread-29879 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated
2006-11-18 07:52:06 Thread-29879 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 07:52:06 Thread-29879 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 07:52:06 Thread-29879 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 07:52:06 Thread-29879 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 08:52:09 Thread-30532 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated
2006-11-18 08:52:09 Thread-30532 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 08:52:09 Thread-30532 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 08:52:09 Thread-30532 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 08:52:09 Thread-30532 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 09:52:10 Thread-31185 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated
2006-11-18 09:52:10 Thread-31185 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 09:52:10 Thread-31185 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 09:52:10 Thread-31185 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 09:52:10 Thread-31185 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
2006-11-18 10:11:23 Thread-31401 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-18 10:11:29 Thread-31404 ERROR collector: Error merging upload on demand files metric belonging to target EnterpriseManager0.isora62.audi.de_HTTP Server
2006-11-18 10:11:30 Thread-31405 ERROR collector: Error merging upload on demand files metric belonging to target EnterpriseManager0.isora62.audi.de_Web Cache
2006-11-18 10:11:44 Thread-31408 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-18 18:01:16 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
2006-11-20 12:46:29 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
2006-11-20 12:46:29 Thread-1 WARN main: failed to start upload manager, disabled
2006-11-20 12:46:39 Thread-32 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-20 12:47:00 Thread-40 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-20 18:17:28 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
2006-11-21 08:55:25 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
2006-11-21 08:55:25 Thread-1 WARN main: failed to start upload manager, disabled
2006-11-21 08:55:34 Thread-31 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-21 08:55:55 Thread-38 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-21 11:03:25 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
2006-11-21 11:13:35 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
2006-11-21 11:13:35 Thread-1 WARN main: failed to start upload manager, disabled
2006-11-21 11:13:43 Thread-32 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de -
8. Re: OMS/sysman/recv directory filled up with xml-files
Please stop all the opmnctl procees, agent and delete following from agent box
<AOH>\sysman\emd\agntstmp.txt
<AOH>\sysman\emd\lastupload.xml
<AOH>\sysman\emd\upload\*
<AOH>\sysman\emd\recv\
<AOH>\sysman\emd\rstate\*
<AOH>\sysman\emd\collection\*
start all opmn and agent process and let me know the result. -
9. Re: OMS/sysman/recv directory filled up with xml-files
Hi Mnazim,
two questions before i remove files :-)
1) in Oracle DocID 293940.1 it is adviced for a "clean start agent" to delete as well the .../emd/protocol.ini and issue a /emctl clearstate command followed by /emctl upload
should i do that?
2) I found two agents: one in <ORACLE_BASE>/oms10g/sysman/emd which i suppose to be the socalled 'central-agent'
the other one in <ORACLE_BASE>/agent10g/sysman/emd which is monitoring just the node
for which agent should i perform the "clean_up"?
thanks
Zaphod -
-
11. Re: OMS/sysman/recv directory filled up with xml-files
didn't work - the number of files is still encreasing
I suppose it cannot be solved by any operation with the AGENT, because it is an issue within the OMS - maybe some kind of misconfiguration (a parameter somewhat like: delete afterupload=true)
best regards
Zaphod -
12. Re: OMS/sysman/recv directory filled up with xml-files
From your log entries I can see:
emoms.trc:
2006-11-21 13:04:49,950 [MetadataLoad http://[b]isora74.audi.de:15746/emd/main/:A0000000002.xml] ERROR XMLLoader.Splitter submitDiscardStateJob.2232 - Exception occurred during DiscardStateJob submission.
java.sql.SQLException: ORA-20224: The following set of target parameters reference an invalid target: (targetNames,targetTypes)
emagent.trc:
2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,[b]isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated
Question:
1. What is the difference between isora62.audi.de and isora74.audi.de?
2. Is 15746 the correct port number -
13. Re: OMS/sysman/recv directory filled up with xml-files
Hi Ugonic,
we run a Veritas Cluster System. It consists of two nodes (physical machines) and some thirty Cluster Services (logical machines) which run on one of the nodes and may switch to tho other node (failover).
The Network System is CFS - ClusterFileSystem (which is different to NFS)
isora62 and isora74 are Cluster Services with a logical IP-adress. Therefore each agent needs a separate port. Port 15626 for isora62 and port 15746 for isora74
isora62 is the ClusterService on which the OMS, the OMR and a target-agent are running.
isora74 is a ClusterService on which a database (maybe more) is running. -
14. Re: OMS/sysman/recv directory filled up with xml-files
Hi,
change the line in emd.properties of every agent on the cluster
from
AgentListenOnAllNICs=TRUE
to
AgentListenOnAllNICs=FALSE
then you don't need separate ports if every agent is registered
with a different hostname (logical machines and physical machines) to the OMS.
The only exception is on the OMS host, where you have 2 agents,
one from the iasconsole (OMS OH) and one "add. agent" (extra OH),
which is monitoring the OMS and repository
The error in the agent trace file:
em_error=nmupm child process timed out. Terminated
can be fixed with the following line at the end of emd.properties:
NMUPM_TIMEOUT=60
(increase the value if needed, it's in seconds).
The agent <ORACLE_BASE>/oms10g/sysman/emd should never upload
to the OMS, it is only for the iasconsole.
The files in the recv directory on the OMS should be loaded into the repository
by the Loader of the OMS, or moved to the errors subdirectory (also done by the Loader).
Make sure, that your repository is up, has enough space, and
the OMS is up, running and connected to the repository.
Register and Participate in Oracle's online communities. Learn from thousand of experts, get answers to your questions and share knowledge with peers.