1 2 Previous Next 26 Replies Latest reply on Mar 30, 2007 6:57 PM by mrronn

recv directory filled up with xml-files

545934 Nov 15, 2006 3:30 PM

I suppose there is some misconfiguration in the OMS.
The Agents deliver their xml-files to the OMS. There they are stored in the $OMS_HOME/sysman/recv directory until they get handled by OMS. Normally you should not see more than 2 or 3 files in that (buffer)-directory.
I started installing agents on 19.Oct - from then on the xml-files were stored in the recv-directory until the the file-system was filled up to 100%.
First I stopped all agents and am now looking for advice to:
- clean OMS (i suppose there are more things to do than just delete the 129.000 files)
- find the item of misconfiguration

Hope there is some knowledge out there in the web.

best regards
Dieter

1. Re: OMS/sysman/recv directory filled up with xml-files

277993 Nov 15, 2006 6:09 PM (in response to 545934)

Manually run emctl upload
Like Show 0 Likes(0)

Actions
2. Re: OMS/sysman/recv directory filled up with xml-files

545934 Nov 17, 2006 7:59 AM (in response to 277993)

Hi Ugonic,
thank you for the tip, but it don't work.
As far as I know, the upload-command causes the agent to move the xml-files to OMS. But they are already there.
oracle@:/oracle/product/102grid/oms10g/sysman/recv # ls -la|wc -l
130127
I need to force the OMS to put them to the Repository and delete them subsequently.
Like Show 0 Likes(0)

Actions
3. Re: OMS/sysman/recv directory filled up with xml-files

277993 Nov 17, 2006 1:18 PM (in response to 545934)

When you tried this did you check whether the number of files reduced?

You can go to the Agent in Grid Control by select Targets > All Targets, click on the Agent, then under Upload on the screen, check the value of Last Successful Upload and Data Pending Upload to see if they corresponds to the waiting files or the last time you tried the upload.

Click on Upload Metric Data button. If you get any error that relate to maximum upload files or size, keep clicking it until it uploads all. If it does not work, you may have to increase the maximum allowed upload and size in emd.properties to accomodate that size.

Also check the log files for relaed errors that will indicate what is happening
Like Show 0 Likes(0)

Actions
4. Re: OMS/sysman/recv directory filled up with xml-files

mnazim-Oracle Nov 17, 2006 3:31 PM (in response to 545934)

Please paste the emoms.trc and emagent..trc here
Like Show 0 Likes(0)

Actions
5. Re: OMS/sysman/recv directory filled up with xml-files

545934 Nov 20, 2006 4:07 PM (in response to mnazim-Oracle)

Hello Mnazim,

no, the number of files did not decrease after the emctl upload.
I'm quite shure that the infomation of the xml-files is proceeded to the repository because the grid control surface shows information which come with the xml-files. Maybe the just are not deleted.

How can i attach the trace-files? I think cut and paste is not a good idea because the emagent.trc is 123kB and the emoms.trc is 609kB.
Could i send them via e-mail?

best regards
Zaphod
Like Show 0 Likes(0)

Actions
6. Re: OMS/sysman/recv directory filled up with xml-files

mnazim-Oracle Nov 20, 2006 4:10 PM (in response to 545934)

Could you please paste the last 50 lines of both the trc file?
Like Show 0 Likes(0)

Actions
7. Re: OMS/sysman/recv directory filled up with xml-files

545934 Nov 21, 2006 12:36 PM (in response to mnazim-Oracle)

tail -50 emoms.trc:
2006-11-21 13:04:49,950 [MetadataLoad http://isora74.audi.de:15746/emd/main/:A0000000002.xml] ERROR XMLLoader.Splitter submitDiscardStateJob.2232 - Exception occurred during DiscardStateJob submission.
java.sql.SQLException: ORA-20224: The following set of target parameters reference an invalid target: (targetNames,targetTypes)
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 14883
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 6877
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7009
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7090
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7543
ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 8387
ORA-06512: at "SYSMAN.MGMT_JOBS", line 259
ORA-06512: at "SYSMAN.MGMT_JOBS", line 86
ORA-06512: at line 1

     at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
     at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:304)
     at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:271)
     at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:625)
     at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:183)
     at oracle.jdbc.driver.T4CCallableStatement.execute_for_rows(T4CCallableStatement.java:872)
     at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1162)
     at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2932)
     at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3023)
     at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4132)
     at oracle.sysman.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:645)
     at oracle.sysman.emdrep.jobs.JobEngine.scheduleJob(JobEngine.java:870)
     at oracle.sysman.emdrep.jobs.JobEngine.submitJob(JobEngine.java:778)
     at oracle.sysman.emSDK.job.JobManager.submitJob(JobManager.java:157)
     at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.submitDiscardStateJob(XMLDocumentSplitter.java:2221)
     at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.onEndUpload(XMLDocumentSplitter.java:1748)
     at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.endElement(XMLDocumentSplitter.java:979)
     at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1257)
     at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:314)
     at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:281)
     at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:196)
     at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.split(XMLDocumentSplitter.java:451)
     at oracle.sysman.emdrep.dbjava.loader.XMLLoaderContext.loadFromStream(XMLLoaderContext.java:1672)
     at oracle.sysman.emdrep.dbjava.loader.XMLLoader.DirectLoad(XMLLoader.java:491)
     at oracle.sysman.emdrep.receiver.FxferRecv.DirectLoad(FxferRecv.java:993)
     at oracle.sysman.emdrep.receiver.FxferRecv.processFile(FxferRecv.java:1368)
     at oracle.sysman.emdrep.receiver.FxferRecv.receiveFile(FxferRecv.java:1614)
     at oracle.sysman.emdrep.receiver.FxferRecv.doPost(FxferRecv.java:764)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
     at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
     at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:336)
     at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:663)
     at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:330)
     at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:830)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:224)
     at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:133)
     at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
     at java.lang.Thread.run(Thread.java:534)
2006-11-21 13:14:44,217 [AJPRequestHandler-ApplicationServerThread-12] ERROR audit.AuditManager isAuditOn.157 - Repository may not be up.

tail -50 emagent.trc:
2006-11-18 05:52:10 Thread-28575 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 05:52:10 Thread-28575 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 05:52:10 Thread-28575 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 06:22:15 Thread-28901 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

2006-11-18 06:22:15 Thread-28901 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 06:22:15 Thread-28901 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 06:22:15 Thread-28901 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 06:22:15 Thread-28901 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 07:52:06 Thread-29879 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

2006-11-18 07:52:06 Thread-29879 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 07:52:06 Thread-29879 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 07:52:06 Thread-29879 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 07:52:06 Thread-29879 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 08:52:09 Thread-30532 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

2006-11-18 08:52:09 Thread-30532 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 08:52:09 Thread-30532 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 08:52:09 Thread-30532 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 08:52:09 Thread-30532 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 09:52:10 Thread-31185 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

2006-11-18 09:52:10 Thread-31185 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 09:52:10 Thread-31185 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
. Stopping execution
2006-11-18 09:52:10 Thread-31185 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
2006-11-18 09:52:10 Thread-31185 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

2006-11-18 10:11:23 Thread-31401 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-18 10:11:29 Thread-31404 ERROR collector: Error merging upload on demand files metric belonging to target EnterpriseManager0.isora62.audi.de_HTTP Server
2006-11-18 10:11:30 Thread-31405 ERROR collector: Error merging upload on demand files metric belonging to target EnterpriseManager0.isora62.audi.de_Web Cache
2006-11-18 10:11:44 Thread-31408 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-18 18:01:16 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
2006-11-20 12:46:29 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
2006-11-20 12:46:29 Thread-1 WARN main: failed to start upload manager, disabled
2006-11-20 12:46:39 Thread-32 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-20 12:47:00 Thread-40 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-20 18:17:28 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
2006-11-21 08:55:25 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
2006-11-21 08:55:25 Thread-1 WARN main: failed to start upload manager, disabled
2006-11-21 08:55:34 Thread-31 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-21 08:55:55 Thread-38 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
2006-11-21 11:03:25 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
2006-11-21 11:13:35 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
2006-11-21 11:13:35 Thread-1 WARN main: failed to start upload manager, disabled
2006-11-21 11:13:43 Thread-32 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
Like Show 0 Likes(0)

Actions
8. Re: OMS/sysman/recv directory filled up with xml-files

mnazim-Oracle Nov 21, 2006 3:35 PM (in response to 545934)

Please stop all the opmnctl procees, agent and delete following from agent box
<AOH>\sysman\emd\agntstmp.txt
<AOH>\sysman\emd\lastupload.xml
<AOH>\sysman\emd\upload\*
<AOH>\sysman\emd\recv\
<AOH>\sysman\emd\rstate\*
<AOH>\sysman\emd\collection\*

start all opmn and agent process and let me know the result.
Like Show 0 Likes(0)

Actions
9. Re: OMS/sysman/recv directory filled up with xml-files

545934 Nov 21, 2006 4:00 PM (in response to mnazim-Oracle)

Hi Mnazim,

two questions before i remove files :-)
1) in Oracle DocID 293940.1 it is adviced for a "clean start agent" to delete as well the .../emd/protocol.ini and issue a /emctl clearstate command followed by /emctl upload
should i do that?
2) I found two agents: one in <ORACLE_BASE>/oms10g/sysman/emd which i suppose to be the socalled 'central-agent'
the other one in <ORACLE_BASE>/agent10g/sysman/emd which is monitoring just the node
for which agent should i perform the "clean_up"?

thanks
Zaphod
Like Show 0 Likes(0)

Actions
10. Re: OMS/sysman/recv directory filled up with xml-files

mnazim-Oracle Nov 22, 2006 4:29 PM (in response to 545934)

please clean the second agent only.
Like Show 0 Likes(0)

Actions
11. Re: OMS/sysman/recv directory filled up with xml-files

545934 Nov 27, 2006 1:53 PM (in response to mnazim-Oracle)

didn't work - the number of files is still encreasing

I suppose it cannot be solved by any operation with the AGENT, because it is an issue within the OMS - maybe some kind of misconfiguration (a parameter somewhat like: delete afterupload=true)

best regards
Zaphod
Like Show 0 Likes(0)

Actions
12. Re: OMS/sysman/recv directory filled up with xml-files

277993 Nov 27, 2006 3:56 PM (in response to 545934)

From your log entries I can see:

emoms.trc:

2006-11-21 13:04:49,950 [MetadataLoad http://[b]isora74.audi.de:15746/emd/main/:A0000000002.xml] ERROR XMLLoader.Splitter submitDiscardStateJob.2232 - Exception occurred during DiscardStateJob submission.
java.sql.SQLException: ORA-20224: The following set of target parameters reference an invalid target: (targetNames,targetTypes)

emagent.trc:

2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,[b]isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

Question:

1. What is the difference between isora62.audi.de and isora74.audi.de?
2. Is 15746 the correct port number
Like Show 0 Likes(0)

Actions
13. Re: OMS/sysman/recv directory filled up with xml-files

545934 Nov 27, 2006 7:24 PM (in response to 277993)

Hi Ugonic,
we run a Veritas Cluster System. It consists of two nodes (physical machines) and some thirty Cluster Services (logical machines) which run on one of the nodes and may switch to tho other node (failover).
The Network System is CFS - ClusterFileSystem (which is different to NFS)
isora62 and isora74 are Cluster Services with a logical IP-adress. Therefore each agent needs a separate port. Port 15626 for isora62 and port 15746 for isora74
isora62 is the ClusterService on which the OMS, the OMR and a target-agent are running.
isora74 is a ClusterService on which a database (maybe more) is running.
Like Show 0 Likes(0)

Actions
14. Re: OMS/sysman/recv directory filled up with xml-files

448674 Nov 28, 2006 9:29 AM (in response to 545934)

Hi,
change the line in emd.properties of every agent on the cluster
from
AgentListenOnAllNICs=TRUE
to
AgentListenOnAllNICs=FALSE
then you don't need separate ports if every agent is registered
with a different hostname (logical machines and physical machines) to the OMS.
The only exception is on the OMS host, where you have 2 agents,
one from the iasconsole (OMS OH) and one "add. agent" (extra OH),
which is monitoring the OMS and repository

The error in the agent trace file:
em_error=nmupm child process timed out. Terminated
can be fixed with the following line at the end of emd.properties:
NMUPM_TIMEOUT=60
(increase the value if needed, it's in seconds).
The agent <ORACLE_BASE>/oms10g/sysman/emd should never upload
to the OMS, it is only for the iasconsole.

The files in the recv directory on the OMS should be loaded into the repository
by the Loader of the OMS, or moved to the errors subdirectory (also done by the Loader).
Make sure, that your repository is up, has enough space, and
the OMS is up, running and connected to the repository.
Like Show 0 Likes(0)

Actions

1 2 Previous Next

Go to original post

Join the world’s largest interactive community dedicated to Oracle technologies.