1 2 Previous Next 26 Replies Latest reply on Mar 30, 2007 6:57 PM by mrronn

    OMS/sysman/recv directory filled up with xml-files

    545934
      I suppose there is some misconfiguration in the OMS.
      The Agents deliver their xml-files to the OMS. There they are stored in the $OMS_HOME/sysman/recv directory until they get handled by OMS. Normally you should not see more than 2 or 3 files in that (buffer)-directory.
      I started installing agents on 19.Oct - from then on the xml-files were stored in the recv-directory until the the file-system was filled up to 100%.
      First I stopped all agents and am now looking for advice to:
      - clean OMS (i suppose there are more things to do than just delete the 129.000 files)
      - find the item of misconfiguration

      Hope there is some knowledge out there in the web.

      best regards
      Dieter
        • 1. Re: OMS/sysman/recv directory filled up with xml-files
          277993
          Manually run emctl upload
          • 2. Re: OMS/sysman/recv directory filled up with xml-files
            545934
            Hi Ugonic,
            thank you for the tip, but it don't work.
            As far as I know, the upload-command causes the agent to move the xml-files to OMS. But they are already there.
            oracle@:/oracle/product/102grid/oms10g/sysman/recv # ls -la|wc -l
            130127

            I need to force the OMS to put them to the Repository and delete them subsequently.
            • 3. Re: OMS/sysman/recv directory filled up with xml-files
              277993
              When you tried this did you check whether the number of files reduced?

              You can go to the Agent in Grid Control by select Targets > All Targets, click on the Agent, then under Upload on the screen, check the value of Last Successful Upload and Data Pending Upload to see if they corresponds to the waiting files or the last time you tried the upload.

              Click on Upload Metric Data button. If you get any error that relate to maximum upload files or size, keep clicking it until it uploads all. If it does not work, you may have to increase the maximum allowed upload and size in emd.properties to accomodate that size.

              Also check the log files for relaed errors that will indicate what is happening
              • 4. Re: OMS/sysman/recv directory filled up with xml-files
                mnazim-Oracle
                Please paste the emoms.trc and emagent..trc here
                • 5. Re: OMS/sysman/recv directory filled up with xml-files
                  545934
                  Hello Mnazim,

                  no, the number of files did not decrease after the emctl upload.
                  I'm quite shure that the infomation of the xml-files is proceeded to the repository because the grid control surface shows information which come with the xml-files. Maybe the just are not deleted.

                  How can i attach the trace-files? I think cut and paste is not a good idea because the emagent.trc is 123kB and the emoms.trc is 609kB.
                  Could i send them via e-mail?

                  best regards
                  Zaphod
                  • 6. Re: OMS/sysman/recv directory filled up with xml-files
                    mnazim-Oracle
                    Could you please paste the last 50 lines of both the trc file?
                    • 7. Re: OMS/sysman/recv directory filled up with xml-files
                      545934
                      tail -50 emoms.trc:
                      2006-11-21 13:04:49,950 [MetadataLoad http://isora74.audi.de:15746/emd/main/:A0000000002.xml] ERROR XMLLoader.Splitter submitDiscardStateJob.2232 - Exception occurred during DiscardStateJob submission.
                      java.sql.SQLException: ORA-20224: The following set of target parameters reference an invalid target: (targetNames,targetTypes)
                      ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 14883
                      ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 6877
                      ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7009
                      ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7090
                      ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 7543
                      ORA-06512: at "SYSMAN.MGMT_JOB_ENGINE", line 8387
                      ORA-06512: at "SYSMAN.MGMT_JOBS", line 259
                      ORA-06512: at "SYSMAN.MGMT_JOBS", line 86
                      ORA-06512: at line 1

                           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
                           at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:304)
                           at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:271)
                           at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:625)
                           at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:183)
                           at oracle.jdbc.driver.T4CCallableStatement.execute_for_rows(T4CCallableStatement.java:872)
                           at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1162)
                           at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:2932)
                           at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3023)
                           at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4132)
                           at oracle.sysman.util.jdbc.PreparedStatementWrapper.execute(PreparedStatementWrapper.java:645)
                           at oracle.sysman.emdrep.jobs.JobEngine.scheduleJob(JobEngine.java:870)
                           at oracle.sysman.emdrep.jobs.JobEngine.submitJob(JobEngine.java:778)
                           at oracle.sysman.emSDK.job.JobManager.submitJob(JobManager.java:157)
                           at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.submitDiscardStateJob(XMLDocumentSplitter.java:2221)
                           at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.onEndUpload(XMLDocumentSplitter.java:1748)
                           at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.endElement(XMLDocumentSplitter.java:979)
                           at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1257)
                           at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:314)
                           at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:281)
                           at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:196)
                           at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.split(XMLDocumentSplitter.java:451)
                           at oracle.sysman.emdrep.dbjava.loader.XMLLoaderContext.loadFromStream(XMLLoaderContext.java:1672)
                           at oracle.sysman.emdrep.dbjava.loader.XMLLoader.DirectLoad(XMLLoader.java:491)
                           at oracle.sysman.emdrep.receiver.FxferRecv.DirectLoad(FxferRecv.java:993)
                           at oracle.sysman.emdrep.receiver.FxferRecv.processFile(FxferRecv.java:1368)
                           at oracle.sysman.emdrep.receiver.FxferRecv.receiveFile(FxferRecv.java:1614)
                           at oracle.sysman.emdrep.receiver.FxferRecv.doPost(FxferRecv.java:764)
                           at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
                           at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
                           at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
                           at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:336)
                           at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:663)
                           at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:330)
                           at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:830)
                           at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:224)
                           at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:133)
                           at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:192)
                           at java.lang.Thread.run(Thread.java:534)
                      2006-11-21 13:14:44,217 [AJPRequestHandler-ApplicationServerThread-12] ERROR audit.AuditManager isAuditOn.157 - Repository may not be up.






                      tail -50 emagent.trc:
                      2006-11-18 05:52:10 Thread-28575 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

                      2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 05:52:10 Thread-28575 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
                      . Stopping execution
                      2006-11-18 05:52:10 Thread-28575 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
                      2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 06:22:15 Thread-28901 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

                      2006-11-18 06:22:15 Thread-28901 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 06:22:15 Thread-28901 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
                      . Stopping execution
                      2006-11-18 06:22:15 Thread-28901 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
                      2006-11-18 06:22:15 Thread-28901 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 07:52:06 Thread-29879 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

                      2006-11-18 07:52:06 Thread-29879 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 07:52:06 Thread-29879 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
                      . Stopping execution
                      2006-11-18 07:52:06 Thread-29879 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
                      2006-11-18 07:52:06 Thread-29879 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 08:52:09 Thread-30532 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

                      2006-11-18 08:52:09 Thread-30532 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 08:52:09 Thread-30532 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
                      . Stopping execution
                      2006-11-18 08:52:09 Thread-30532 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
                      2006-11-18 08:52:09 Thread-30532 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 09:52:10 Thread-31185 ERROR fetchlets.oslinetok: Process stdout = em_error=nmupm child process timed out. Terminated

                      2006-11-18 09:52:10 Thread-31185 ERROR engine: [host,isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 09:52:10 Thread-31185 ERROR engine: [isora62.audi.de,host] Failure in step [3] : LoadInternal. emerror=nmupm child process timed out. Terminated
                      . Stopping execution
                      2006-11-18 09:52:10 Thread-31185 ERROR engine: [isora62.audi.de,host] : Load : nmeeam_GetMetricData failed
                      2006-11-18 09:52:10 Thread-31185 ERROR engine: [host,isora62.audi.de,Load] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                      2006-11-18 10:11:23 Thread-31401 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
                      2006-11-18 10:11:29 Thread-31404 ERROR collector: Error merging upload on demand files metric belonging to target EnterpriseManager0.isora62.audi.de_HTTP Server
                      2006-11-18 10:11:30 Thread-31405 ERROR collector: Error merging upload on demand files metric belonging to target EnterpriseManager0.isora62.audi.de_Web Cache
                      2006-11-18 10:11:44 Thread-31408 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
                      2006-11-18 18:01:16 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
                      2006-11-20 12:46:29 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
                      2006-11-20 12:46:29 Thread-1 WARN main: failed to start upload manager, disabled
                      2006-11-20 12:46:39 Thread-32 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
                      2006-11-20 12:47:00 Thread-40 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
                      2006-11-20 18:17:28 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
                      2006-11-21 08:55:25 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
                      2006-11-21 08:55:25 Thread-1 WARN main: failed to start upload manager, disabled
                      2006-11-21 08:55:34 Thread-31 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
                      2006-11-21 08:55:55 Thread-38 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
                      2006-11-21 11:03:25 Thread-3 WARN http: 256,-1: nmehl_httpListener: signaled to exit from emctl
                      2006-11-21 11:13:35 Thread-1 WARN command: Job Subsystem Timeout set at 600 seconds
                      2006-11-21 11:13:35 Thread-1 WARN main: failed to start upload manager, disabled
                      2006-11-21 11:13:43 Thread-32 ERROR collector: Error merging upload on demand files metric belonging to target isora62.audi.de
                      • 8. Re: OMS/sysman/recv directory filled up with xml-files
                        mnazim-Oracle
                        Please stop all the opmnctl procees, agent and delete following from agent box
                        <AOH>\sysman\emd\agntstmp.txt
                        <AOH>\sysman\emd\lastupload.xml
                        <AOH>\sysman\emd\upload\*
                        <AOH>\sysman\emd\recv\
                        <AOH>\sysman\emd\rstate\*
                        <AOH>\sysman\emd\collection\*

                        start all opmn and agent process and let me know the result.
                        • 9. Re: OMS/sysman/recv directory filled up with xml-files
                          545934
                          Hi Mnazim,

                          two questions before i remove files :-)
                          1) in Oracle DocID 293940.1 it is adviced for a "clean start agent" to delete as well the .../emd/protocol.ini and issue a /emctl clearstate command followed by /emctl upload
                          should i do that?
                          2) I found two agents: one in <ORACLE_BASE>/oms10g/sysman/emd which i suppose to be the socalled 'central-agent'
                          the other one in <ORACLE_BASE>/agent10g/sysman/emd which is monitoring just the node
                          for which agent should i perform the "clean_up"?

                          thanks
                          Zaphod
                          • 10. Re: OMS/sysman/recv directory filled up with xml-files
                            mnazim-Oracle
                            please clean the second agent only.
                            • 11. Re: OMS/sysman/recv directory filled up with xml-files
                              545934
                              didn't work - the number of files is still encreasing

                              I suppose it cannot be solved by any operation with the AGENT, because it is an issue within the OMS - maybe some kind of misconfiguration (a parameter somewhat like: delete afterupload=true)

                              best regards
                              Zaphod
                              • 12. Re: OMS/sysman/recv directory filled up with xml-files
                                277993
                                From your log entries I can see:

                                emoms.trc:

                                2006-11-21 13:04:49,950 [MetadataLoad http://[b]isora74.audi.de:15746/emd/main/:A0000000002.xml] ERROR XMLLoader.Splitter submitDiscardStateJob.2232 - Exception occurred during DiscardStateJob submission.
                                java.sql.SQLException: ORA-20224: The following set of target parameters reference an invalid target: (targetNames,targetTypes)

                                emagent.trc:

                                2006-11-18 05:52:10 Thread-28575 ERROR engine: [host,[b]isora62.audi.de,_LoadInternal] : nmeegd_GetMetricData failed : em_error=nmupm child process timed out. Terminated

                                Question:

                                1. What is the difference between isora62.audi.de and isora74.audi.de?
                                2. Is 15746 the correct port number
                                • 13. Re: OMS/sysman/recv directory filled up with xml-files
                                  545934
                                  Hi Ugonic,
                                  we run a Veritas Cluster System. It consists of two nodes (physical machines) and some thirty Cluster Services (logical machines) which run on one of the nodes and may switch to tho other node (failover).
                                  The Network System is CFS - ClusterFileSystem (which is different to NFS)
                                  isora62 and isora74 are Cluster Services with a logical IP-adress. Therefore each agent needs a separate port. Port 15626 for isora62 and port 15746 for isora74
                                  isora62 is the ClusterService on which the OMS, the OMR and a target-agent are running.
                                  isora74 is a ClusterService on which a database (maybe more) is running.
                                  • 14. Re: OMS/sysman/recv directory filled up with xml-files
                                    448674
                                    Hi,
                                    change the line in emd.properties of every agent on the cluster
                                    from
                                    AgentListenOnAllNICs=TRUE
                                    to
                                    AgentListenOnAllNICs=FALSE
                                    then you don't need separate ports if every agent is registered
                                    with a different hostname (logical machines and physical machines) to the OMS.
                                    The only exception is on the OMS host, where you have 2 agents,
                                    one from the iasconsole (OMS OH) and one "add. agent" (extra OH),
                                    which is monitoring the OMS and repository

                                    The error in the agent trace file:
                                    em_error=nmupm child process timed out. Terminated
                                    can be fixed with the following line at the end of emd.properties:
                                    NMUPM_TIMEOUT=60
                                    (increase the value if needed, it's in seconds).
                                    The agent <ORACLE_BASE>/oms10g/sysman/emd should never upload
                                    to the OMS, it is only for the iasconsole.

                                    The files in the recv directory on the OMS should be loaded into the repository
                                    by the Loader of the OMS, or moved to the errors subdirectory (also done by the Loader).
                                    Make sure, that your repository is up, has enough space, and
                                    the OMS is up, running and connected to the repository.
                                    1 2 Previous Next