10 Replies Latest reply on Nov 1, 2007 6:31 PM by 594099

    Errors in emoms.trc

    599718
      Hello getting following erros in emoms.trc can anyone help me to find a reason why this heppens?
      Oms 10.2.0.3 on 2003 server
      Agnet 10.2.0.3 on 2003 server


      2007-10-18 11:53:00,056 [XMLLoader0 80000000327.xml] WARN eml.XMLLoader LoadFiles.682 - Marking the file for retry : 80000000327.xml after receiving exceptionjava.sql.SQLException: ORA-14400: inserted partition key does not map to any partition

      2007-10-18 11:53:05,587 [XMLLoader0 80000000327.xml] ERROR eml.XMLLoader markFileAsError.826 - Failed to load 80000000327.xml even after 2 attempts; moving to errors directory. Error:
      java.sql.SQLException: ORA-14400: inserted partition key does not map to any partition

           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:137)
           at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:304)
           at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:271)
           at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:625)
           at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:181)
           at oracle.jdbc.driver.T4CPreparedStatement.execute_for_rows(T4CPreparedStatement.java:629)
           at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1153)
           at oracle.jdbc.driver.OraclePreparedStatement.sendBatch(OraclePreparedStatement.java:3207)
           at oracle.jdbc.driver.PhysicalConnection.commit(PhysicalConnection.java:1245)
           at oracle.jdbc.OracleConnectionWrapper.commit(OracleConnectionWrapper.java:87)
           at oracle.jdbc.OracleConnectionWrapper.commit(OracleConnectionWrapper.java:87)
           at oracle.sysman.emdrep.dbjava.loader.XMLLoaderContext.handleSubDocument(XMLLoaderContext.java:2272)
           at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.endElement(XMLDocumentSplitter.java:893)
           at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1257)
           at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:314)
           at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:281)
           at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:196)
           at oracle.sysman.emdrep.dbjava.loader.XMLDocumentSplitter.split(XMLDocumentSplitter.java:466)
           at oracle.sysman.emdrep.dbjava.loader.XMLLoaderContext.loadFromStream(XMLLoaderContext.java:1817)
           at oracle.sysman.emdrep.dbjava.loader.XMLLoader.LoadFile(XMLLoader.java:576)
           at oracle.sysman.emdrep.dbjava.loader.XMLLoader.LoadFiles(XMLLoader.java:667)
           at oracle.sysman.emdrep.dbjava.loader.XMLLoader.run(XMLLoader.java:1314)
           at java.lang.Thread.run(Thread.java:534)
        • 1. Re: Errors in emoms.trc
          277993
          When was the last time you run the EM maintenance procedures?
          • 2. Re: Errors in emoms.trc
            599718
            After installation havent done any special maintenance procedures.
            Have 3 agents running ,getting errors only from 1 host.Others are fine.
            • 3. Re: Errors in emoms.trc
              71161
              This may occur if the Repository has not been reorganized and if there are many truncated
              partitions in the Grid Management Repository. SolutionTo fix this issue, please execute the following steps:


              1. Stop the all the Management Agents until the problem is solved.
              $ emctl stop agent
              This step is recommended as this will stop the uploading of new files to the directories
              $ORACLE_HOME/sysman/recv and $ORACLE_HOME/sysman/recv/errors.
              A growing backlog of xml files in the OMS $ORACLE_HOME/sysman/recv sub-directory means a longer time for the OMS to insert the outstanding xml files in the repository. And as long as the upload issue is not fixed, the xml files will continue being moved to the $ORACLE_HOME/sysman/recv/errors sub-directory. Hence even if collected, data moved to the errors sub-directory will never be inserted into the repository, thus making the data collection useless.


              2. Stop all the Management Servers and let the Management Repository Database up and running
              $ emctl stop oms

              3. Connect to the database hosting the Central Management Repository with the SYSMAN account

              4. Execute the following SQL statement
              SQL > exec emd_maintenance.analyze_emd_schema('SYSMAN')
              SQL> exec emd_maintenance.partition_maintenance
              Warning: depending of the amount of work to be done, exec emd_maintenance.partition_maintenance can take a while,
              up to more than one hour. DO NOT cancel this procedure.
              This procedure stops and restartds SYSMAN dbms_jobs. If it is cancelled or does not complete successfully
              for any reason, you can hit the bug:
              Bug 5231347 Running Emd_Maintenance.Partition_Maintenance Deletes Sysman Jobs
              In that case, rerun exec emd_maintenance.partition_maintenance should fix the issue.
              If you cannot run exec emd_maintenance.partition_maintenance and want to fix the dbms_job issue, check
              Note 370695.1 Running EMD_MAINTENANCE.PARTITION_MAINTENANCE removes SYSMAN dbms_jobs



              5. Restart the Management Servers


              6. Restart the Management Agents


              7. Verify that the issue is fixed.

              kind regards
              Terry
              • 4. Re: Errors in emoms.trc
                599718
                Did all steps but problem still exists
                • 5. Re: Errors in emoms.trc
                  599718
                  I checked all xml files in error directory all files and all metrics belongs to one database target, can it be caused by same problem on target database?
                  • 6. Re: Errors in emoms.trc
                    71161
                    Mmmm, I found the solution on metalink and it still looks like te same problem.

                    Could you try to delete the database target in EM wait some minutes and after that add de database target again?
                    • 7. Re: Errors in emoms.trc
                      71161
                      Try this statement in de OMS repository

                      select job
                      , log_user
                      , schema_user
                      , what
                      , total_time
                      , instance
                      from dba_jobs;

                      let me know.

                      Terry
                      • 8. Re: Errors in emoms.trc
                        599718
                        Here what i have


                        JOB     LOG_USER     SCHEMA_USER     WHAT     TOTAL_TIME     INSTANCE
                        61     SYSMAN     SYSMAN     EMD_NOTIFICATION.CHECK_FOR_SEVERITIES();     3     0     
                        62     SYSMAN     SYSMAN     EM_PING.MARK_NODE_STATUS();     7     0     
                        63     SYSMAN     SYSMAN     EM_SEVERITY_REPOS.EXECUTE_REPOS_SEVERITY_EVAL();     0     0     
                        64     SYSMAN     SYSMAN     MGMT_JOB_ENGINE.apply_purge_policies();     4     0     
                        65     SYSMAN     SYSMAN     emd_maintenance.analyze_emd_schema('SYSMAN');     0     0     
                        66     SYSMAN     SYSMAN     EMD_RT_ROLLUP.ROLLUP();     1     0     
                        67     SYSMAN     SYSMAN     EMD_E2E_ROLLUP.ROLLUP();     1     0     
                        68     SYSMAN     SYSMAN     MGMT_GENSVC_AVAIL.EXEC_AVAIL_JOB();     1     0     
                        69     SYSMAN     SYSMAN     EM_PURGE.APPLY_PURGE_POLICIES();     0     0     
                        70     SYSMAN     SYSMAN     MGMT_RCA.PURGE_RCA_RESULTS();     0     0     
                        71     SYSMAN     SYSMAN     BEGIN MGMT_BSLN_INTERNAL.PURGE_EXTRACT_RAWDATA;MGMT_BSLN.COMPUTE_ALL_STATISTICS;/*GC*/END;     0     0     
                        72     SYSMAN     SYSMAN     BEGIN MGMT_BSLN.SET_ALL_THRESHOLDS;/*GC*/END;     1     0     
                        73     SYSMAN     SYSMAN     EM_POLICY.CLEAR_EXPIRED_SUPPRESSIONS;     1     0     
                        58     SYSMAN     SYSMAN     EM_TASK.WORKER(1);     18     0     
                        59     SYSMAN     SYSMAN     EM_TASK.WORKER(2);     2     0     
                        60     SYSMAN     SYSMAN     EMD_LOADER.ROLLUP();     4     0     
                        74     SYSMAN     SYSMAN     EM_TASK.RESUBMIT_FAILED_TASK();     0     0     
                        18     SYS     SYSMAN     MGMT_CNTR_TT.clean_up_old_ticket_recs;     18     0
                        • 9. Re: Errors in emoms.trc
                          599718
                          I have opened xml in$ORACLE_HOME/sysman/recv and $ORACLE_HOME/sysman/recv/errors

                          Here what i have

                          <!--

                               Time: 2007-10-24 17:06:07
                               From: https://bro-ms-tst-01.corp:3872/emd/main/

                          -->

                          <UPLOAD OMS_PROTOCOL_VERSION="10.2.0.2.0" UPLOAD_TYPE="data" EMD_URL="https://bro-ms-tst-01:3872/emd/main/" MERGE_TIMESTAMP="2007-10-24 17:06:07">
                          <ROWSET OMS_PROTOCOL_VERSION="10.2.0.2.0" TABLE="MGMT_METRICS_RAW">
                          <ROW>
                               <TARGET_GUID>813B01FC52A62A682973606DBE969667</TARGET_GUID>
                               <METRIC_GUID>9A5C74B222A4BD75F06702501E2A914F</METRIC_GUID>
                               <KEY_VALUE/>
                               <COLLECTION_TIMESTAMP>2007-10-24 16:51:22</COLLECTION_TIMESTAMP>
                               <VALUE>0.4</VALUE>
                          </ROW>
                          <ROW>
                               <TARGET_GUID>813B01FC52A62A682973606DBE969667</TARGET_GUID>
                               <METRIC_GUID>D64CA8B131AB24FED935E0ED99C684CB</METRIC_GUID>
                               <KEY_VALUE/>
                               <COLLECTION_TIMESTAMP>2007-10-24 16:51:22</COLLECTION_TIMESTAMP>
                               <VALUE>0</VALUE>
                          </ROW>

                          Agent collecting statistics in wrong format without key.I What can cause this problem?
                          Thanks
                          • 10. Re: Errors in emoms.trc
                            594099
                            Have you tried this process? I had a similar problem and this resolved it and allowed my agents to upload to the repository.

                            a.) Stop the OMS (all in case you have multiple OMS)
                            OMS_HOME/opmn/bin/opmnctl stopall

                            b.) Stop the Repository database and listener.

                            c.) Startup the Repository database and listener.

                            d.) Start the OMS
                            OMS_HOME/opmn/bin/opmnctl startall


                            Bobby