This discussion is archived
10 Replies Latest reply: Jan 16, 2013 7:50 AM by JeriF RSS

Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:

JeriF Newbie
Currently Being Moderated
I'm looking for a solution to the following error with a couple of targets in Grid Control 11g. "Error getting data for target <machinename>. Exception: oracle.sysman.emSDK.emd.comm.MetricGetException: Result has repeating key value : 0 C:".

This is a new GridControl 11g installation. GC installation itself is on 32-bit Windows 2008 server with target hosts/agents being Windows 2008 x64. Most GC targets are successfully sending CPU, Disk I/O and Memory performance data, with the exception of 3 machines which throw the above error. Commonality with these 3 machines is that they all have SAN connectivity. I've tried applying to one of the machines the Windows MPIO hotfix (http://support.microsoft.com/kb/974878/EN-US) as noted in a OMS v10 post I found here but it did not resolve the problem.

I also saw mention of adjusting collection metrics to solve issues with specific data collections but since these machines pull no performance data at all- CPU, disk, or memory, I'm unclear which metric to adjust. These 3 machines seem to provide all other expected data to GC.

Where to look?

TIA.
  • 1. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    Loc Nhan Expert
    Currently Being Moderated
    Agent trace/log files (e.g. <agent_home>/sysman/log/emagent.trc) should show what metrics cause the error.

    - Loc
  • 2. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    JeriF Newbie
    Currently Being Moderated
    Perfect. Thanks, Loc.
  • 3. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    JeriF Newbie
    Currently Being Moderated
    Newbie blues. I've been looking at this log and have searched on a few of the keywords to no avail. Poked around in the console and finally found where the metric and policy settings are located and noted the two metrics with Collection Error status (Disk Activity and Load) but, the individual metrics in those groups look okay (none empty, and have reasonable values set).

    Here's the tail of the log; if someone could shed some light I'd be grateful. FWIW, the disk/memory/IO failures have been there since initial agent install (and prior to adding the SQL agent).

    Thx, Jeri.


    2013-01-11 11:23:54,198 Thread-7040 WARN collector: <nmecmc.c> Error exit. Error message: Result has repeating key value : 0 C:
    2013-01-11 11:28:54,268 Thread-6416 ERROR engine: [host,FDS9.hcsad.LOCAL,DiskActivity] : nmeegd_GetMetricData failed : Result has repeating key value : 0 C:
    2013-01-11 11:28:54,282 Thread-6416 ERROR engine: [FDS9.hcsad.LOCAL,host] Failure in step [0] : DiskActivity. Result has repeating key value : 0 C:. Stopping execution
    2013-01-11 11:28:54,298 Thread-6416 ERROR engine: [FDS9.hcsad.LOCAL,host] : Load : nmeeam_GetMetricData failed
    2013-01-11 11:28:54,314 Thread-6416 ERROR engine: [host,FDS9.hcsad.LOCAL,Load] : nmeegd_GetMetricData failed : Result has repeating key value : 0 C:
    2013-01-11 11:28:54,336 Thread-6416 WARN collector: <nmecmc.c> Error exit. Error message: Result has repeating key value : 0 C:
    2013-01-11 11:30:29,703 Thread-6312 WARN engine: cannot find metricColl for [FDS9 SQL, microsoft_sqlserver_database, DB_freespace]
    2013-01-11 11:33:54,565 Thread-3896 ERROR engine: [host,FDS9.hcsad.LOCAL,DiskActivity] : nmeegd_GetMetricData failed : Result has repeating key value : 0 C:
    2013-01-11 11:33:54,587 Thread-3896 ERROR engine: [FDS9.hcsad.LOCAL,host] Failure in step [0] : DiskActivity. Result has repeating key value : 0 C:. Stopping execution
    2013-01-11 11:33:54,605 Thread-3896 ERROR engine: [FDS9.hcsad.LOCAL,host] : Load : nmeeam_GetMetricData failed
    2013-01-11 11:33:54,623 Thread-3896 ERROR engine: [host,FDS9.hcsad.LOCAL,Load] : nmeegd_GetMetricData failed : Result has repeating key value : 0 C:
    2013-01-11 11:33:54,641 Thread-3896 WARN collector: <nmecmc.c> Error exit. Error message: Result has repeating key value : 0 C:
    2013-01-11 11:35:57,042 Thread-5820 ERROR engine: [host,FDS9.hcsad.LOCAL,DiskActivity] : nmeegd_GetMetricData failed : Result has repeating key value : 0 C:
    2013-01-11 11:35:57,059 Thread-5820 WARN collector: <nmecmc.c> Error exit. Error message: Result has repeating key value : 0 C:
  • 4. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    EricvdS Expert
    Currently Being Moderated
    I had the same issue once and after applying the mentioned hotfix the issue was solved.
    In the Performance Monitor window the storage devices are now named correctly? If not I think the hotfix was not applied correctly.

    Eric
  • 5. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    JeriF Newbie
    Currently Being Moderated
    Thanks for the thought Eric. I'll keep looking.

    Still, I can't help but wonder about the fact that GC can't access CPU or memory either on any of these 3 machines... seems to me to be a clue that this isn't entirely a MPIO issue.

    Edited by: JeriF on Jan 14, 2013 7:04 AM
  • 6. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    EricvdS Expert
    Currently Being Moderated
    Your correct, I didn't read about the cpu and memory issues. Since it is occurs only on one server maybe reinstalling the agent is the answer.
    But are the storage devices are now named correctly?

    Eric
  • 7. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    JeriF Newbie
    Currently Being Moderated
    No, I'm pretty much convinced now the MPIO is indeed the root issue. And its present, along with the memory and CPU collection problem, on 3 machines. And all 3 machines have SAN connectivity; I had just tried applying the hotfix on one of them.

    I don't generally look for potential disk performance issues from PerfMon though so had never noticed the disk anomoly there before. Now that I've seen that (as a result of this effort), I wonder if GC is trying to get all of these simple stats (memory, cpu, disk) from the same Windows source so when it dies on retrieving disk data, the rest of the query for memory and CPU fails too as a result. Dunno.

    I suspect you're right though... that even though all 3 machines have MPIO installed (a default in 2008 but not 2008 R2?), its not configured properly. I'm hesitant to mess with it too much since PowerPath and the other tools I use are currently happy. And then to ice the cake, these 3 machines are scheduled to be virtualized within the next week or so, which means significant backend changes, so this may turn out to be a non-issue issue (ie., a waste of time).

    I had hoped to pull some performance benchmark data with GC prior to virtualizing these machines, especially since they're SQL, but now I'm thinking it may be better to just sit tight. We ran some tests with OATS today and can do some pre/post virtualization performance comparisons that way, along with the stats I pulled from MSSQL itself while those tests ran. Certainly better than nothing.

    Of course, if you're a MPIO-config guru and want to tell me how to configure it, I'm most happy to listen:) Everything I found googling about it this morning appeared to be 2008 R2 specific (the switches are different in plain ol' 2008), and finally the non-issue issue potential of MPIO convinced me to just run tests from OATS instead.
  • 8. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    EricvdS Expert
    Currently Being Moderated
    Thanks for the update.

    Sorry, I'm no MPIO expert, so I can't help you I that department.

    For what it is worth. We have our Oracle db's (on W2008R2) not virtualized and see no I/O-issues. SQL-Server is however virtualized and they do have I/O-issues. Changing the queue depth did give some relief.
    For Oracle licence reasons we have our Weblogic and AS virtual servers limited only to a small number of hosts they can run on.

    Eric
  • 9. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    JeriF Newbie
    Currently Being Moderated
    That's a handy hint on virt MSSQL. Thanks. I'm expecting exactly that kind of fun with virtualizing it, particularly given that my current physical architecture utilizes high spindle counts. Having known physical performance benchmarks will go a long way in helping me address the potential for general 'seems slower' complaints.
  • 10. Re: Exception: oracle.sysman.emSDK.emd.comm.MetricGetException:
    JeriF Newbie
    Currently Being Moderated
    Quick update in case someone else views this thread in the future. The issue turned out to be related to the version of PowerPath (EMC). So, even though the MS hotfix was applied and should have fixed the problem (it actually fixed all drives but C), the error continued due to PowerPath. Fix then became to upgrade PowerPath.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points