12 Replies Latest reply on Apr 16, 2019 1:37 PM by lindafrancis

    Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.

    Sreek-Oracle

      Greetings All,

      I got a filesystem, showing up as a warning in OEM (showing as 84% full). Re-evaluate the incident is telling me the incident is still valid, but In reality on the filesystem itself it's only 69% full.

       

      irmn01sn01-dt-ovs.fiab.local:/export/DEV-FAH/u01_fidmdevapp8app9
                            180G  123G   58G  69% /u01   

       

      oracle@fah-idm1 bin]$ ./emctl listplugins agent
      Oracle Enterprise Manager Cloud Control 12c Release 5
      Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
      ---------------------------------------------------------------
      oracle.sysman.oh    12.1.0.5.0           /emagent/oem1/plugins/oracle.sysman.oh.agent.plugin_12.1.0.5.0
      oracle.sysman.emas  12.1.0.8.0           /emagent/oem1/plugins/oracle.sysman.emas.agent.plugin_12.1.0.8.0

      [oracle@fah-idm1 bin]$ ./emctl status agent
      Oracle Enterprise Manager Cloud Control 12c Release 5
      Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
      ---------------------------------------------------------------
      Agent Version          : 12.1.0.5.0
      OMS Version            : 12.1.0.5.0
      Protocol Version       : 12.1.0.1.0
      Agent Home             : /emagent/oem1/agent_inst
      Agent Log Directory    : /emagent/oem1/agent_inst/sysman/log
      Agent Binaries         : /emagent/oem1/core/12.1.0.5.0
      Agent Process ID       : 2300
      Parent Process ID      : 2230
      Agent URL              : https://fidmdevapp8.fiab.local:3872/emd/main/
      Local Agent URL in NAT : https://fidmdevapp8.fiab.local:3872/emd/main/
      Repository URL         : https://oem.figs.ing.intranet:4900/empbs/upload
      Started at             : 2018-12-11 16:46:17
      Started by user        : oracle
      Operating System       : Linux version 2.6.39-400.297.3.el5uek (amd64)
      Last Reload            : (none)
      Last successful upload                       : 2019-02-14 07:50:03
      Last attempted upload                        : 2019-02-14 07:50:03
      Total Megabytes of XML files uploaded so far : 29.15
      Number of XML files pending upload           : 0
      Size of XML files pending upload(MB)         : 0
      Available disk space on upload filesystem    : 89.82%
      Collection Status                            : Collections enabled
      Heartbeat Status                             : Ok
      Last attempted heartbeat to OMS              : 2019-02-14 07:54:29
      Last successful heartbeat to OMS             : 2019-02-14 07:54:29
      Next scheduled heartbeat to OMS              : 2019-02-14 07:55:29

      ---------------------------------------------------------------
      Agent is Running and Ready
      [oracle@fah-idm1 bin]$ ./emctl config agent listtargets
      Oracle Enterprise Manager Cloud Control 12c Release 5
      Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
      [fidmdevapp8.fiab.local, host]
      [fidmdevapp8.fiab.local:3872, oracle_emd]
      [agent12c1_1_fidmdevapp8.fiab.local, oracle_home]
      [fidmdevapp7.fiab.local:3872_proxy, oracle_emd_proxy]
      [oracle@fah-idm1 bin]$

       

      Can this be a ZFSSA plugin issue ?

       

        • 1. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
          Ian Baugaard

          Hi,

           

          It seems (To me) as though you're looking at two different things. Check the metrics for the fah-idm1 host "Host -> Monitoring -> All Metrics -> Filesystems -> Filesystem Space Available (%)".

           

          The values for /export/DEV-FAH/u01_fidmdevapp8app9 should match up with what you're seeing at the OS level.

           

          Your first screenshot shows storage utilization at the ZFS pool level, which isn't necessarily the same as the Filesystem Utilization you're referring to.

           

          Hope this helps

           

          Ian

          • 2. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
            Sreek-Oracle

            Greetings Ian,

            Appreciate your response  - I can understand your standpoint that  df doesn't understand descendent datasets or whether snapshots exist. i will get back you in a bit..

            • 3. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
              Sreek-Oracle

              Greetings Ian,

              NAME                                                 USED  AVAIL  REFER  MOUNTPOINT
              exalogic/local/DEV-OFSAA/u01_dfgs2ovm03              258G  72.2G   228G  /export/DEV-OFSAA/u01_dfgs2ovm03   <<<<<< 72GB Availale

               

               

              so on client :

              Filesystem    Size  Used Avail Use% Mounted on
              172.22.8.17:/export/DEV-OFSAA/u01_dfgs2ovm03                 300G  228G   73G  76% /u01

               

              D                                                                                              603892
              Metric                                                                                         SpacePercentageUtilized
              Metric Group                                                                                LogicalStatus
              ShareID                                                                                       exalogic:DEV-OFSAA/u01_dfgs2ovm03
              Target                                                                                         drel01sn01.fiab.local(Oracle ZFS Storage Appliance)Target Information
              Incident Created                                                                           20-Feb-2019 12:04:45 CET
              Last Updated                                                                                20-Feb-2019 12:04:45 CET
              Summary             Share (exalogic:DEV-OFSAA/u01_dfgs2ovm03)      Percentage Used is 80.47 and has crossed warning (80) or critical(90) threshold.
              Internal Event Name                                                                     LogicalStatus:SpacePercentageUtilized
              Event Type                                                                                  Metric Alert
              Category                                                                                      Capacity

               

              So its the agent that showing incorrect value isn't

              Kind Regards

               

              • 4. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                Ian Baugaard

                I would still disagree, as you're trying to do an apples-to-apples comparison of a host metric and a ZFS metric.

                 

                What are the results for the following query against the repository database?

                 

                SELECT 
                mc.target_name
                , mc.key_value
                , ROUND( 100 - mc.value, 2 ) fs_used_perc
                , mc.collection_timestamp
                FROM sysman.mgmt$metric_current mc
                WHERE mc.target_type = 'host' 
                AND mc.metric_name = 'Filesystems' 
                AND mc.metric_column = 'pctAvailable' 
                AND mc.target_name = 'fah-idm1' ;
                
                

                 

                Regards


                Ian

                • 5. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                  Sreek-Oracle

                  Ian, can you pls guide me where to execute this ?  on agent or on OMS ?

                  • 6. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                    Ian Baugaard

                    Hi,

                     

                    You execute that against the OMS repository database, as either the SYSMAN or some other privileged user

                     

                    Regards

                     

                    Ian

                    • 7. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                      Sreek-Oracle

                      Greetings Ian,

                       

                      SQL> SELECT mc.target_name, mc.key_value, ROUND( 100 - mc.value, 2 ) fs_used_perc, mc.collection_timestamp FROM sysman.mgmt$metric_current mc WHERE mc.target_type = 'host'
                      AND mc.metric_name = 'Filesystems' AND mc.metric_column = 'pctAvailable' AND mc.target_name = 'fah-idm1' ;  

                      no rows selected

                      SQL>

                       

                       

                      • 8. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                        Ian Baugaard

                        Hi,

                         

                        Just make sure that the host name matches the name as discovered in OEM. I'm expecting the value returned here to match the output from the df command.

                         

                        Ian

                        • 9. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                          Sreek-Oracle


                          SQL> SQL> SELECT
                          mc.target_name
                          , mc.key_value
                          , ROUND( 100 - mc.value, 2 ) fs_used_perc
                          , mc.collection_timestamp
                          FROM sysman.mgmt$metric_current mc
                          WHERE mc.target_type = 'host'
                          AND mc.metric_name = 'Filesystems'
                          AND mc.metric_column = 'pctAvailable'
                          AND mc.target_name = 'ardm01db04.fiab.local' ;  2    3    4    5    6    7    8    9   10

                          TARGET_NAME               KEY_VALUE                 FS_USED_PERC COLLECTIO
                          ------------------------- ------------------------- ------------ ---------
                          ardm01db04.fiab.local     /                                23.64 13-MAR-19
                          ardm01db04.fiab.local     /boot                             7.99 13-MAR-19
                          ardm01db04.fiab.local     /u01                             87.76 13-MAR-19

                           

                          But the query works now, with the proper hostname in it.

                          Currently everything looks okay . we are continuing to monitor the same.

                          • 10. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                            Sreek-Oracle

                            This seems to be intermittent issue,   valuse showing up incorrectly 85%
                            ++++++++  OEM INCIDENT ++++++++
                            ID 633133
                            Metric Group Recovery Area
                            Metric Recovery Area Free Space (%)
                            Target dfgs2o (Cluster Database)

                            Incident Created Apr 1, 2019 3:06:04 PM CEST
                            Last Updated Apr 10, 2019 3:03:10 PM CEST
                            Summary db_recovery_file_dest_size of 1181116006400 bytes is 85.29% used and has 173686128640 remaining bytes available.
                            Internal Event Name Recovery_Area:Free_Space

                            Last Known Value                           84.404
                            Last Collection Timestamp          Apr 11, 2019 2:47:03 PM CEST
                            ++++++++++++++++

                            ++++++++++++++ ACTUAL SQL QUERY  ++++++++++++
                            SQL> select * from v$recovery_area_usage;

                            FILE_TYPE            PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES
                            -------------------- ------------------ ------------------------- ---------------
                            CONTROL FILE                          0                         0               0
                            REDO LOG                              0                         0               0
                            ARCHIVED LOG                      17.44                     10.57             103
                            BACKUP PIECE                          0                         0               0
                            IMAGE COPY                            0                         0               0
                            FLASHBACK LOG                     13.82                      5.09              38
                            FOREIGN ARCHIVED LOG                  0                         0               0

                            7 rows selected.

                            SQL> show parameter db_recovery_file_dest_size

                            NAME                                 TYPE                             VALUE
                            ------------------------------------ -------------------------------- ------------------------------
                            db_recovery_file_dest_size           big integer                      1100G
                            SQL>

                            • 11. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                              Sreek-Oracle

                              Ian,

                              Query only shows LOCAL filesystems, not ZFS filesystems how can we modify to grab nfs share aswell

                               

                              Example. Your query against the OEM database:

                              mc.target_name
                              , mc.key_value
                              , ROUND( 100 - mc.value, 2 ) fs_used_perc
                              , mc.collection_timestamp
                              FROM sysman.mgmt$metric_current mc
                              WHERE mc.target_type = 'host'
                              AND mc.metric_name = 'Filesystems'
                              AND mc.metric_column = 'pctAvailable'
                              AND mc.target_name = 'dfgs51vm01.fiab.local' ;

                              Output:
                              dfgs51vm01.fiab.local / 11,7 15-04-19
                              dfgs51vm01.fiab.local /boot 13,5 15-04-19
                              dfgs51vm01.fiab.local /emagent 24,74 15-04-19

                              Customer query is  against the OEM database:

                              select mc.target_name
                              , mc.key_value
                              , mc.value
                              , mc.collection_timestamp
                              ,mc.metric_column
                              FROM sysman.mgmt$metric_current mc
                              WHERE mc.target_type = 'sun_storage_7000'
                              AND mc.metric_column = 'SpacePercentageUtilized'
                              AND mc.target_name  LIKE 'drel01sn01%'
                              and mc.key_value like '%dfgs51vm01%';

                              Output:
                              drel01sn01.fiab.local exalogic:DEV-OBIEE/u01_dfgs51vm01 80 14-04-19 SpacePercentageUtilized

                              ZFS (nfs share command output)
                              drel01sn01:> confirm shell zfs list -t all -o name,volsize,used,referenced,reservation,refreservation,usedbyrefreservation,usedbysnapshots | grep dfgs51vm01
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01                                                                         -   193G   165G    220G       none              0     27.6G
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01@.auto-dev-obiee_-20190407T231500UTC                                     -   342M   172G       -          -              -         -
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01@.auto-dev-obiee_-20190408T231500UTC                                     -   301M   172G       -          -              -         -
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01@.auto-dev-obiee_-20190409T231500UTC                                     -   416M   189G       -          -              -         -
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01@.auto-dev-obiee_-20190410T231500UTC                                     -   253M   189G       -          -              -         -
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01@.auto-dev-obiee_-20190411T231500UTC                                     -   255M   190G       -          -              -         -
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01@.auto-dev-obiee_-20190412T231500UTC                                     -   170M   165G       -          -              -         -
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01@.auto-dev-obiee_-20190413T231500UTC                                     -   167M   165G       -          -              -         -
                              exalogic/local/DEV-OBIEE/u01_dfgs51vm01@.auto-dev-obiee_-20190414T231500UTC                                     -   135M   165G       -          -              -         -
                              On the OS of this host dfgs51vm01:
                              172.22.8.17:/export/DEV-OBIEE/u01_dfgs51vm01
                                                    220G  166G   55G  76% /u01


                              So OEM shows 4% higher than the actual usage on the host itself.

                              • 12. Re: Warning in OEM (showing as 84% full) but on the filesystem itself it's only 69% full.
                                lindafrancis

                                I think your suggestion would be helpful for me. I will let you know if this works for me Thanks and keep posting such a informative forum.

                                I love Oracle