0 Replies Latest reply on Aug 18, 2020 1:07 PM by Venkata Thiruveedhi-Oracle

    Understanding Always on Monitoring(AOM) and Configuring AOM in 13c Cloud Control

    Venkata Thiruveedhi-Oracle

      AOM (Always on Monitoring) is useful for the EM Administrators to send email notifications to the users even when the OMS itself is down. The Enterprise Manager Always-On Monitoring provides the ability to monitor critical target status and metric alerts. The service continuously monitors critical targets through the Enterprise Manager Agent and can be easily configured to send email notifications for these events to administrators.

       

      Always-On Monitoring can be configured to send notifications at any time, but is particularly useful when experiencing downtime of 13c Cloud Control OMS or repository database for maintenance operations such as upgrade and patching.  The Always-On Monitoring is synchronized with Enterprise Manager to reuse the configuration of monitored targets as well as requisite notification data such as notification contacts and email gateway configuration. Once properly configured and synchronized, the service will receive alerts from Enterprise Manager Agents and send email notifications to the appropriate administrators.

      Steps to use AOM:

      1. Installing the Always-On Monitoring(AOM)

      2. Configuring the Always-On Monitoring (AOM)

       

       

      1. Installing the Always-On Monitoring (AOM)

      a. Prerequisites for installation of AOM

      1. Create/Install a new database to store the EMS schema on target machine where AOM needs to be configured

      2. Copy emkey to the 13c OEM repository database as below:

      <MIDDLEWARE_HOME>/bin>./emctl config emkey -copy_to_repos
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      Enter Enterprise Manager Root (SYSMAN) Password :

      The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
      After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

      3. Ensure Java version 1.7 is available on the machine where AOM will be installed

      4. SMTP Mail server details needs to be configured in 13c OEM console

      b. Installation of Always-On Monitoring(AOM)

      1. Copy the AOM software (ems_13.1.0.0.0.zip) from <MIDDLEWARE_HOME>/sysman/ems on the OMS machine to the target machine where AOM will be installed
      2. Unzip the file ems_13.1.0.0.0.zip on the target machine, this creates a ems directory with the following content:

      [oracle@venkat ems]$ pwd
      /u02/em13c/ems/ems
      [oracle@venkat ems]$ ls -ltr
      total 1572
      drwxr-xr-x. 4 oracle oinstall 4096 Dec 12 15:06 sql
      -rw-r--r--. 1 oracle oinstall 1596182 Dec 12 17:36 ems_13.1.0.0.0.jar
      drwxr-xr-x. 2 oracle oinstall 4096 Dec 12 17:36 lib
      drwxr-xr-x. 2 oracle oinstall 4096 Dec 12 17:36 scripts

       

      2. Configuring the Always-On Monitoring (AOM)

      a. Configuring  Always-On monitoring (AOM)

      1. set JAVA_HOME to Java 1.7

                Example: export JAVA_HOME=/u02/em13c/middleware/oracle_common/jdk

      2. Run emsca script to configure AOM as below:

                $ cd <AOM_HOME>/scripts
                $ ./emsca

       

                 emsca script will prompt for following details:
                 Always-On Monitoring Repository Connection String
                 Create Always-On Monitoring Repository User [n]
                 Always-On Monitoring Repository SYSDBA Username :
                 Always-On Monitoring Repository SYSDBA Password :
                 Always-On Monitoring Repository Username [ems] :
                 Always-On Monitoring Repository Password [ems] :
                 Enterprise Manager Repository Connection String :
                 Enterprise Manager Repository Username : sysman
                 Enterprise Manager Repository Password :

      Example:

      $ cd <AOM_HOME>/scripts
      $ ./emsca
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ---------------------------------------------------------------
      Always-On Monitoring Repository Connection String : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
      Create Always-On Monitoring Repository User [n] : y
      Always-On Monitoring Repository SYSDBA Username : sys
      Always-On Monitoring Repository SYSDBA Password :
      Always-On Monitoring Repository Username [ems] :
      Always-On Monitoring Repository Password [ems] :
      Enterprise Manager Repository Connection String : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
      Enterprise Manager Repository Username : sysman
      Enterprise Manager Repository Password :
      Creating Always-On Monitoring repository user ems
      Enter Enterprise Manager Middleware Home : /u02/em13c/middleware
      Connecting to Always-On Monitoring repository.
      Creating Always-On Monitoring repository schema
      Creating repository storage for Targets data.
      Creating repository storage for Alerts and Availability data.
      Creating repository storage for Notification Metadata data.
      Creating repository storage for Target Metric Metadata data.
      Registering Always-On Monitoring instance
      Always-On Monitoring Upload URL: https://venkat.idc.oracle.com:8081/upload
      Oracle PKI Tool : Version 12.1.3.0.0
      Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.

      Certificate was added to keystore

      This creates ems schema inside the database on target machine to store the AOM data

      b. AOM Post Configuration Tasks

      1. Remove emkey from 13c OEM repository database
               <OMS_HOME>/bin>./emctl config emkey -remove_from_repos

      2. By default, AOM will not be running. Before starting the AOM for the first time, need to perform sync operation so that the data in 13c OEM repository database is in sync with AOM repository database.

      $ cd <AOM_HOME>/scripts
      $ ./emsctl sync
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Connecting to Always-On Monitoring repository.
      Starting synchronization with EM.
      Synchronizing with EM repository: sysman@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
      Synchronizing Targets data.
      Synchronizing Alerts and Availability data.
      Synchronizing Notification Metadata data.
      Synchronizing Target Metric Metadata data.
      Synchronization complete at : Mon Dec 21 07:17:06 EST 2015

      3. Once sync is completed successfully, start the AOM as below:

      $ cd <AOM_HOME>/scripts
      $ ./emsctl start
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Starting Always-On Monitoring.
      Notifications Enabled : false
      Total Downtime Contacts Configured : 0
      Always-On Monitoring is up.

      4. Verify the AOM status as below:

      $ cd <AOM_HOME>/scripts
      $ ./emsctl status
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Always-On Monitoring Version : 13.1.0.0.0
      Always-On Monitoring Home : /u02/em13c/ems/ems
      Started At : December 21, 2015 7:25:37 AM EST
      Last Repository Sync : December 21, 2015 7:15:48 AM EST
      Upload URL : https://venkat.idc.oracle.com:8081/upload
      Always-On Monitoring Process ID : 27549
      Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
      Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
      Notifications Enabled : false
      Total Downtime Contacts Configured : 0

      5. Set the AOM upload URL on OMS side so that agent uploads information to the AOM repository database:

      $ cd <MIDDLEWARE_HOME>/bin
      $./emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value " https://venkat.idc.oracle.com:8081/upload"
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      SYSMAN password:
      Property oracle.sysman.core.events.ems.emsURL has been set to value https://venkat.idc.oracle.com:8081/upload for all Management Servers
      OMS restart is not required to reflect the new property value


      6. Configure Downtime Contacts on 13c OEM side:

      Downtime contacts can be configured from any of the following methods:

      a. For each target, navigate to the Target Properties page, which can be accessed from the target's homepage. From the target menu on the target homepage, select Target Setup and then Properties. The Target Properties page displays.

               •  Click on Edit and specify the email address in the Downtime Contact target property. You can specify multiple email addresses by separating them with commas.
               • Use the EM CLI set_target_property_value verb.
                      emcli set_target_property_value -property_records="target_name:target_type:property_name:property_value"

      Example:
      $ ./emcli set_oms_property -property_name='oracle.sysman.core.events.ems.downtimeContact' -property_value='venkatapraveent@gmail.com'
      Property "oracle.sysman.core.events.ems.downtimeContact" has been successfully set to value "venkatapraveent@gmail.com" on all Management Servers.
      Server restart is not required for the property change to get reflected


      b. Per-target downtime contact based on event rules:

                Always-On Monitoring may also send email notifications to different users for each target. These contacts are generated in Enterprise Manager based on the event rules for that target. Therefore, as event rules are changed in Enterprise Manager, the contacts must be re-generated and an incremental synchronization performed. By leveraging the event rule setup, the downtime contact will be generated based on the email recipient for the event rule for a Target Availability event type where Down status has been selected.

       

      Note:
      Although downtime contacts are generating using only Target Availability event rules, Always-On Monitoring will send notifications for both target availability and metric threshold alerts.
      You can review and update the recipients of your target availability (status down) event rules. Doing so allows you to generate a list of downtime contacts using EM CLI or by submitting the downtime contact generation job.


      Once downtime contacts are configured, perform sync on AOM to update the AOM repository database with downtime contact information.

      Example:
      $ cd <AOM_HOME>/scripts
      $ ./emsctl sync
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Connecting to Always-On Monitoring repository.
      Starting synchronization with EM.
      Synchronizing with EM repository: sysman@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
      Synchronizing Targets data.
      Synchronizing Alerts and Availability data.
      Synchronizing Notification Metadata data.
      Synchronizing Target Metric Metadata data.
      Synchronization complete at : Mon Dec 21 07:52:01 EST 2015


      Verify the total downtime contacts value in AOM status output

      $ cd <AOM_HOME>/scripts
      $./emsctl status
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Always-On Monitoring Version : 13.1.0.0.0
      Always-On Monitoring Home : /u02/em13c/ems/ems
      Started At : December 21, 2015 7:25:37 AM EST
      Last Repository Sync : December 21, 2015 7:51:59 AM EST
      Upload URL : https://venkat.idc.oracle.com:8081/upload
      Always-On Monitoring Process ID : 27549
      Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
      Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
      Notifications Enabled : false
      Total Downtime Contacts Configured : 2


      7. By default notifications are disabled, need to enable the notifications for the AOM to send email notifications to the downtime contacts configured in step above. Enable Notifications option on AOM as below:

      [oracle@venkat scripts]$ ./emsctl enable_notification
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Notifications have been enabled. There are downtime contacts configured.
      Connecting to Always-On Monitoring repository.
      Starting synchronization with EM.
      Synchronizing with EM repository: sysman@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
      Synchronizing Targets data.
      Synchronizing Alerts and Availability data.
      Synchronizing Notification Metadata data.
      Synchronizing Target Metric Metadata data.
      Synchronization complete at : Mon Dec 21 07:56:54 EST 2015


      Verify the notification status from the AOM status output:

      [oracle@venkat scripts]$ ./emsctl status
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Always-On Monitoring Version : 13.1.0.0.0
      Always-On Monitoring Home : /u02/em13c/ems/ems
      Started At : December 21, 2015 7:25:37 AM EST
      Last Repository Sync : December 21, 2015 7:56:53 AM EST
      Upload URL : https://venkat.idc.oracle.com:8081/upload
      Always-On Monitoring Process ID : 27549
      Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
      Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
      Notifications Enabled : true
      Total Downtime Contacts Configured : 2


      8. Trigger any metric alert for a target for which downtime contacts are configured and verify if the email alert is received.

      Sample email alert:

      Subject: EM Event: Warning:em13cdb.idc.oracle.com - User SYS logged on from venkat.idc.oracle.com.

      Email Body:

      Target type=Database Instance
      Target name=em13cdb.idc.oracle.com
      Message=User SYS logged on from venkat.idc.oracle.com.
      Severity=Warning
      Event reported time=Dec 22, 2015 8:09:24 AM EST
      Event Type=Metric Alert
      Event name=UserAudit:username
      Metric Group=User Audit
      Metric=Audited User
      Key Value=SYS_venkat.idc.oracle.com
      Sent by Oracle Enterprise Manager - Always-On Monitoring


      9. To disable notification on AOM side, run the following command:

      [oracle@venkat scripts]$ ./emsctl disable_notification
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Notifications have been disabled.

       

      [oracle@venkat scripts]$ ./emsctl status
      Oracle Enterprise Manager Cloud Control 13c Release 1
      Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
      ------------------------------------------------------------------
      Always-On Monitoring Version : 13.1.0.0.0
      Always-On Monitoring Home : /u02/em13c/ems/ems
      Started At : December 21, 2015 7:25:37 AM EST
      Last Repository Sync : December 21, 2015 7:56:53 AM EST
      Upload URL : https://venkat.idc.oracle.com:8081/upload
      Always-On Monitoring Process ID : 27549
      Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
      Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
      Notifications Enabled : false
      Total Downtime Contacts Configured : 2


      Log Location

      Log files record Always-On Monitoring events that occur during operation and are generated as follows:
              • emsca logs: emsca.err (only errors), emsca.log.0 (rotating log file that contains all output including errors).
              • ems logs: ems.err (only errors), ems.log.0 (rotating log file that contains all output including errors).
      These files are located in the $EMS_HOME/logs directory.