Forum Stats

  • 3,757,530 Users
  • 2,251,242 Discussions
  • 7,869,855 Comments

Discussions

Understanding Always on Monitoring(AOM) and Configuring AOM in 13c Cloud Control

AOM (Always on Monitoring) is useful for the EM Administrators to send email notifications to the users even when the OMS itself is down. The Enterprise Manager Always-On Monitoring provides the ability to monitor critical target status and metric alerts. The service continuously monitors critical targets through the Enterprise Manager Agent and can be easily configured to send email notifications for these events to administrators.

Always-On Monitoring can be configured to send notifications at any time, but is particularly useful when experiencing downtime of 13c Cloud Control OMS or repository database for maintenance operations such as upgrade and patching.  The Always-On Monitoring is synchronized with Enterprise Manager to reuse the configuration of monitored targets as well as requisite notification data such as notification contacts and email gateway configuration. Once properly configured and synchronized, the service will receive alerts from Enterprise Manager Agents and send email notifications to the appropriate administrators.

Steps to use AOM:

1. Installing the Always-On Monitoring(AOM)

2. Configuring the Always-On Monitoring (AOM)

1. Installing the Always-On Monitoring (AOM)

a. Prerequisites for installation of AOM

1. Create/Install a new database to store the EMS schema on target machine where AOM needs to be configured

2. Copy emkey to the 13c OEM repository database as below:

<MIDDLEWARE_HOME>/bin>./emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :

The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

3. Ensure Java version 1.7 is available on the machine where AOM will be installed

4. SMTP Mail server details needs to be configured in 13c OEM console

b. Installation of Always-On Monitoring(AOM)

1. Copy the AOM software (ems_13.1.0.0.0.zip) from <MIDDLEWARE_HOME>/sysman/ems on the OMS machine to the target machine where AOM will be installed
2. Unzip the file ems_13.1.0.0.0.zip on the target machine, this creates a ems directory with the following content:

[[email protected] ems]$ pwd
/u02/em13c/ems/ems
[[email protected] ems]$ ls -ltr
total 1572
drwxr-xr-x. 4 oracle oinstall 4096 Dec 12 15:06 sql
-rw-r--r--. 1 oracle oinstall 1596182 Dec 12 17:36 ems_13.1.0.0.0.jar
drwxr-xr-x. 2 oracle oinstall 4096 Dec 12 17:36 lib
drwxr-xr-x. 2 oracle oinstall 4096 Dec 12 17:36 scripts

2. Configuring the Always-On Monitoring (AOM)

a. Configuring  Always-On monitoring (AOM)

1. set JAVA_HOME to Java 1.7

          Example: export JAVA_HOME=/u02/em13c/middleware/oracle_common/jdk

2. Run emsca script to configure AOM as below:

          $ cd <AOM_HOME>/scripts
          $ ./emsca

           emsca script will prompt for following details:
           Always-On Monitoring Repository Connection String
           Create Always-On Monitoring Repository User [n]
           Always-On Monitoring Repository SYSDBA Username :
           Always-On Monitoring Repository SYSDBA Password :
           Always-On Monitoring Repository Username [ems] :
           Always-On Monitoring Repository Password [ems] :
           Enterprise Manager Repository Connection String :
           Enterprise Manager Repository Username : sysman
           Enterprise Manager Repository Password :

Example:

$ cd <AOM_HOME>/scripts
$ ./emsca
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Always-On Monitoring Repository Connection String : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
Create Always-On Monitoring Repository User [n] : y
Always-On Monitoring Repository SYSDBA Username : sys
Always-On Monitoring Repository SYSDBA Password :
Always-On Monitoring Repository Username [ems] :
Always-On Monitoring Repository Password [ems] :
Enterprise Manager Repository Connection String : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
Enterprise Manager Repository Username : sysman
Enterprise Manager Repository Password :
Creating Always-On Monitoring repository user ems
Enter Enterprise Manager Middleware Home : /u02/em13c/middleware
Connecting to Always-On Monitoring repository.
Creating Always-On Monitoring repository schema
Creating repository storage for Targets data.
Creating repository storage for Alerts and Availability data.
Creating repository storage for Notification Metadata data.
Creating repository storage for Target Metric Metadata data.
Registering Always-On Monitoring instance
Always-On Monitoring Upload URL: https://venkat.idc.oracle.com:8081/upload
Oracle PKI Tool : Version 12.1.3.0.0
Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.

Certificate was added to keystore

This creates ems schema inside the database on target machine to store the AOM data

b. AOM Post Configuration Tasks

1. Remove emkey from 13c OEM repository database
         <OMS_HOME>/bin>./emctl config emkey -remove_from_repos

2. By default, AOM will not be running. Before starting the AOM for the first time, need to perform sync operation so that the data in 13c OEM repository database is in sync with AOM repository database.

$ cd <AOM_HOME>/scripts
$ ./emsctl sync
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Connecting to Always-On Monitoring repository.
Starting synchronization with EM.
Synchronizing with EM repository: [email protected](DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Mon Dec 21 07:17:06 EST 2015

3. Once sync is completed successfully, start the AOM as below:

$ cd <AOM_HOME>/scripts
$ ./emsctl start
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Starting Always-On Monitoring.
Notifications Enabled : false
Total Downtime Contacts Configured : 0
Always-On Monitoring is up.

4. Verify the AOM status as below:

$ cd <AOM_HOME>/scripts
$ ./emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version : 13.1.0.0.0
Always-On Monitoring Home : /u02/em13c/ems/ems
Started At : December 21, 2015 7:25:37 AM EST
Last Repository Sync : December 21, 2015 7:15:48 AM EST
Upload URL : https://venkat.idc.oracle.com:8081/upload
Always-On Monitoring Process ID : 27549
Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
Notifications Enabled : false
Total Downtime Contacts Configured : 0

5. Set the AOM upload URL on OMS side so that agent uploads information to the AOM repository database:

$ cd <MIDDLEWARE_HOME>/bin
$./emctl set property -name "oracle.sysman.core.events.ems.emsURL" -value " https://venkat.idc.oracle.com:8081/upload"
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
SYSMAN password:
Property oracle.sysman.core.events.ems.emsURL has been set to value https://venkat.idc.oracle.com:8081/upload for all Management Servers
OMS restart is not required to reflect the new property value


6. Configure Downtime Contacts on 13c OEM side:

Downtime contacts can be configured from any of the following methods:

a. For each target, navigate to the Target Properties page, which can be accessed from the target's homepage. From the target menu on the target homepage, select Target Setup and then Properties. The Target Properties page displays.

         •  Click on Edit and specify the email address in the Downtime Contact target property. You can specify multiple email addresses by separating them with commas.
         • Use the EM CLI set_target_property_value verb.
                emcli set_target_property_value -property_records="target_name:target_type:property_name:property_value"

Example:
$ ./emcli set_oms_property -property_name='oracle.sysman.core.events.ems.downtimeContact' -property_value='[email protected]'
Property "oracle.sysman.core.events.ems.downtimeContact" has been successfully set to value "[email protected]" on all Management Servers.
Server restart is not required for the property change to get reflected


b. Per-target downtime contact based on event rules:

          Always-On Monitoring may also send email notifications to different users for each target. These contacts are generated in Enterprise Manager based on the event rules for that target. Therefore, as event rules are changed in Enterprise Manager, the contacts must be re-generated and an incremental synchronization performed. By leveraging the event rule setup, the downtime contact will be generated based on the email recipient for the event rule for a Target Availability event type where Down status has been selected.

Note:
Although downtime contacts are generating using only Target Availability event rules, Always-On Monitoring will send notifications for both target availability and metric threshold alerts.
You can review and update the recipients of your target availability (status down) event rules. Doing so allows you to generate a list of downtime contacts using EM CLI or by submitting the downtime contact generation job.


Once downtime contacts are configured, perform sync on AOM to update the AOM repository database with downtime contact information.

Example:
$ cd <AOM_HOME>/scripts
$ ./emsctl sync
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Connecting to Always-On Monitoring repository.
Starting synchronization with EM.
Synchronizing with EM repository: [email protected](DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Mon Dec 21 07:52:01 EST 2015


Verify the total downtime contacts value in AOM status output

$ cd <AOM_HOME>/scripts
$./emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version : 13.1.0.0.0
Always-On Monitoring Home : /u02/em13c/ems/ems
Started At : December 21, 2015 7:25:37 AM EST
Last Repository Sync : December 21, 2015 7:51:59 AM EST
Upload URL : https://venkat.idc.oracle.com:8081/upload
Always-On Monitoring Process ID : 27549
Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
Notifications Enabled : false
Total Downtime Contacts Configured : 2


7. By default notifications are disabled, need to enable the notifications for the AOM to send email notifications to the downtime contacts configured in step above. Enable Notifications option on AOM as below:

[[email protected] scripts]$ ./emsctl enable_notification
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Notifications have been enabled. There are downtime contacts configured.
Connecting to Always-On Monitoring repository.
Starting synchronization with EM.
Synchronizing with EM repository: [email protected](DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
Synchronizing Targets data.
Synchronizing Alerts and Availability data.
Synchronizing Notification Metadata data.
Synchronizing Target Metric Metadata data.
Synchronization complete at : Mon Dec 21 07:56:54 EST 2015


Verify the notification status from the AOM status output:

[[email protected] scripts]$ ./emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version : 13.1.0.0.0
Always-On Monitoring Home : /u02/em13c/ems/ems
Started At : December 21, 2015 7:25:37 AM EST
Last Repository Sync : December 21, 2015 7:56:53 AM EST
Upload URL : https://venkat.idc.oracle.com:8081/upload
Always-On Monitoring Process ID : 27549
Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
Notifications Enabled : true
Total Downtime Contacts Configured : 2


8. Trigger any metric alert for a target for which downtime contacts are configured and verify if the email alert is received.

Sample email alert:

Subject: EM Event: Warning:em13cdb.idc.oracle.com - User SYS logged on from venkat.idc.oracle.com.

Email Body:

Target type=Database Instance
Target name=em13cdb.idc.oracle.com
Message=User SYS logged on from venkat.idc.oracle.com.
Severity=Warning
Event reported time=Dec 22, 2015 8:09:24 AM EST
Event Type=Metric Alert
Event name=UserAudit:username
Metric Group=User Audit
Metric=Audited User
Key Value=SYS_venkat.idc.oracle.com
Sent by Oracle Enterprise Manager - Always-On Monitoring


9. To disable notification on AOM side, run the following command:

[[email protected] scripts]$ ./emsctl disable_notification
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Notifications have been disabled.

[[email protected] scripts]$ ./emsctl status
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
------------------------------------------------------------------
Always-On Monitoring Version : 13.1.0.0.0
Always-On Monitoring Home : /u02/em13c/ems/ems
Started At : December 21, 2015 7:25:37 AM EST
Last Repository Sync : December 21, 2015 7:56:53 AM EST
Upload URL : https://venkat.idc.oracle.com:8081/upload
Always-On Monitoring Process ID : 27549
Always-On Monitoring Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emspdb)))
Enterprise Manager Repository : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=venkat.idc.oracle.com)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=em13pdb.idc.oracle.com)))
Notifications Enabled : false
Total Downtime Contacts Configured : 2


Log Location

Log files record Always-On Monitoring events that occur during operation and are generated as follows:
        • emsca logs: emsca.err (only errors), emsca.log.0 (rotating log file that contains all output including errors).
        • ems logs: ems.err (only errors), ems.log.0 (rotating log file that contains all output including errors).
These files are located in the $EMS_HOME/logs directory.