Utilizing the Power of the WebLogic Diagnostic Framework [ARTICLE]

Version 5

    Martin Heinzl, the author of WebLogic Diagnostic Framework (Rampant TechPress, 2015) and Advanced WebLogic Server Automation (Rampant TechPress, 2014) shows how to use WebLogic Diagnostic Framework services to get a detailed view into deployed applications and runtime server performance.




    The WebLogic Diagnostic Framework (WLDF) is an integrated framework in WebLogic that collects, saves and analyzes runtime data for monitoring or troubleshooting. It provides a number of services that are executed within the WebLogic Server (WLS) virtual machine. Using these services, you can gather information that will help you to get a detailed view into the runtime performance of your server instances and also into the deployed applications. It is a very valuable tool for error location and diagnostic operations.


    WLDF is a powerful subsystem of WebLogic. It is not a separate product and does not have to be installed separately. WLDF is integrated in WebLogic, runs in each WLS instance, and provides components and APIs that enable the user to access runtime information from WebLogic MBeanServer. (Note that it does not provide access to the ConfigurationMBeans, only to the runtime MBeans.) WLDF is not meant to be a tool for creating reports about the configuration. It is targeted to work with runtime information. WLDF can be used to collect data from different MBeans or create state snapshots. Other WLDF components can perform certain analysis or actions based on the collected data. The data can be accessed via external tools or displayed using the integrated dashboard. The main goal of WLDF is to provide access to insight information into the runtime state of servers and applications in order to monitor health or detect, analyze problem situations, and to react (if necessary) with specified actions. Almost all features of WLDF can be enabled or disabled dynamically with configuration changes and without requiring server restart or application re-deployment.


    Based on project observations, it is surprising that WLDF is not used more often. For many customers, WLDF may look very complicated and thus unsuitable for their needs. This article will break down that complexity into usable chunks and encourage everybody to consider WLDF as an option for monitoring and troubleshooting.


    WLDF Components


    WLDF provides a very powerful framework-and its structure and component dependencies look quite complicated. However, the structure of the APIs is, in fact, not very difficult to understand; only due to the wealth of functionality does WLDF initially appear complicated. WLDF consists of the following:

    • Data creators (basically, the standard and custom MBeans available in WLS)
    • Data collectors (Logger and Harvester components)
    • Image Capture component
    • Archive component
    • Accessor component
    • Instrumentation component
    • Watch and Notification component
    • Monitoring dashboard
    • Flight recorder integration


    Logger and Harvester (and also the Image Capture feature) collect and consume runtime information created by the different data creators. Normally, data creators are the different runtime MBeans inside the runtime MBeanServer of the WebLogic instances. The collected data is used by the Archive component for persistence and by the Watch and Notification components for monitoring and analysis. The data access subsystem communicates with the Harvester and the Logger to get real-time information and with the archive to get historical data.


    Data Collectors


    Many components in WebLogic collect runtime information. As already mentioned, the two major components that are important for WLDF are runtime MBeans (standard and custom MBeans) and also the WLDF's own instrumentation. It is impossible-or at least would be very time consuming and a waste of space-to collect all possible data. The different data collectors in WLDF are designed to collect only the attributes and data that are important for your actual monitoring or investigation goal. The components used to gather the diagnostic data are called Harvester and Logger. The diagnostic image feature can also collect a complete  snapshot.


    Data Processing and Archiving


    As it is impossible for administrators to sit in front of many different screens, pressing refresh all the time so that they do not lose an important data event, WLDF provides components that can analyze the data and react to certain events. The notification and the watch subsystems of WLDF have to be configured if automated reactions on certain events are required. Reactions can be based on rules, which have to be configured in the WLDF configuration MBeans.


    The following diagram from the official Oracle documentation shows the full complexity and dependency of the WLDF implementation and its various subsystems.




    Configuration and Usage


    WLDF is available out-of-the-box with WebLogic. There are a number of different ways to configure and use it, including configuration done via the WebLogic console, using automated scripts (WLST) and programs (JMX), and by direct configuration in the WebLogic XML configuration files. The data created and collected by WLDF can be used by different tools provided by Oracle or other tools that get notifications or collect data using one of the various APIs.


    The dashboard integrated with WLS Console can show metrics values collected by WLDF Harvester. WLS-specific Java Flight Recorder (JFR) events generated by WLDF, and other events generated by other producers, can be viewed and analyzed with Java Mission Control (JMC). Every configuration item in WebLogic, as well as all runtime states, are represented as Java Management Extensions (JMX) managed beans (MBeans). The same applies to every aspect of the WLDF configuration. The configuration MBeans are stored in the domain configuration and/or in additional outsourced xml files, which are referenced in the main config.xml file.   Whenever you create WLDF artifacts by running WLST scripts, running JMX programs, or creating configuration items using the WLS Console, WebLogic will create configuration MBeans. Even if you edit the config.xml files directly, WLS will create MBeans after the admin server has been restarted.


    (Please note that, due to size, it is impossible to present examples for WLST, JMX, WebConsole and even configuration files in this article, which focuses on WLST.)


    Using the WLS Console


    The most obvious way, of course, is the WebLogic console, with which every configuration activity can be done. These configurations will be lost when the domain gets deleted but, especially for a short test, this is a very useful approach. Whichever approach is used, the WebLogic console is very useful to verify and control if all settings are performed and activated as needed.



    Sections in the WebLogic home area dedicated to WLDF


    Using WLST / JMX


    The most common usage of WLDF will be with automated processes based on WLST or JMX. WLST is the best choice for administrators who want to set up WLDF configurations, activate or deactivate them, or modify configurations in a scriptable way.  It is script-based and easier to use than JMX.


    For automated environment setup based on Java, or even for tools setting up WLDF configurations on-the-fly, it is better to use the JMX API directly from Java code.


    Setting Up a WLDF Configuration


    Creating a New Diagnostic Module


    The first step when working with WLDF is to create a new diagnostic module. A module basically consists of nothing but a name and a description. The important aspect is that a module creates the container MBean inside WebLogic for the module configurations.

    Why do we need diagnostic modules?  A diagnostic module has no own functionality, but provides a bundle facility to combine a number of configuration elements. It can be viewed as similar to the Enterprise ARchive (EAR) package in J2EE.  Just as an EAR package contains Enterprise JavaBeans) (EJB) applications (*.jar), web applications (*.war), resource adapters (*.rar) and possibly others, a diagnostic module bundles all configuration items that belong together.


    The following example (WLST script) creates an empty diagnostic module:


    # connect to the admin server
    # switch to edit mode and start editing
    # switch to root and create a new module
    # switch to the new module and set the description text
    cmo.setDescription('This is a new WLDF module')
    # save and activate the changes


    Built-in diagnostic module


    Starting with WLS 12.1.2, WebLogic comes with pre-built WLDF modules (Low, Medium and High). These “built-in diagnostic system modules” are designed to collect common values from important WebLogic MBeans. Different modules (or "default profiles") are provided to collect data for WebLogic runtime, virtual machine, WebLogic services like Java Database Connectivity (JDBC), Java Message Service (JMS) and also for application  containers .  It is possible to enable a Low, Medium or High built-in module dynamically, or to switch them off completely. They can also be used as the basis for creating new WLDF modules, which can be further customized.


    The following WLST script shows an example of how to create a new diagnostic module based on a predefined built-in module:


    # connect to the admin server
    # switch to edit mode and start editing
    cmo.createWLDFSystemResourceFromBuiltin('TemplateTestModule', Medium)
    cmo.setDescription('This is a module which is based on the medium template')
    # save and activate the changes


    Data Archives


    Data archives are important WLDF components that persist data and let consumers use current and historical data for further analysis. Keeping historical data for long-term analysis is a feature that WebLogic does not offer for the normal Runtime MBean attributes. All information offered from the Runtime MBean attributes is current-state data. With data archives, WLDF extends this and allows historical data recording.


    Basically, WLDF supports two different ways to archive diagnostic data: in a file store (default) or in a JDBC-based database. The  following script shows how to set up a file-based archive in WLDF:


    # connect to the admin server
    # switch to edit mode and start editing
    # navigate to server specific ServerDiagnosticConfig mbean of the admin server
    cd ('/Servers/AdminServer/ServerDiagnosticConfig/AdminServer')
    # set the directory name
    # example of setting other values
    # save and activate the changes


    Note that a filestore is created by default. The example above shows how users can customize its location and other characteristics if needed, but this is not mandatory.


    Note: never forget to perform proper data housekeeping. Data collection—especially if a huge amount of data is collected—can quickly create large data archives. Furthermore, analysis will take more time if more historical data has to be considered.


    The following script shows how to set up housekeeping for a specific archive using WLST:


    # connect to the admin server
    # switch to edit mode and start editing
    # switch to the server specific diagnostic configuration
    cd ('/Servers/AdminServer/ServerDiagnosticConfig/AdminServer')
    # create a new retirement policy for aged data
    # change to the new policy and configure it
    # save and activate the changes




    The concept in WLDF that collects metric data is called Harvester. Data harvesters are configurations defined in WebLogic that describe which attributes from which Runtime MBeans must to be gathered at defined sampling rates. The collected information can then be used to analyze the metrics for certain conditions, or can be handed over to the archiving components in order to keep it as historical data. Collecting data with a harvester is especially useful for system-state monitoring and for analyzing the performance of the complete server or a subsystem of the server.


    Here is an example of setting up a Harvester with a WLST script:


    # connect to the admin server
    # switch to edit mode and start editing
    # change to the diagnostic module with the name "HarvesterTest"
    # create a metric for the mbean type JVMRuntimeMBean
    # change to metric MBean
    # configure which attributes (2 in this exampel) should be harvested
    set('HarvestedAttributes',jarray.array([String('HeapFreePercent'), String('HeapSizeMax')], String))
    # configure on which WebLogic server instances this metric should be executed
    set('HarvestedInstances',jarray.array([String('com.bea:Name=MS1,ServerRuntime=MS1,Type=JVMRuntime'), String('com.bea:Name=MS2,ServerRuntime=MS2,Type=JVMRuntime')], String))
    # finally set the MBeanServer to be used (usually always ServerRuntime)
    # save and activate the changes




    While metrics are configurations to gather specific sets of MBean attributes together as a metric, WebLogic offers the generation of state snapshots. These snapshots are called “diagnostic images.” Image capture can be triggered from different sources in WebLogic, and can be events, actions on the administration console, a WLST script, or a JMX program event.


    Issue an image request using WLST:


    # connect to the admin server
    cd ('/WLDFRuntime/WLDFRuntime/WLDFImageRuntime/Image')
    # issue image capture task


    It is possible to list all image tasks:


    # connect to the admin server
    cd ('/WLDFRuntime/WLDFRuntime/WLDFImageRuntime/Image')
    # issue image capture task


    Watch out for diagnostic conditions


    Watches consume the data collected by metric, instrumentation events  or  log files and—based on rules—filter out the data that you’re interested in. If a Watch finds an interesting piece of information, all associated notifications will be invoked in order to process their tasks. Therefore, instead of you waiting and searching for special events or conditions to happen, the system will inform you whenever this is the case. This is based on the different rules that can be configured for each Watch.


    The following WLST script shows a simple example of a monitor for an MBean attribute:


    # start the edit mode  (must be already connected to the admin server !)
    cd (‘/’)
    # create the watch MBean
    #change to the new MBean (note the deep path)
    # set to type “Harvester”
    # enable the watch
    # create the rule expression
    AdminServer,Type=JVMRuntime//HeapFreePercent} < 10)')
    # save work and activate changes


    Notifications and alarms for certain system states


    Notifications are configured actions within a diagnostic module that can be called whenever a Watch rule expression has been evaluated to “true.” Note that notifications are defined in the scope of a diagnostic module, but may need additional resource configurations from the underlying WebLogic. One example is the email notification that requires a JavaMail provider to be configured on the WebLogic level. Up to WebLogic 12.1.3, WebLogic supports 5 different kinds of notifications.  Please be aware that Oracle will add a more generic notification type in WebLogic 12.1.4, which will allow the execution of a WLST command or script.


    The following types of notifications are available up to WebLogic 12.1.2 and 12.1.3:


    Java Management Extensions (JMX)Send a JMX notification.
    Java Message Service (JMS)Send a JMS message.
    Simple Mail Transfer Protocol (SMTP)Send an email based on the Java Mail API. This requires that a mail provider is configured in WebLogic and registered into the JNDI of the managed-server where the watch expression has been evaluated to true.
    Simple Network Management Protocol (SNMP)Send an SNMP trap.
    Diagnostic image creationCreate a diagnostic image.


    The following script demonstrates how to create a JMX notification using WLST:


    # change to the diagnostic module
    # create the new notification
    # switch to the new notification
    # enable the notification
    # define the type of the notification


    Access to data stored in the diagnostic archives


    Data recorded by other WLDF components (especially the different Harvester components) will be kept in the data archives. Depending on the WebLogic version, the WLDF configuration and other factors, different archives are available that contain different categories of information (e.g., Harvester data, log event data, etc).


    The API defined in WLDF allows the user to access the data from the different data stores. If the server is running, then access is possible by using the Administration Console, JMX, or WLST.


    To better understand the data accessor component, it is important to understand how WLDF keeps the diagnostic data. Because WLDF by itself is not using network communication, by default all the data recorded in a WLS (admin or managed server) is kept on this server in the data archives. The data accessor therefore has to work on the basis of individual servers and has to provide access capabilities on a per-server basis. Each server may maintain several data  stores .  It is possible, however, to configure a JDBC-based archive to store collected data in a central database.


    Which data stores are available?


    Data recorded by WLDF is stored in different logical data stores. The distribution is based on the type of the data (e.g., harvested data, server logs, http access logs, etc.).


    The content of these stores is similar to tabular data: each row corresponds to one entry in the store and the headlines of each column describe the content.


    The following list includes the different types of data stores that can be maintained on a WLS:


    • HTTP_LOG


    The following example shows how to discover the available data stores using WLST:


    cd ('/WLDFRuntime/WLDFRuntime/WLDFAccessRuntime/Accessor')
    availNames = cmo.getAvailableDiagnosticDataAccessorNames()
    print 'Availables Names\n------------------------------'
    for name in availNames:
        print '   ' + name


    Each store-specific WLDFDataAccessRuntimeMBean instance offers different methods to query data from the store. This MBean offers methods to get all selected rows as a bulk. These methods are called retrievedataRecords, and the MBean offers a few methods with this name but with different signatures.


    Especially for large queries and result sets, this MBean also offers a kind of paging mechanism called a "Cursor,"  which lets the client decide how many data rows are returned with every call (argument of the "fetch" method).


    The following WLST script will load the complete result, set in chunks of 10 datarows:


    # connect to server
    # switch to the server runtime 
    # (if connected to the admin you need to use domainRuntime and switch to the desired server runtime)
    # change to the accessor runtime
    cd ('WLDFRuntime/WLDFRuntime/WLDFAccessRuntime/Accessor/WLDFDataAccessRuntimes') 
    # change to the desired runtime (archive specific)  - in this case the harvester data
    cd ('HarvestedDataArchive')
    # open the cursor
    myCursor = cmo.openCursor(Long(0), Long.MAX_VALUE, None)
    # while more data is available, read the next 10 rows and print them
    while cmo.hasMoreData(myCursor):
        print cmo.fetch(myCursor,10)
        print ""
        print ""


    Offline Access Using WLST


    WLST provides a special operation called "exportDiagnosticData," which allows the user to extract diagnostic data in offline mode. Note that this method can be used only if the server is not running. If the server is running, the archive is locked and calling this method will result in a WLST error.


    Further, you will not have access to the MBeans of WebLogic, and therefore cannot query configuration or runtime information (e.g., available data store instances).


    Advanced Features


    WLDF provides a number of advanced features that are very powerful albeit more difficult to use, but their discussion is beyond the scope of this article. Basically, the most important advanced feature is application-specific Thus far, all monitoring has been based on standard WebLogic MBeans. In order to monitor application-specific values, the customer application has to be instrumented with monitoring hooks.  A common way of doing this is using AOP (aspect oriented programming) technologies, and WLDF offers such a framework called DyeInjection. Because this article discusses WLDF fundamentals, DyeInjection will not be discussed or explained as it is a comprehensive topic on its own.


    The Idea of Monitoring Profiles


    While monitoring, troubleshooting, root cause analysis and other areas are important activities in every production system, these tasks are hard to anticipate because error conditions are often not known upfront. Further, monitoring requirements change dramatically, depending on the nature of the application. All these different activities require the collection of a different set of metrics, as well as a different set of alarms, watches or notifications.


    Monitoring and troubleshooting activities also depend on the environment in which they are performed and on the actual person who has to do it. If a developer has to do some monitoring or troubleshooting in their development environment, it is rather straightforward, as they know precisely what to look for and where to look for it—and knows/controls the application. In most organizations, the higher-level environments (like integration testing, non-functional testing, production-like environment, user acceptance testing (UAT), and especially production and training environment) are owned and controlled by a group of administrators that implements a kind of hosting service for the organization.  The individual developer does not (and should not!) have direct access to the application, application server or even machine.


    In these environments, the situation is different.  A group of people who do not have insights into the application (code) must operate and maintain a large number of different applications. According to Murphy's law, the difficult problems will of course show up only in the production or other higher-level environments; this group must therefore constantly monitor the health of their applications and infrastructure (including the application servers). They must also be prepared to activate data collection in case of error conditions so that they can deliver data sufficient to allow developers or support groups to analyze problem situations. In these environments, application-specific metric collection is required; it is also impossible to have all data collection running all the time (due to resource issues for CPU, disk and database usage).


    One successful solution is something we will call "monitoring profiles"; you might also see it referred to as "setup" or "constellations" (or something else). The idea behind monitoring profiles is that many different combinations of metric collections and alarms/notifications will be prepared and made available for the administrator group. Each combination is called a "profile." With WLDF, it is even possible to install all the different profiles in all WebLogic instances but leave them deactivated or untargeted so that these configurations are not active and neither require nor waste resources.  It is also possible to create multiple WLDF modules in WLS releases older than 12.1.2. The difference between WLS 12.1.2 and older releases is that, prior to 12.1.2, only one WLDF module could be active on a managed server. WLS 12.1.2+ allows multiple WLDF modules to be concurrently active on a managed server.


    The administrator group will be trained on which profile must be activated for specific special error situations.


    Artifact Profiles


    Different granularities of profiles are possible. The finest grained category are profiles that contain a certain WLDF component.  Frankly, this is more an academic approach than a real-life project approach.  But—if  really done well—it can be a good foundation for the more coarse-grained profiles types discussed afterwards.


    A Harvester watch, for example, can be packed into a profile that contains just the Harvester and all its settings (like attribute rules). A Watch profile will contain a watch description and all rules belonging to this watch.


    A notification profile contains only a notification description (e.g., a JMX notification profile, a personal email notification profile, an email notification profile that send emails to an alerting system, an SNMP notification, etc.).


    Subject Matter Profiles


    Based on the artifact defined in the last section, this type of profile defines all WLDF components that together define everything that is needed for a specific subject. For example, a profile to monitor transactions will have a Harvester for the JTARuntime, watches with rules for stuck transactions or long-running transactions, and a notification system that may send an alert email. It can be seen as a combination of artifacts.


    Diagnostic Module Profiles


    At first glance, this looks similar to the component profile discussed earlierbut there’s a very important difference: this kind of profile defines a complete diagnostic module. Whenever this profile is applied, the complete diagnostic module will be activated.


    It is not expected that different profiles will be combined. Prior to WLS 12.1.2, the activation of such a profile requires either the deletion of any other defined profile or the un-targeting of any other profile.  Since WLS 12.1.2, it is possible to have multiple active profiles so that activation of a profile does not mean that other profiles—if any—are affected.




    WebLogic has a built-in framework for data gathering, monitoring, filtering and notification called WLDF (WebLogic Diagnostic Framework). In addition to standard MBean monitoring, this framework offers many powerful and advanced features. One of these is an AOP-like application code instrumentation called DyeInjection, which is the technique for marking request flows based on certain criteria, such as origin of the request, originating user, etc.). The Instrumentation component can make use of DyeInjection by inspecting the request markers (or dyes) and generated data only for selected requests, thus limiting the volume of generated data and performance overheads. Even if it looks complex at  first glance, every administrator should take it into account for monitoring, troubleshooting or even dynamic reactions for single WLS domains.




    Excerpts included from the book WebLogic Diagnostic Framework - Understanding and Using WLDF through Examples and Profiles, by Martin Heinzl (ISBN: 0986119407).


    About the Author


    Martin Heinzl is a principal consultant with over 15 years’ experience in middleware, system architecture and enterprise systems. His main focus is on the architecture, integration, operation and security of distributed systems based on J2EE, CORBA and Java. He is currently on a team of experts behind the global middleware application hosting platform of a global bank.  He is a member of the Oracle customer advisory board for WebLogic in EMEA. He is also the author of Advanced WebLogic Server Automation with WLST and JMX.


    Note:This article has been reviewed by the relevant Oracle product team and found to be in compliance with standards and practices for the use of Oracle products.