Dynamic Load Balancing in GlassFish Application Server Blog


    GlassFish is a fully Java EE 5-compliant application server with enterprise-ready features available under two OSI-approved licenses. Among many other enterprise-level features, GlassFish provides a very good self-management functionality extendable using the Java Management eXtension (JMX) standard.

    The GlassFish application server provides good facilities for cluster management and load balancing. Still, sometimes we need to have more fine-grained control over how our cluster nodes are loaded with requests. One such condition can happen in a shared environment when we are using the processing power to host several applications, ranging from databases to batch job processing to application servers serving requests that come from different customers. Such conditions can lead us to change the load routed toward each GlassFish instance during different time slices. We can change the routed load manually, but having the ability to change it automatically based on defined rules is something very desirable.

    This article will explain how we can use GlassFish self-management facilities, JMX, and the Application Server Management eXtension (AMX) APIs to change the load balancer configuration dynamically. Sample code to illustrate these processes is provided throughout the article and is also available in the Resources section.

    Aside from reviewing some background information, we will need to perform three steps in order to complete the example to change load routed toward each application server instance:

    1. Developing an MBean, which can change the load balancer configuration.
    2. Creating the management rule and deploying it.
    3. Testing the system with new features.

    Fundamentals of JMX and AMX APIs

    JMX refers to several JSRs that allow software vendors and developers to have a common technology for exposing management and monitoring capabilities of their Java-based applications.

    Usually when we say JMX, we are referring to a set of at least two JSRs: Java Management Extensions (JSR 3) and the JMX Remote API (JSR 160). Although there are other specifications that complement JMX, like the J2EE Management Specification (JSR 77), the most utilized JSRs are JSR 3 and JSR 160.

    JMX has three important components:

    • MBean: MBeans are probe-level objects of the JMX specification. There are four kinds of MBeans for different use cases; the most popular MBean type is Dynamic.
    • MBeanServer: This is the access point for MBeans. All MBeans should be registered with an MBeanServer. It is the access point that allows a variety of operations such as invoking MBeans' operations, changing attributes of MBeans, event listener registration, and so on.
    • Connectors and Adapters: These allow different types of management consoles to create a connection to anMBeanServer from outside of the host JVM. Popular connectors are RMI and the JMX Messaging Protocol (JMXMP; defined by JSR 160). Popular adapters are the SNMP adapter defined in Java Platform Profiling Architecture (JSR 163), and the web (HTML/HTTP) adapter.

    Using JMX, we guarantee that any other software that knows JMX can interact with our MBeans in order to monitor specific attributes or change some others that are exposed by our MBeans, or to listen for different types of JMX events published by our MBeans.

    As mentioned, accessing MBeans is only possible usingMBeanServer, and it is not a convenient way for many users, as it will involve them with JMX, which not an easy API to deal with. Calling an MBean operation will go through reflection, so we will need to know which MBean we want to use, the method name, the method parameters and signature, etc. We can find the required MBean by using its ObjectName, which is the name that we register our MBeans with. An ObjectNamecore is a string that follows an schema to make a queryable representation of registered MBeans. TheMBeanServer provides methods to query the registry based on the tree-like representation formed by theObjectName instances.

    AMX is present to interact with GlassFish application server MBeans, free of the difficulties discussed above. AMX is a set of APIs that expose management MBeans as client-side dynamic proxies that implement AMX interfaces. These AMX interfaces are mostly included in com.sun.appserv.management.config and some of its sub-packages. We can use AMX APIs on the client side in order to manage almost all JSR 77-approved Java EE objects like servers, applications, resources, etc., without knowing anything about JMX. However, all of these EE objects are also manageable through plain JMX.

    Using AMX we can:

    1. Change application server configurations--create resources, delete resources, enable or disable, etc.
    2. Manage servers, node agents, clusters, etc.
    3. Receive notifications for almost any event happening in the application server and react accordingly.
    4. Monitor the state of many objects that are hosted inside the application server. These include EJBs, web applications, enterprise applications, connection pools, etc.

    When we install GlassFish in the cluster or enterprise profile, we create a domain of GlassFish application servers that initially has only one member. This first member is the Domain Administration Server (DAS) and it is a single point of management and administration for every cluster or instance that we may add to this domain. All related configuration of every domain object (such as instances, clusters, connection pools, etc.) is stored in DAS and can be changed directly using DAS. AMX allows us to connect to DAS and perform every required operation on application server MBeans in order to change the configuration of mentioned domain objects. There are some helper classes included incom.sun.appserv.management.helper that further ease working with AMX APIs. The following example shows how we can change the weight of a clustered server instance using these APIs.Weight is a number that indicates the percent of requests that the load balancer sends to the instance.

    AppserverConnectionSource ASConnection = Connect.connectNoTLS("", 8686, "admin", "adminadmin"); DomainRoot dRoot = ASConnection.getDomainRoot(); Map<String, ClusterConfig> clusters = dRoot.getContaineeMap(XTypes.CLUSTER_CONFIG); ClusterConfig clusterConf= clusters.get("Cluster-1"); Map <String, ClusteredServerConfig< servers = clusterConf.getClusteredServerConfigMap(); ClusteredServerConfig instance1 = servers.get("instance-01"); instance1.setLBWeight("25"); 

    As you can see, we simply get the clustered server instance and change its configuration--without knowing anything about JMX. This is what the AMX APIs provide us with. In the AMX design, all objects are branches of the domain root, which seamlessly maps to DAS. In the next section, we will discuss what management rules are and how we can use them to implement our defined requirements.

    Basics of GlassFish Management Rules

    A management rule is a set of:

    • Event: An event uses the JMX notification mechanism to trigger actions. Events can range from an MBean attribute change to specific log messages.
    • Action: Actions are associated with events and are triggered when related events happen. Actions can be MBeans that implement the NotificationListener interface.

    Important types of events are as follows:

    • Monitor events: These type of events trigger an action based on an MBean attribute change.
    • Notification events: MBeans can implementNotificationBroadcaster in order to send notifications to all listeners that registered their interest on its event notifications.
    • System events: This is a set of predefined events that come from the internal infrastructure of GlassFish application server. These events include: lifecycle, log, timer, trace, and cluster events.

    The focus of this article is on cluster events, which are members of the system events family. There are three cluster events, fired when a cluster is started or stopped, or has failed. Our sample code listens for the cluster start event and upon receiving this event, it schedules and start a timer. This timer calls methods in order to perform the logic for deciding whether it is required to update instance weights or not. This decision will be made based on the content of a configuration file with the following elements:

    1. Name of the cluster whose instance weights we will update.
    2. Acceptable delay between entering a time slice and updating instance weights. This value is the interval on which the previously mentioned timer will execute the task that performs our logic to decide on updating instance weights.
    3. Some time slices, where each time slice is a set of start and end times and instance names with their weights during that slice of time.

    A sample configuration file that can be used by the article sample code is shown in the following snippet.

     <config interval="10000" cluster-name="Cluster-1"> <slice start-time="06:40:00 AM" end-time="10:40:00 AM"> <instance name="instance-01" weight="55"/> <instance name="instance-02" weight="55"/> </slice> <slice start-time="03:00:00 PM" end-time="06:00:00 PM"> <instance name="instance-01" weight="38"/> <instance name="instance-02" weight="38"/> </config>

    This configuration file indicates that there is a cluster namedCluster-1 and it is required to reconfigure its instance weights during two time slices. Acceptable delay between entering the time slice and updating the instance weights is 1000 ms; i.e., one second.

    We saw that each management rule has an event that will trigger an action. An action is a MBean that implements theNotificationListener interface, which has one to-be-implemented method named handleNotification. When the management rule triggers the action it will call this method.

    By looking at the JMX tutorial and references we can find that the most basic type of MBeans are standard MBeans. These MBeans are composed of an interface that must follow a specific naming schema and the class that implements that interface. The interface must end with word MBean. In this article, the action MBean is a custom MBean that, in addition to its own MBean interface named ClusterWeightManagerMBean, implements theNotificationListener interface in order to receive the cluster's start notification. So the final action MBean class implements the following two interfaces:

    • ClusterWeightManagerMBean: This interface includes public methods that our MBean makes available for a JMX agent or tools like JConsole.
    • NotificationListener: Implementing this interface allows our MBean to listen for a cluster start event after we register its interest for that kind of event by defining a management rule.

    The action MBean discussed above needs a configuration file to read time slices from and use them during its operations, and the configuration file address is a property of MBeans that we need to persist for future use. Here, GlassFish comes to help us by providing some facilities for storing and initializing the JavaBean-patterned property of MBeans. GlassFish stores the property value in the domain.xml file and initializes it each time an object of the MBean gets constructed.

    In order to let GlassFish read and write the property value, we should provide setter and getter methods accessible from a management agent; as we already saw, any method that needs to be accessible by a management agent should be included in the MBean interface. The following code snippet showsClusterWeightManagerMBean with a pair of setter and getter methods to read and write theconfigurationFilePath attribute.

    public interface ClusterWeightManagerMBean { public String getConfigurationFilePath(); public void setConfigurationFilePath(String configurationFilePath); } 

    The ClusterWeightManager class implements this interface to form an acceptable MBean. It should also implement theNotificationListener interface that is required to receive notifications from a management rule. Before getting into details of ClusterWeightManager's implementation, let's take a look at a helper class named TimeSlice, which holds each time slice's information. The following listing shows the TimeSlice class.

    public class TimeSlice { private Map<String, String> instanceWeights; private Date startDate; private Date endDate; public TimeSlice(Map<String, String> instanceWeights, Date startDate, Date endDate) { this.instanceWeights = instanceWeights; this.startDate = startDate; this.endDate = endDate; } public Map<String, String> getInstanceWeights() { return instanceWeights; } public void setInstanceWeights(Map<String, String> instanceWeights) { this.instanceWeights = instanceWeights; } public int compareTime(Date curDate) { int result = 0; if (curDate.compareTo(endDate) == 1) { result = 1; } if (curDate.compareTo(startDate) == -1) { result = -1; } return result; } } 

    Finally we need to implement the core functionalities that reside in the ClusterWeightManager, which is the MBean implementation class. The full code would be too long to include here; see the sample code ZIP in the Resources section. For now, let's focus on the important parts.

    Before investigating any other method, we should discussapplyInstanceWeights, which applies the weight of each instance based on the timeSlice configuration that we pass to it. Here's the implementation of theapplyInstanceWeights method.

    private void applyInstanceWeights(TimeSlice currectTimeslice) { String instanceNamePrefix = "amx:j2eeType=X-ClusteredServerConfig,name="; Map<String, String> instanceWeights = currectTimeslice.getInstanceWeights(); try { Set instanceNames = instanceWeights.keySet(); Iterator It = instanceNames.iterator(); while (It.hasNext()) { String instanceName = (String) (It.next()); String instanceWeight = instanceWeights.get(instanceName); ObjectName name = new ObjectName(instanceNamePrefix + instanceName); Attribute attrib = new Attribute("LBWeight", new Integer(instanceWeight)); mBeanServer.setAttribute(name, attrib); } } catch (Exception ex) { ex.printStackTrace(); } }

    By looking carefully at the above code snippet, you can see that we pass an object containing the current time slice to this method and it will use its instanceWeights property, which is a map of instance names and instance weights, to update instance weights. As you may have noticed, theinstanceNamePrefix value is not complete, and in the next few steps we append the instance name to make it a completeObjectName in order to let theMBeanServer locate its corresponding object (a clustered instance). Then we use this ObjectName andMBeanServer to change the value of theLBWeight attribute of the located instance.

    One other important method is handleNotification, which is inherited from NotificationListener. It starts the instance weight update process when it receives the cluster start notification.

    public void handleNotification(Notification notification, Object handback) { initialize(); this.startManager(); } 

    After receiving the notification, it calls theinitialize method, which parses the XML file and creates an ArrayList of all time slices included in the configuration file. The startManager method schedules and starts the timer that we talked about earlier.

    You may ask, "What if our cluster is already running? Should we restart it after we deploy the management rule?" In order to give an answer to these questions, let's take a look at what JMX provides for a similar situation and then to the solution that we used. Usually when we need to perform some tasks after registering an MBean, we implement the MBeanRegistration interface and use its postRegister method to perform the required logic. But in our case, we cannot do this. Why? Because we have an XML file that contains the instance weight configuration, and we put its address into one of our MBean variables;postRegister will execute before setters, so we don't have the configuration file address during the post-registration phase to read the configuration. What we can do is use thesetConfigurationFilePath method to check the cluster state and perform required actions if cluster is already running.

    In the setConfigurationFilePath method we simply use the cluster name to create the complete ObjectNameand then check the state attribute of the corresponding object using mBeanServer: 1means that the cluster is running, -1 indicates that the cluster failed, and 0 means the cluster is stopped.

    public void setConfigurationFilePath(String configurationFilePath) { this.configurationFilePath = configurationFilePath; String clusterPrefix = "amx:j2eeType=X-J2EECluster,name="; initialize(); try { ObjectName name = new ObjectName(clusterPrefix + clusterName); Integer state = (Integer) mBeanServer.getAttribute(name, "state"); if (state == 1) { isTimerRunning = true; this.startManager(); } } catch (Exception ex) { ex.printStackTrace(); }

    Now, in the handleNotification method we add code to check an object-level property to see whether the timer is already running before we start another instance of the timer.

    Creating the Management Rule and Deploying It

    Assuming you've unzipped the sample code and compiled JAR file, the class files package and the sample configuration file are available for this step. Copy the the entire package hierarchy (which is inside a folder named classes in the attached sample code) into domain_dir/applications/mbeans/ in order to make the classes available to your domain class finder. Start the application server and use the following asadmincommand to deploy the MBean into the DAS instance, making sure that you change the configuration file path according to where you copied the configuration file.

    create-mbean --host localhost --port 4848 --user admin --name ClusterInstanceWeightsManager --attributes ConfigurationFilePath=c\:/config.xml samples.glassfish.management.clustermanager.ClusterWeightManager 

    Deploying the ClusterWeightManager MBean is equal to including some configuration elements in the domain.xmlfile and registering the ClusterWeightManager MBean with the appropriate MBeanServer. During each startup, the application server will register all MBeans included indomain.xml with the MBeanServer. Now we have our ClusterWeightManager MBean registered with the management agent, and when it receives notifications it can start the logic and update instance weights if required. But before it can receive any notifications, it should announce its interest in receiving the cluster start notification. Creating a cluster management rule is how we register our MBean's interest for one of the cluster events.

    We need to define the management rule, which will automatically send the cluster start notification to our MBean when the cluster starts. This management rule is the registration of our MBean as a listener for cluster events. One additional task of the management rule is filtering events, so our class will only receive an event type if we define our interest in it. The followingasadmin command will create the required management rule.

    create-management-rule --eventtype cluster --host localhost --port 4848 --user admin --ruleenabled=true --action ClusterInstanceWeightsManager --event loglevel INFO --recordevent=true --eventproperties name=start ClusterManagementRule 

    We can query our registered MBeans and management rules using following asadmin commands.

    >list-management-rules --user admin >list-mbeans --user admin 

    Testing the System

    In order to make this configuration automatically apply to your load balancer, you need to do some configuration using the GlassFish application server administration console. So locate your load balancer in the navigation tree and make sure to check the Automatically Apply Changes checkbox in the load balancer configuration page, under the General tab. To test the system, add two time slices close to each other and let the system run for enough time to cross from one one time slice to the other, and then open the GlassFish administration console. From the navigation tree locate your cluster, check the Instances tab in the cluster configuration page. You should notice that the instance weights are changed accordingly.


    This article shows an small sample of using GlassFish's self-management features to perform some administration tasks that can be difficult and time-consuming to do manually. By looking at the sample code and concepts introduced in the article, you should be able to create your own MBeans and management rules to perform your daily administration tasks. On the other hand, AMX can be a good door for application server internal configuration and states. In the Resources section you can find many good references for going deeper into each concept or API that was introduced or used in this article.