In this post I introduce Samplr: an open source, intelligent sampling profiler that can be embedded in any Java application for automatic identification of performance bottlenecks
Tools of the Trade
If you read the Developer Power story in the July/August issue of Oracle Java Magazine, you already know that I love VisualVM. It is the best tool I know for inspecting the inner workings of a running Java system.
The feature I use most in VisualVM is the sampling profiler. This is a very non-intrusive, easy to use profiler that works by taking regular samples of JVM running thread stacks and aggregating the results over time. Attaching this profiler to a live JVM is a quick and effective way to pinpoint hot spots in code. The sampling profiler is lightweight enough that it can be safely attached even to production systems.VisualVM Sampler snapshot view
Working as a consultant, I have had numerous opportunities to use the VisualVM sampler to diagnose performance problems in live applications. When faced with a performance issue my usual approach is to try to reproduce the problem in a controlled test environment, and then use VisualVM sampling to locate the performance bottlenecks in code.
This approach does not always work. Sometimes the right kind of test environment is not available. Other times the test environment is available but its not an exact replica of the production system and hece the problem does not happen there. And even with rhe right kind of environment sometimes the conditions that trigger the problem in production are not fully known.But since the sampling profiler is so lightweight, it is often possible to just connect it to a live production system and observe its behaviour firsthand.
Sampling a production system is not devoid of challenges: it requires de cooperation of Ops people (whom are not always easily convinced that sampling is safe in production environments) and lots of patience as one have to sort through a lot of thread stack information in order to locate the threads processing the relevant requests.And if your production system is clustered (which is often true) it is not practical to connect VisualVM to every single cluster node.
After some time using this great tool, I started to wonder: wouldn't it be nice if it was possible to show the sampling profiler the location of request boundaries both in space (code locations) and time (begin and end) and then have it sample just the right threads at the right times? It would be even nicer if this functionality could be somehow embedded in an application so that slow requests would be automatically recognized and sampled in production system without the need of Ops help or manual intervention.
A few weeks ago I stopped daydreaming and created Samplr, an open source sampling profiler based on and compatible with VisualVM that can be easily embedded into any Java application.
Samplr is designed to be included as a library in any kind of Java application. Once initialized, it starts a supervisory thread that is notified by applications when requests start and when they finish. These notifications associate the request with a specific thread and time slot.
The following sequence diagram depicts the interaction between the Samplr threads and the request processing threads:Samplr example sequence
Once the monitoring thread identifies that a request is interesting, it wakes up a sampling thread that will take regular snapshots of the target thread stack. The definition of an "interesting" request depends upon how Samplr is configured. In the example above, Samplr is configured to sample only slow requests (requests that take longer than a predefined time threshold to complete).
Samplr is distributed as a self contained JAR file which should be incorporated into the target application using the standard library inclusion methods.The application then needs to initialize Samplr by creating an instance of RequestManager. There needs to be just one instance of this class per application, and it should be initialized with a set of parameters that tells it what requests to look for, how to sample them and how to store the results. An example initialization is provided below:
The code snippet above shows Samplr being initialized with instructions to sample requests that take longer than 30s (30000ms). It will record its results in the /home/glassfish/samplr-output directory. It will sample requests until they finish or for 3 minutes (300000ms), whatever happens first. More initialization examples can be found on the class RequestManagerTest.java.
Once RequestManager is initialized it needs to be notified about request boundaries (start an finish). Requests are represented by the Request class and can be anything: a method call, a web request being processed by a servlet or any other unit of work. You can subclass Request in order to provide contextual information that needs to be recorded alongside the thread sampling in order to help debugging. An example is provided below:
Notifying Samplr about request boundaries is very simple: just call RequestManager.requestStarting and RequestManager.requestFinished at the appropriate code locations and Samplr will do the rest:
Collecting and Analyzing results
Samplr currently supports saving results to a filesystem. Once the sampling finishes, the request information and sampling results are saved into the configured output directory. Each request is saved in a unique sub-directory with the structure depicted below:
The most interesting file is request-sampling.nps: this file contains the saved thread sampling information in nps format. This file can be loaded and analyzed in either VisualVM or in the Netbeans IDE. The resulting visualization can be seen in the following screenshot:
Samplr nps files contain information only for a single request, which was flagged for sampling by the initialization criteria. This makes analyzing Samplr results much easier than analyzing VisualVM results, as all irrelevant information is already filtered out.
The files request-info.txt and sampling-info.txt contain contextual information about the sampled request that can further help understanding the results.
Current status and future directions
Samplr is very young, but is already being used in one production system, with great results. It has been designed for extensibility and has a lot of room for improvement. A promising improvement area is request boundary demarcation: currently request boundaries must be programmatically demarcated. Here are some ideas I plan to implement eventually in order to make using Samplr easier:
- A Servlet Filter that automatically initializes Samplr and register requests, enabling automatic profiling for slow web requests.
- An EJB 3.1 interceptor to do the same for EJB method invocations
- A Java agent for transparent integration into running systems through bytecode instrumentation.
Samplr is free and open source. You are welcome to try it out and also to help make it better.