1 2 Previous Next


19 posts

Moving to Blogger Blog

Posted by carcassi Oct 18, 2012
I'll continue the overview of the design, focusing on the fluent API. We left with some basic spec for the object that do the live processing of the data. A user will need to create a whole set for each pipeline, plus the set of instructions to give to the data sources (what data to read, where to put it and who to notify). We need to construct and expression language, which allows to: * define each part of the pipeline * allow to construct an expression by reusing, mixing and matching different parts * allow different users to add their own pieces without requiring modification of the framework

Expression classes

At the end of the day, we'll want something like:
PVReader reader = PVManager.read(<read expression>)...
For example:
// Read the channel named "my channel" at maximum 10 Hz (i.e. will skip the extra)
PVReader<Object> reader = PVManager.read(channel("my channel")).maxRate(ofHertz(10));

// Queue and read all the new values for channel "my channel" at maximum 1 Hz (i.e. all notification will arrive, for batch processing)
PVReader<List<Object>> reader = PVManager.read(newValuesOf(channel("my channel"))).maxRate(ofHertz(1));

// Read the channel named "my channel" at maximum 1 Hz (i.e. will skip the extra)
PVReader<Map<String, List<Object>>> reader = PVManager.read(mapOf(newValuesOf(channels("channel1", "channel2", "channel3")))).maxRate(ofHertz(1));
In other word, we need to say "read this", and we need a way to say what this is. We may be tempted to have one Expression class. But if you look closely, some expressions (like channel) will get processed at the data source rate, some other (like mapOf) will get processed at the desired rate, and some (like newValuesOf) will convert a source rate expression to a desired rate expression (i.e. will implement one of those Collectors - queues, caches and the like). So, we really need at least 2 kinds: a DesiredRateExpression and a SourceRateExpression. So we can define, for example:
public static SourceRateExpression<Object> channel(String name) {...}

public static <T> DesiredRateExpression<List<T>> newValuesOf(SourceRateExpression<T> exp) {...}
But if you look closely, again, some expressions (like channels - note the plural) actually give us a list of expressions. So we need extra 2: DesiredRateExpressionList and SourceRateExpressionList. So we can define:
public static SourceRateExpressionList<Object> channels(String... names) {...}

public static <T> DesiredRateExpression<Map<String, T>> mapOf(DesiredRateExpressionList<T> expressions) {...}
And so on. If we also want to support writable expressions, we will need a WriteExpression and a WriteExpressionList. If we want the same expressions to be read and write, we will need a DesiredRateReadWriteExpression and SourceRateReadWriteExpression. So, 5 single expression (source, desired, write + 2 combinations), and 5 lists (one for each). This is where Java is pain, though: since we don't have multiple inheritance, these all need to be interfaces. But, since we still need to provide implementations, we need provide an extra 10 classes. Moreover, interfaces cannot have package private members, which means we may have to expose more things than we strictly need. So, the org.epics.pvmanager.expression has 10 interfaces plus 10 implementation classes. Kind of a pain. Fortunately, this is only a burden when you are implementing new expression, not when you are using the API. When you are using it, you get an expression fully checked: if an operator needs a expression read at the source rate, that's the only thing it can get; if it requires a list, it can get both a list or a single expression (which is a list of one); if it needs a number, it gets a number, and so on. Any user can create his own operators, mix them with the others, and the power of the API is the combination of all the different operators. Separate note: I'd like to put more code examples, but I have to find a better way to write. having to escape each < is really a pain...

Building in the cloud Blog

Posted by carcassi Aug 10, 2012
This week I moved the continuous build of 5 projects to the cloud instance of Jenkins provided by Cloudbees. Here's a brief summary of my experience.

Backstory (for those interested)

A few years ago, I setup a Jenkins/Hudson instance. It started off to build my project, but it became the host for a few other projects within the EPICS collaboration. With the growth, I had the following problems:
  • It's a pain to maintain. The system was supported using breadcrumbs of time from myself and the network administrator who, with the commissioning of the facility at the door, has been increasingly busy. So even things that should be simple, like add Java 7 support, would effectively never get done.
  • I can't really give full permission to outside collaborator. To be able to give it following the cybersecurity policy would have been a hassle (and rightly so).
  • The special status of BNL (the facility I work for). Other collaborator may feel funny telling their management that the software they are writing is being build somewhere else... like admitting in some way that they can't do it themselves.
So, for these and other reasons, I started to look at having a Jenkins instance running somewhere else. So I started to look at Cloudbees. After a few false starts, this week I sat down and did it. It took, in the end, only a day and a half. Here's the result: https://openepics.ci.cloudbees.com/.

Building a Maven project hosted SourceForge project

All 5 projects are:
  • Hosted on SourceForge
  • Maven projects
  • Have a maven site to be deployed
  • Deploy their artifact in a maven repository also hosted on SourceForge
The first thing to do was to create a user account for the build, in my case epics-jenkins. This way, when the build contacts the forge, it is clear who it is and we can give appropriate permissions. The next thing is to authorize the Jenkins/Cloudbees build to authenticate as epics-jenkins: when we create a job, we can see the public key that will be used (it's the same one for a whole account); we can grab that and add it to the keys used by the epics-jenkins account on SourceForge. After that, we should add epics-jenkins to all our projects, making sure that we add enough permission so that it can access the sourceforge shell/scp/rsync services. We also need to setup the credentials for when we are going to deploy the artifacts that are the output of the build. We need to give maven a settings.xml with the specifics. There is already a tutorial on how to do that. This was improved since the last time a looked, and it now presents a good option for Windows too. The next thing to do is to setup our project. Given that these were maven projects, that was  very straight forward (there is a maven specific job in Jenkins/Hudson). What makes it easier is that they have added a custom pre-step and post-step to the maven task, which is essential. To do a maven deploy, you need to pre-create a shell on sourceforge for epics-jenkins, or the scp will fail. So, in your pre-step you'll need to execute the command:
ssh epics-jenkins,epics@shell.sourceforge.net create
after that, the deploy will actually work. I also hadto add the scp wagon (by default in maven 2, needs to be explicitly added in maven 3). For deploying the site, I actually prefer to let it build locally and then use a post-step:
rsync -r -e ssh --delete target/site/* epics-jenkins,epics-util@web.sourceforge.net:htdocs/
As the final step, we should not forget to add: The build has now been working well for this week. I started to give access to other people in the collaboration. I still need to move a number of other projects (not all with a build as simple as the first 5 projects), but so far so good!

I've been working for a number of years now in the NSLS-II Control System group, creating tools that hook up to the control system. What I do is soft-real time stuff (I can drop data on the floor, I don't have hard latency requirements, etc.)  mostly dealing with writing clients to display the data and let operator interact with the control system.

In these conditions, you have have to think through all your threading, throttling, caching and queueing, or you'll have unresponsive UIs that do not even show current data. Surprisingly, the problems are always the same, but they are tedious and easy to get wrong. I was able to put together a general purpose library that takes care of all the plumbing, but before going into that I though I would give an overview of the problems.

These issues are actually relevant to most UIs in any language, (typically the one that don't shine for responsiveness) so a brief introduction may be useful to many.

What not to do

Let's say you have some library that notifies you asynchronously (a control system, a web service, a file system, SNMP, ...), something like:

source.addNotificationListener(new NotificationListener() {
    public void notification(NotificationEvent event) {
        // Now what?

You are tempted to do something like:

source.addNotificationListener(new NotificationListener() {
    public void notification(NotificationEvent event) {

Which is horrible because UI subsystem are typically single threaded, so this is just a mess. So, you consult your toolkit documentation (or any blog since this has been blogged to death), and you figure out how to dispatch a Runnable to the UI thread. So you write something like:

source.addNotificationListener(new NotificationListener() {
    public void notification(final NotificationEvent event) {
        MyToolkit.asyncExec(new Runnable() {

Now this is less horrible, but you may still have a problem: the event may just point  to a buffer that is guaranteed to have the event data only within the callback. After that, it may get rewritten with a new event, so you may or may not be accessing the right data, without proper synchronization, which is still horrible. Even if you get the synchronization right, you still need to notify the UI whenever that object has actually changed, or it won't repaint leading to the worst thing you could have: the data in memory is not the data on screen. Better to have an application crash, die and display no information at all than display the wrong information!

So you may need to make a copy, and an immutable one would be better. If it's mutable, you may need to worry about locking and check what effect can the locking have on your UI thread (every ms lost there, is lost forever).

OK, so you get the javadocs, you figure out that you indeed need to make a copy, you make it immutable, and you have:

source.addNotificationListener(new NotificationListener() {
    final Bar barCopy = copyBar(event.getBar());
    public void notification(NotificationEvent event) {
        MyToolkit.asyncExec(new Runnable() {

Good, now you have something that at least writes the right data on the right thread... which works ok when it does, and it's horrible under load. See: you are pumping tasks in the (single) UI thread at the rate that you get from your source (possible from multiple threads). Say you have 100 listeners, and each notifies at 10Hz: now you have 1000 notifications a second hammering the (single) UI thread. Which is, again, horrible.

First, this may be a complete waste. The screen refreshes at 60/70Hz, so any update faster than that is useless. If it's text you are displaying, and you are changing it 50 times a second, who is going to be able to read it? Updates faster than a couple of Hz may be more "precise" but can be counterproductive in this case.

Second, you can start queuing up tasks faster than the event thread can process, which can completely clog the event thread making the ui unresponsive, and can, after some time, exhaust the memory you have available, and crash the whole application (if lucky) or leave it in a completely undeterminate state (if not lucky). Not a good thing.

And here is the main take-home message: when you are listening to asynchronous events, you need to decouple the rate at which they are received from the rate at which they are processed. You often need to aggregate, so that a number of events on one side generates a single event on the other. And while you are it, you may want to do the heavy lifting computation, so that your UI (if you have it) gets only what it needs to put on screen when it's done.

What you need

In the vast majority of cases, you'll end up with some incarnation of this:

On the right half you have processing that goes at the rate set by the source. On the left, you have processing at a rate set by the destination. Don't be fooled: there is always an optimal rate above which things either get very inefficient or extremely useless. You need to show data to the UI? Above 50 Hz is going to be useless. You have to write data in a database? You may not want to send more than a query a second. You need to write to disk? You may want to fill a buffer. So, even in the optimal case, you'll have to decouple rates.

In the middle, you have what I call the collector: it takes data at one rate, and it is being asked at another. This is where the synchronization needs to be done. And this is where you have to decide what to do with the extra notifications: do I only need a cache with the last one? or do I need a cache with the last N? or do I need a cache with the ones that came in the last N seconds? or do I want a queue, with only the new ones? And if the queue fills up, which data should I dump? Should I save it to do disk? And if the disk is full? Should I send an alarm to someone?

Between the source and the collector, you may have some processing. You may want to extract only the data you are interested so you don't waste memory in the queue, or you do some pre-computation on it, including the copying we talked about.

On the other side, you have what I call the scanner: it starts the processing at the desired rate. This could be a timer, could start based on some notification from the collector (maybe based on how much data was collected), could be triggered by readiness from the target subsystem. Whatever is the logic that starts it, the scanner will need to throttle back: it will need to adapt the rate based on what the target subsystem can handle. Suppose you want to update your UI at 50 HZ, but your UI thread is busy: you need to skip, or you are just going to compound the problem. Same thing if your database is not responding as fast as you'd like. By skipping and consolidating the processing in fewer notifications, you will typically have a system that is sturdier and much better at "catching up" when it's behind.

The scanner will trigger some other processing, aggregating the data both in time and from different sources, computing the end result. And in the end, you will ship the final result to the target subsystem. In the UI case, it will be the queueing to the UI thread of your toolkit.

Plenty of cases fit this picture, and, once you look at it this way, it's actually pretty simple. But it's tedious: you have to do all these things, always, for all the applications that you are building, and you have to test that the collector is doing it right, that the scanner is throttling back, ... If you, like me, need to crank different applications, then you'll want a framework to do this. A framework that can work with whatever the destination needs to have and the source gives. A framework that does not care whether you are using Swing, SWT or if you have a UI at all! Where you can unit test the pipeline with mock datasources and without the UI. And that is what I have been working on. And that's the topic for another post.



Generics puzzle Blog

Posted by carcassi Aug 3, 2011


You know that generics are too hard when you can't even figure out why something does not compile.

Found this (simplified example of something that I have encountered):

 interface Test1<T> {
 public List<T> getList();
 public void test1(Test1<?> arg) {
 List<?> result = arg.getList();
 interface Test2<T> {
 public List<Collection<T>> getList();
 public <T> void test2Typed(Test2<T> arg) {
 List<Collection<T>> result = arg.getList();
 public void test2Captured(Test2<?> arg) {
 List<Collection<?>> result = arg.getList();

test2Captured() does not compile:

org/epics/pvmanager/expression/GenericTest.java:[33,48] incompatible types
found : java.util.List<java.util.Collection<capture#878 of ?>>
required: java.util.List<java.util.Collection<?>>

Now, I am not casting anything. I am just assigning to the same type. Yes, it's a different capture, but I don't see what I could do that I couldn't do with a direct call to the argument... And why would test1() compile? I don't see the difference...

What am I missing?


If you, like me, need to build libraries that need to be tested against a live server (integration tests), here's what you can do.

Have all your integration tests either in seprate "itest" packages, such as my.library.itest, or have the test classes end in ITest, such as my.library.ConnectITest. In the main section, have the surefire plugin exclued running the integration tests:


 This way, by default, a freshly checked out copy of your project will build successfully (regardless of whether the infrastructure for the tests is present). Then add an "ITest" profile that configures surefire to remove the exclude:


You can also add an property to specify the location of your test service (so that you are not hardcoding the location, and if you accidentally include an integration test with the unit tests, you see it failing during a normal build because it cannot find the location). When you are done, if you run "maven build -PITest" you will run all tests (including the integration tests) and if you run "maven build" you'll only run the unit.

Within NetBeans, you can right click on your Maven projection, "Set Configuration" and select ITest, at which point you will always be running all tests. Or you can go back and select the default, and run only the unit.

I have started to look for some framework to put together in the same web page different contributions from loosely coupled applications. After reading about claims of portlet's demise, I was pointed to OpenSocial, Shingdig and the like. Now, I couldn't care less about the "Social" part of this (I am writing applications to manage data of particle accelerators and light sources... not teenage romantic life) but the ability to break down things in gadgets and have the option of assemble pieces coming from loosely coupled applications written in different languages (as long that their output is html), that's something I'd like. And, as I understand, I could use Shingdig on my own internal, set it up...

Now, the interesting part was: how long did it take me to take one of the applets I made and create a gadget out of it? And make it appear in my iGoogle (which is implemented on top of OpenSocial)? Less time than is taking me to write this blog entry...

You create an XML file such as:

<?xml version="1.0" encoding="UTF-8" ?>
  <ModulePrefs title="IRMIS Component Browser" author_email="carcassi@bnl.gov" height="600" width="800">
    <Require feature="opensocial-0.8" />
  <Content type="html">
    <applet height="100%" width="100%" code="gov.bnl.irmis.applets.ComponentBrowserApplet"             codebase="https://irmis-dev.bnl.gov/IRMIS/lib" archive="IRMISApplets.jar" id="applet">
    <param name="irmis.service" value="https://irmis-dev.bnl.gov/IRMISDataService" />
    <param name="java.util.logging.config.file" value="https://irmis-dev.bnl.gov/IRMIS/logging.properties" />
    <param name="uploadLog" value="https://irmis-dev.bnl.gov/IRMIS/uploadLog" />
    <param name="progressbar" value="true" />
    <param name="draggable" value="true" />
    <param name="separate_jvm" value="true">
    <param name="java_arguments" value="-Xmx256m">
    <param name="image" value="IRMIS360.png">
    <param name="boxborder" value="false">
    <param name="centerimage" value="true">

Note that there nothing in the html except an applet tag, stretched to take the whole height and width. Then you make it available at a public URL, open your iGoogle page, give the URL of the xml file and your applet is in iGoogle! (which I can't show because the attachment upload seems to be broken...)

 This is great for me, but I think it's even more important for someone looking into using JavaFX in this context. And there are a couple of things that definetely need fixing, which are generic applet plugin support and probably have nothing to do with OpenSocial/iGoogle specifically:

  • Cursor. If an html popup appears over the applet, the cursor keeps oscillating between what the applet wants and what the popup wants.
  • Mouse drag. If you start a mouse drag anywhere in the page (say you are moving a gadget) and the drag passes over the applet, the drag is terminated. So, you can't drag _over_ and applet, you need to drag _around_.
  • Alpha. If a popup appears and grays out the page underneath, the applets are not greyed out (probably painted on top) so it shines like a punch in the eye
  • Shared jvm per codebase. If you break up an application into multiple gadget, it would be nice if they could share the same jvm so that the applets can communicate (selection in one changes the other), minimize library load and share data (my applets need to get about 60 MB of data...)

Note that the first three are _not_ a problem with Flash.

It is generally a good idea to ask the user before closing an application without saving the data. If you deploy an applet, the situation is even worse: the user may get distracted, follow some link, forget what he was doing, and close the whole browser without realizing he closed the applet too. Fortunately, it's very simple to tell the browser to display a warning.

Before closing a page for any reason (reload, closing the tab, close the browser, ...), a "modern" browser will call window.onbeforeunload. If you want to alert the user, you simply need to create a javascript function that calls your applet and returns the warning message.

Here is an example. This is an applet with a checkbox and a textfield. Note the two methods isPreventClose() and getPreventCloseMessage(), which define whether the applet needs to block the closing of the page and what message to prompt.

package test.applet;

public class MyApplet extends javax.swing.JApplet {

    public void init() {
        try {
            java.awt.EventQueue.invokeAndWait(new Runnable() {
                public void run() {
        } catch (Exception ex) {

    // To be called in javascript
    public boolean isPreventClose() {
        return preventClose.isSelected();

    // To be called in javascript
    public String getPreventCloseMessage() {
        return preventCloseMessage.getText();

    private void initComponents() {
        // Unintresting: simply creates the form 
        preventClose = new javax.swing.JCheckBox();
        label = new javax.swing.JLabel();
        preventCloseMessage = new javax.swing.JTextField();
        preventClose.setText("Prevent close");
        label.setText("Message: ");
        preventCloseMessage.setText("You have unsaved changes");
        javax.swing.GroupLayout layout = new javax.swing.GroupLayout(getContentPane());
                        .addComponent(preventCloseMessage, javax.swing.GroupLayout.DEFAULT_SIZE, 304, Short.MAX_VALUE)))
                .addGap(7, 7, 7)
                    .addComponent(preventCloseMessage, javax.swing.GroupLayout.PREFERRED_SIZE, javax.swing.GroupLayout.DEFAULT_SIZE, javax.swing.GroupLayout.PREFERRED_SIZE))
                .addContainerGap(javax.swing.GroupLayout.DEFAULT_SIZE, Short.MAX_VALUE))
    private javax.swing.JLabel label;
    private javax.swing.JCheckBox preventClose;
    private javax.swing.JTextField preventCloseMessage;


And here is the webpage that uses the applet. The important part is the javascript. The first thing is to register the function confirmExit() to the before unload event. The function simply calls the applet (referenced by the id myapplet) to see whether it needs to prevent the closing and what message needs to be displayed.


<title>Applet HTML Page</title>
<script type="text/javascript">
  window.onbeforeunload = confirmExit;

  function confirmExit()
    try {
      if (document.myapplet.isPreventClose()) {
        return document.myapplet.getPreventCloseMessage();
    } catch (ex) {
      // Can't access the applet: could be not even loaded

<h1>How to prevent an applet from closing (with unsaved data)</h1>
<applet id="myapplet" codebase="classes" code="test/applet/MyApplet.class" width="350" height="200">
<p>Enable the check box to prevent the applet from closing. The text in the textbox will be displayed in the pop-up.</p>


Note that if the javascript function returns null, the alert message is still displayed: you need to either let the function end with no return statement or return undefined. null and undefined in javascript are two different things.

Since generics were out and I started using them, there were always a few cases in which I couldn't make them do what I wanted. I always thought it was my problem, and that I didn't understand what was going on... Turns out: it's not. There are at least two things that are implemented in a way that break what I thought were very safe expectations.


Before Generics, you would write something like:

    List list = Collections.EMPTY_LIST;

Adding generics you get a warning:

    List<String> list1 = Collections.EMPTY_LIST;

GenericsGotchas.java:[19,40] [unchecked] unchecked conversion
found   : java.util.List
required: java.util.List<java.lang.String>

The problem is that we have a raw type on the right, and a generic type on the other. We could suppress the warning every single time, but there is a better option: using a generic method and let the compiler infer and match the types

        List<String> list2 = Collections.emptyList();

This compiles without a problem. But this does not:

        List<String> list3 = new ArrayList<String>(Collections.emptyList());

GenericsGotchas.java:[21,29] cannot find symbol
symbol  : constructor ArrayList(java.util.List<java.lang.Object>)
location: class java.util.ArrayList<java.lang.String

You would expect that it would, right? But it doesn't and you get an  ERROR, not a warning. If you really want to put in one line, you can use (as suggest by the comments) the following

        List<String> list3 = new ArrayList<String>(Collections.<String>emptyList());

The problem here is that the language is behaving in a way I was not expecting. My expectation is that whetever you have on the right side can be used directly as a parameter to a function. Generics break this expectation. That the follwoing refactoring:

    a = expression;

is always valid. Unfortunately, it's not true in this corner case. So, when I am refactoring my code, or decorating an expression, I have now one more thing to remember and to pay attention to.

Casting generics type

Casting of generic types is inherently unsafe because of erasure. We all know that. But sometimes we have to do it, we expect the compiler to give us a warning and we explicitly ignore it together with a comment explaining why. There is one case in which this does not quite work. And it comes up if you are using class tokens, so I'll use those as example. A simple class object:

    Class<List> class1 = List.class;

For a List of String you get:

    Class<List<String>> class2 = List.class;
GenericsGotchas.java:[24,41] incompatible types
found   : java.lang.Class<java.util.List>
required: java.lang.Class<java.util.List<java.lang.String>>

And this is an error, rightly so. So you may expect to simply add a cast, as you normally do in these cases for non generic types:

    Class<List<String>> class3 = (Class<List<String>>) List.class;
GenericsGotchas.java:[25,63] inconvertible types
found   : java.lang.Class<java.util.List>
required: java.lang.Class<java.util.List<java.lang.String>>

It does not work either. I understand that the compiler can't make the test, but the problem here is that I can't tell it that it's ok to make the conversion: it's an error, not a warning! So, how do you do it?

    Class<List<String>> class4 = (Class<List<String>>) (Class) List.class;

You cast it first to an Class, "forgetting" all of the parameter type, then you are free to cast it to whatever you want. You get the warning, which you can suppress. Yes, this is extermely poor style, but at least it's concise. The expectation is that I can cast one type into the other, and I get a warning if the cast can in theory be done but can't guaranteed at runtime. If your type parameter is generic (a List of a Map of something), this expectation is not met.


I am surprised that I haven't been able to find them discussed anywhere, so I felt compelled to write about it. At least you know how to work around them... I think that all the problems I have been having with generics are one way or another caused by some combination of these two. The rest (erasure, bounds, ...) they all make sense to me. These do not, they feel like bugs... Maybe there is something I am missing, but I don't see why these behaviours are needed.


In this post I am going to sum up things I have learned while creating a fluent API (or internal DSL) in Java. I'll talk about the search API I created for my current position: it's not a toy problem, it's a real problem, which has a significant amount of complexity. Because of that complexity, you get to see techniques and ideas that you don't usually see in toy examples. I am not including the full source, which (if you really want) you can access on the project's site.

The model

First, I need to tell you a little bit about the problem I am trying to solve. IRMIS, the tool that I am developing, needs to keep track of all the components of a particle accelerator and their relationships. For example, you want to keep track where a particular card is, how is it getting power, and how messages from the control system are addressing it. The GUI, for example, would display something like:

IRMIS Screenshot

Each component can be part of multiple hierarchies (control, housing, power), can have multiple children in each, can have multiple parents in each (though 99% of the time it has one). Each component has also a type and a set of properties:

public interface Component {
    ComponentType getComponentType();
    Set<ComponentRelationship> getParents(RelType relType);
    Collection<ComponentRelationship> getChildren(RelType relType);
    Map<String, String> getProperties();

It has a bunch of other stuff too, but to we can limit our attention to this. Each relationship will have:

public interface ComponentRelationship {
    RelType getRelationshipType();
    Component getParent();
    Component getChild();
    String getDescription();

With this bean-like interface you can, at least in principle, access all the information. Yet, it becomes kind of hard to express conceptually simple queries. For example, let's say you want to ask what are all the components that are powering all installed instances of particular type of device. You would have to start from the housing hierarchy, recursively iterate (to get all the components that are installed somewhere), filter out only the ones of the type you want, then from those get all parents in the power hierarchy. This is significantly complicated: you really have to write a script while the specification is really just a sentence.

What I have built is a fluent API that allow you to write that query as:

parents().in(POWER).of(allComponents().where(componentType().isEqualTo("My Type")));

It is not so far from a verbal request and it is similar enough to SQL. It took me more than a couple of months in incremental iterations to reach this design, but once I had it was very very trivial to implement and I was kind of amazed how things fell into place very well. Designing a similar API would take me significant less time now. So, here are some things I have learned that I am using in creating other fluent APIs.

The design end goal

When I started, I didn't really know what the end product should look like. I went around to the usual places, but I found only basic stuff. I look at some libraries for ideas but that was about it. I didn't have a good "recipe".

I came to realize the following: there are two distinct APIs I was actually building. One is the proper fluent API, the one that allows me to actually express the query. The second one is the execution API, the one that allows me to run the query. The two are connected to each other (whenever I add a new expression, I need to implement the search that uses that expression) but you have to de-couple them (as you may want to completely change how the queries are executed without changing how they are written and vice-versa).

The other key problem is that you will want to break down the fluent expression in chunks that you can recombine and use. For example, we may modify the previous query in the following ways:

children().in(POWER).of(allComponents().where(componentType().isEqualTo("My Type")));
parents().in(POWER).of(allComponents().where(componentType().isNotEqualTo("My Type")));

What you want to do is to be able define each piece of the expression independently, so that it can be reused and other pieces can be added or removed as the API evolves. And whenever you add a new expression, you add only the minimum piece of code to execute that part of the expression within the query. It may sound very very complicated, but it's surprisingly not!

Let's start by identifying these sub-expression. For example, I'll need to be able to describe component relationships (parent, child, descendent, ancestor, ...), filtering expression, and the attribute of the component (the type, a key value property, ...) on which is going to be part of the filtering expression. Each of these element will be one class, and I will first define the execution part of the API. For example:

public abstract class ComponentHierarchyRelationship {

     * Returns the components related to the given component according to 
     * this hierarchical relationship.
    public abstract Set<Component> getComponents(Component component);


Each hierarchy relationship is something that given a component, returns a set of components: the parents instance will take a component and return the parents, a set of componetns. To implement each different relationship, we simply implement that abstract method:

    public static ComponentHierarchyRelationship children() {
        return new ComponentHierarchyRelationship() {
            public Set<Component> getComponents(Component component) {
                return ...;

Each instance should be defined as public static, so that it can be added through a static import and it becomes one of the tokens that your fluent API understands. You have a choice of methods or variables: it seems equivalent to me, so pick one and be consistent across the API. You may also create once and cache it (if needed). You should also consider making the execution API private: it boils down whether the user should be able to define his own tokens and "extend" the language. Make public if you really need to allow that. Just to iterated: you add and remove elements of the same type simply by adding and removing static factory methods with the same class.

Now that we see how to define and implement the execution part, let's concentrate on the fluent part. Consider the following class that represents a component field:

public abstract class ReadOnlyComponentField {

     * Returns the value of this field for the given cmpnt.
     * @param cmpnt a component
     * @return the value of the field
    public abstract String extractValue(Component cmpnt);

And the following class that represent a filtering condition, like "the type is foo" or "the owner is bar":

public abstract class ComponentFilter {

     * Returns true if the filter accept the given component.
     * @param cmpnt a component
     * @return true if the component passes the filter
    public abstract boolean accept(Component cmpnt);


Now we add an operation on the field, so that I can express whether a particular field has a value:

public abstract class ReadOnlyComponentField {
    public ComponentFilter isEqualTo(final String value) {
        if (value == null)
            throw new NullPointerException("Value should be not null");
        return new ComponentFilter() {
            public boolean accept(Component cmpnt) {
                return value.equals(extractValue(cmpnt));

Few things to note. The method is added to the abstract class: all instances are going to support this operation. The return type is another class that represent an element of the fluent API: this is how we connect all the pieces together of the fluent API itself. The null pointer check is executed as part of the fluent API, so the exception is raised when the user is putting together the expression. The code within the inner class, instead, runs during the execution of the query itself. The inner class code uses the abstract function to perform the operation: this is how we connect all the pieces of the execution API.

Now consider this:

public abstract class ComponentFilter {

     * A helper class of boolean operations on {@code ComponentFilter}s
    private abstract static class BooleanOperation extends ComponentFilter {

        ComponentFilter filter1;
        ComponentFilter filter2;

        public BooleanOperation(ComponentFilter filter1, ComponentFilter filter2) {
            this.filter1 = filter1;
            this.filter2 = filter2;


     * Returns a filter that is the logical OR of this filter and the given
     * filter.
     * @param anotherFilter a {@code ComponentFilter}
     * @return a new {@code ComponentFilter}
    public ComponentFilter or(ComponentFilter anotherFilter) {
        return new BooleanOperation(this, anotherFilter) {
            public boolean accept(Component cmpnt) {
                return filter1.accept(cmpnt) || filter2.accept(cmpnt);

     * Returns a filter that is the logical AND of this filter and the given
     * filter.
     * @param anotherFilter a {@code ComponentFilter}
     * @return a new {@code ComponentFilter}
    public ComponentFilter and(ComponentFilter anotherFilter) {
        return new BooleanOperation(this, anotherFilter) {
            public boolean accept(Component cmpnt) {
                return filter1.accept(cmpnt) && filter2.accept(cmpnt);

This, as all the comments suggest, implement boolean operations. And remember: these are now going to apply to all filters, however defined. So we can have:

    componentType().isEqualTo("My Type").and(property("owner").isNotEqualTo("carcassi))

And so on. The rest is pretty much defined in the same way: the whole query represents a set. A set itself can be filtered, with a where that accept a filter. ComponentHierarchyRelationship.of can accept a set itself, so we can have nested/recursive queries. To add and remove vocabulary without changing the overall grammar, you simply add/remove static factory methods or add/remove methods of the class.

The end result of your design should be the list of classes that represent the expressions of your fluent API, the list of static factory methods that represents the tokens of your API and the list of operators you allow on each expression. This will take many iterations to get right, but knowing these are the kind of things you want to have in the end is helpful.

Sentence structure

When designing the fluent API I noticed it pays to be aware of what role each fragment plays and be clear of the natural language grammar (not Java or BNF that is). For example, a component field will be a noun, the component filter will be a subordinate clause, the component set (which is also a whole query) is noun sentence fragment, which could contain subordinates (i.e. all components where the type is foo). You need to understand what these are, be strict and make sure that whatever prepositions or verbs you use to combine them always makes some sense.

Some rule of the thumb: subordinates should always be passed as an argument and not chained. For example:

childrenOf().childrenOf().allComponents().where().componentType().isEqualTo("My type")

is a really bad idea (I know: I have tried). It couples together the whole API and you can't break pieces and reuse them easily. You also lose the whole nested structure of the sentence, which actually makes it easier to implement the API.

Coordinates should be chained with a conjunction or adverb but it should still be passed as an argument. For example:

componentType().isEqualTo("My type").and().property("owner").isNotEqualTo("carcassi") 

is bad: pass the coordinate as an argument to end. Make also sure that your conjunction really work for all cases for example:

componentType().isEqualTo("My type").but(property("owner").isNotEqualTo("carcassi"))

works, but this:

componentType().isEqualTo("My type").but(property("owner").isEqualTo("carcassi")) 

does not. Don't try being too cute or you'll get into these situations. 

Another unexpected interesting grammatical feature I encountered is the use of singular and plurals. For example I distinguish between parents() and parent(). The first returns all the parents while the second makes sure that, during the execution, it either returns one component or throws an exception. This is useful because multiple parents in a single hierarchy is a fairly unusual thing (redundant power supply for example) so in many case you simple expect components to have a single parent. By choosing parent() you can enforce that assumption.

What should come across is that designing a fluent interfaces is not just about method chaining. It's really about building up an ad-hoc language inspired by the natural domain language. But, at the end of the day, you are still designing an API so it's not prose either. You'll get sentences that kind of sound like English but aren't. Thinking about the domain sentence structure does usually give you hints of how your design should proceed. (In the same way that looking at the domain knowledge often tells you what classes and objects you will need).

Restricting expression scope

Another trick useful trick I discovered is about limiting the scope of the operation. For example, I do not want to allow the chaining of two where in the expression:

allComponent().where(componentType().isEqualTo("My type")).where(property("owner").isEqualTo("carcassi")

But I also want to be able to define operation on a set, regardless of the presence of the where clause. In this case, you can use subclassing, and let the method return the upper class, so that fewer and fewer methods are available.

allComponents().where(componentType().isEqualTo("My type")).andTheir(descendents())
UnfilteredSet -> FilteredSet                              -> ComponentSet

The first method returns an UnfilteredSet, which extends FilteredSet to define the where() operation. The where operation returns a FilteredSet which extends the ComponentSet to define the andTheir() operation. The ComponentSet has no operation and just defines the method for the execution API. This means that one can't have two where() calls and can't invert the order of where() and andTheir(). The implementation is also easy because the methods of the parent class can simply ignore the optional parameters of the child class: you implement at the narrowest scope.

This method works quite well, especially with the auto-complete features of the IDEs. The problem is that it proliferates classes, so use it judiciously. But where you need it, it works great.

Another way of achieving this is by using exceptions instead of the type system. For example, every hierarchy relationship needs to allow to specify whether it applies to all hierarchies or to a specific one. The way I implemented this is by having the following method:

public abstract class ComponentHierarchyRelationship {
     * The hierarchy type that this relationship refers to. If null, refers to all
     * types.
    protected RelType relType;
     * Limits the relationship to a particular hierarchy.
     * @param relType a hierarchy type
     * @return this
    public ComponentHierarchyRelationship in(RelType relType) {
        if (this.relType != null) {
            throw new IllegalStateException("Relationship type already set");
        this.relType = relType;
        return this;

By default parents() refers to all parents and it can be refined to parents().in(HOUSING), but a double call will result in an exception. You can extend this even to require a particular ordering (if foo is already set, setting bar would throw an exception, forcing bar to be set before foo). This strategy does not pollute the type system, but does not give you compile time safety and it complicates the implementation (you need to check at all possible options and switch between those instead of having the type system doing it for you). Again, the exception is thrown while the expression is being created, not during the query execution.

Sentence order and ending problem

One of the well known feature of fluent API is that they reverse the order of tokens, and it gets a while to get adjusted. In English you don't say:


we say:


A related problem is being able to tell when you stop a method chain. For example, to convert from a FooBuilder to the actual Foo you may have:

Foo foo = foo("the name").owner("me").build();

Here are a couple of ways to avoid that. We are typically used to put the most important attributes first: if the name is required, we pass it as an argument to the constructor and that comes first. But if name is really compulsory, this actually gives us a natural way to end the chaining:

Foo foo = newFoo().owner("me").andName("the name");

The ending problem itself can also be seen as a sentence order problem: we say "build this" and not "this build". And "what is being built" is actually a subordinate clause, which can be arbitrarily complicated. So, we could theoretically and more properly have:

Foo foo = build(aNewFoo().with(owner.equalTo("me").and(name().equalTo("the name"))));

With all the gradation in between

Foo foo = aNewFoo().with(owner.equalTo("me").and(name().equalTo("the name")));
Foo foo = aNewFoo().with(owner("me").name("the name"));

Your taste and your requirements may drive what is better in you case. But you should have these two alternatives in mind: required expression at the end or use endMark(expression) instead of expression.endMark()

Also, if your natural language allows verbs at the end (like German) you can do that. Or you can use the politeness API pattern:

Foo foo = foo("the name").owner("me").please();

Nothing happens if you forget to say please!!! ;-)


In designing a fluent API there are different considerations you have to make, I hope this gives you a glimpse into that. It helps if you have studied about languages and grammars (the computer ones this time). But in the end it's just practice. I still doubt that one can write a whole API to be fluent. What I did was to have the basic building data objects as immutable beans (get thread safety) so the read API is "standard bean API". The search and the write, though, are more like fluent APIs. It seems to work well.

The thing does not work well and I don't really know how to solve is documentation: JavaDocs is not at all suited for this task... and the documentation is too scattered. I have example of the overall usage at the package level and at some of the key classes, but one would really want is to have it automatically documented like you document SQL or other special purpose languages.


While much have been written on checked vs unchecked exceptions, I found very  little practical, down to earth, advice on how and why to use them. Here are my  thoughts after years of rumination.

The common wisdom

What we are typically told is to use checked exceptions for recoverable  conditions and unchecked exceptions for programming errors. For example here or in Effective  Java. Don't get me wrong: I do not think this is particularly bad advice,  but there are a couple of problems.

First of all, what a recoverable conditions is is vastly subjective. This  also gives you the false impression that unchecked exception are not or should  be not handled, which is false as we'll see in a minute. But worst of all it  does not tell me why! Why should I use checked exception for failures I may  expect someone to recover? The horrible answers that I sometime get are in line  of "to force the user to think about those cases". And this is a pile of  nonsense: the only thing that forces him to do is to put an empty catch block!  If he can't be bothered to think about failure modes (for whatever reason) it's  not the compiler that is going to change is mind. So, how are checked exception  better than, say, just documenting the failure mode? What do I get that I  couldn't get otherwise?

Unchecked exception handling

All exceptions are handled, one way or the other.  Either you catch them directly, or you pass them up the chain to whatever  framework you are using where a default error handling will be used. For  example, the default handler in a main method will print out the stack trace and  quit; the servlet container will return some error message, and process the next  call. Unchecked exception will trigger the default handler automatically.

By default, unchecked exceptions bubble up the stack and  are handled by the default mechanism the framework you are using provides.

To be complete: note that there are two kinds of unchecked exceptions in  Java: RuntimeExceptions and Errors. Frameworks will typically behave  differently: a RuntimeException is usually taken to signal that "this particular  operation failed", so you should abort it but may have luck with a different  operation; an Error is taken to signal "this failure affected the whole JVM", so  you should abort everything, not just the current operation.

Checked exception, instead, do not bubble up the stack automatically. Are  there cases where we do not want exception to go up the stack? Where being forced  to write:

  try {
  } catch (Checked e) {
    throw new RuntimeException(e);

is actually better? Is actually something we want?  That we are grateful for?

The problem with unchecked exception

The problem with unchecked exception is that by default  they break encapsulation.

This is true of any exception: to be useful it often has to provide what in  that particular implementation failed. But imagine this: you implement some cache for large amount of data, which uses  the filesystem to store what does not fit in memory. If you do not catch all the  IOException that may be caused by the file access, these will reach the client  code. The worst part is that the client may actually start handling those  exceptions. If you now switch implementation from files to a database, the error  handling is lost.

What you need to do is to define the failure modes of the cache, and then map  all the exceptions you get from underneath to those new exceptions. But the  problem: you must not let the exception from underneath leak through your  abstraction. With unchecked exception you have no way to  guarantee that no exceptions are leaking: you have to manually check all the  code paths.

Checked exceptions help you control in which layer of your  code failures are addressed, and that they do not leak up the stack.

This gives you more control in defining and guaranteeing the proper  interfaces for your code. Therefore the following code

  try {
  } catch (IOException e) {
    throw new UnsupportedFailureException(e);

is actually better in that sense. It encapsulate the failure  type.

The problem with checked exception

The problem with checked exception is that it is the layer  below that defines what exceptions the layer above may not want to expose.

This is completely backwards: it should be the layer on top that should  declare what exceptions it may not want to leak through. It's a problem because  different users of the library may have completely legitimate and contrasting  requirements, but they are stuck with one behavior.

This is especially annoying when there is no layer above. If I am  writing a script, there is nothing on top; a checked exception does not buy me  anything. I may still have to convert all the checked to unchecked just because  the default handler may only handles those, even if in this case that is the  desired behavior.

IMHO, this shows the tension with checked exceptions, and why some times they  are simply in your way. When you want to simply create a prototype, when you are  writing unit tests, when you are writing a script: these are all cases where you  are not interested in controlling all possible error flows, yet you are paying  some price. One may fantasize about ways to solve this, through annotations,  other language features. I am merely clarifying the problem.

New rules

Under this light, many things make more sense to me. In practice it does not  change much what one one do (it's still pretty close to the common wisdom) but I  can put down some rules that I think are more tight and provide more guidance.  Keep in mind the bottom line: checked exceptions are there to make sure that  failures encountered by a specific library or subsystem are only handled by the  correct library or subsystem on top of it. Therefore we can say:

If the error condition can be avoided by correct use of  the API, use unchecked exceptions.

The argument here is not that they should be checked too, but it would be too  verbose, therefore we allow this exemption; or that the code should not be  expected to handle them.  The argument is that you are never going to  create a layer on top handling all the failures caused by the incorrect use of a  library: you fix the original code! For example, NullPointerExceptions are fixed  by checking whether the variable is null, not by handling the exception. Same  thing goes for all the IllegalArgumentExceptions, IllegalStateException,  IndexOutOfBoundsException and the like. You don't want to make sure that they  are handled in the layer above, you simply want to help the user find the  problem and fix it. The default handling (abort the current operation  and report the error) is appropriate in these cases.

If the failure is one that a large number of subsystems  can encounter, use unchecked exceptions.

In this case, you do not have a specific set of failure that are generated by  a particular library, you can get the failure from anywhere. This means that you  cannot write a layer that isolates where such failures are handled, and checked  exceptions cannot help you make sure those failures are handled in on place.  Take OutOfMemoryException: every object allocation could throw it, it is not  even possible to imagine a layer that handles them all without generating  others.

Other Errors, such as UnsupportedClassVersionError, AssertionError,  ..., fall in this category, and we should point out that  they are Errors because after that every piece of code is very likely to be  subject to another failure, so the appropriate action is to abort everything,  not just the current operation. Again, the problem is not that it is impractical  to have them as checked exceptions because they happen everywhere: it is that,  because they happen everywhere, we cannot isolate where the problems happen and  where they are solved.

A corollary of this is:

If the API does not provide access to a specific  subsystem, and is meant to be used at large,
prefer unchecked exceptions which can be avoided by correct use of the API.

If you are writing a library, for example a data structure library like the  Java Collections, that is meant to be used anywhere in the code, then, by  definition, the same failure could be encountered by any subsystem. Now you  would have the error generated by all the uses that potentially merge with each  other. The code that handles the failure of the layer underneath may well be  using that library too, and we get in a case similar to the above one. Checked  exceptions are of little use here because it starts to be less clear who  generates what and who handles what.

In this case, it's not that you should use unchecked exception: you should  strive to write an API where error conditions can be avoided by correct use. The  Collections API, Pattern/Matcher for regular expression, they all use this  strategy. With every checked exception, you are implicitly telling your user  that he needs a layer on top, which is not acceptable if your API is really of  general purpose use.

The remaining cases are failures that are specific to a  subsystem, and that cannot be avoided by correct use of the API.
Use checked exceptions for these.

It's as simple as that, really. Now the user will be able to write a layer on  top of your API where he can guarantee to himself that the failures are not  going to leak. He can implement a database on top of file IO. Another can  implement an object broker on top of the database. And they can all trace where  the flow of failures starts and stops. These exception defines how this  particular subsystem can fail. Which is not all the failures this subsystem  can have: it will still have the ones that it shares with all the others, and it  has the ones that can be avoided through correct use of the API.

If you are catching a checked exception, and you do not  know what you should do with it yet,
wrap it in an UnsupportedFailureException.

I really think such exception should be part of java.lang in the same way  that UnsupportedOperationException is: IDEs could generate instead of empty  catch blocks, blocks that throw this exception in the same way that they  generate method stubs that throws UnsupportedOpertaionException. And when you write  your unit test first, it would fail with that exception.

For now, create one in your project and use it everywhere. The reason you do  this is obvious: you do not want your client to handle that, because you  probably haven't thought exactly what are the consequences of such error anyway.  Every subsystem may actually have some failure modes that are unsupported and  the correct behavior is the behavior provided by the default handler. And you  get documentation for free.


When I explain this to people, they tend to think it's better explanation  than what they could find. It makes sense and accounts for the tension there is  between the "checked people" and the "unchecked people". In practice it doesn't  change much what one should do, except give more understanding of why we do what  we do... which may actually be a lot!

The second generation plug-in really made me reconsider applet as a viable deployment method (though people who are not aware of the improvements think I am crazy). For the project I am working on (http://irmis.sourceforge.net/) we decided that the user interface would be a series of applets that communicate through a Java API to a REST Web Service. The second generation plug-in was really essential to be able to do that. The ability to have each application run in a separate JVM, the ability to give each JVM more than 64MB and the ability to use javascript on top were all necessary.

One thing in particular, though, I think is needed: JVM applet scope. That is: the ability to run a set of applets in the same JVM, controlling which ones go together. The two real cases that I would need are "page" (all applets in the page scope on the same page instance share the same JVM) and "application" (all applets in different pages that declare to be in the same application share the same JVM).

The first is really to have different applets that can interact with each other, and it would actually go together with the ability to drag applets out. You can imagine two applets, one that is essentially a display of some data, and the other with the parameters for the display. You set your display the way you want, and when you are done you drag the display out and close the page.

The second would be used to share data cache. I have multiple pages that work on the same big dataset, and I would like different pages to simply use the same cache.

JVM scope would also automatically decrease the JVM startup time to 0 in certain, with no real effort. And it really allows me to create complex standalone widgets that can be assembled in pages through HTML, javascript or whatever javascript framework is in today.

Any chances?

Filter Blog

By date: