1 2 3 Previous Next

emcmanus

87 posts
I was looking over some old stuff, and found JDistill, a byte-code reduction program I wrote in 1998. Although it won't work unchanged on today's class files, and its copyright status is murky, I thought that the article I wrote might still have some interest. Here it is, with only minor edits. 

JDistill, a program to shrink Java packages
Éamonn McManus <emcmanus@opengroup.org>
3 April 1998

JDistill is a Java program that reduces the size of the class files that make up a Java package. It does this in several ways: 
  • It deletes information that is used for debugging but not actually needed for correct execution of the Java bytecode. This includes line-number information, for example.
  • It traces through all the executable code in the package that is reachable from outside the package in order to determine what is referenced. Anything that is never referenced is deleted. This can include fields and methods of Java classes, or even entire classes themselves.
  • It replaces the names of any methods and fields that have not been deleted and that are not visible outside the package with very short names. This has the side-effect of making the class file much harder to discompile.
  • It detects the frequently very bulky code generated by the current JDK javac compiler (versions up to 1.2beta2 at least) to initialize arrays, and replaces it with code that takes up much less space in the class file.
  • A command-line option tells JDistill that it is being given the whole Java program, i.e., no new classes will be added. In this case many more methods and fields can be deleted or renamed.
This document explains the rationale for JDistill, shows how to use it to distill (shrink) packages, and gives some results for the standard JDK packages in 1.1.5. The final section goes into detail about the implementation. 

Rationale

Java class files serve two distinct purposes. They containbytecode to be executed when the program using them is run; and they contain information used when other Java classes are compiled. For a Java program that is a product, this additional information for the compiler is not needed and can be deleted. 

Deleting the extra information makes the class files smaller, and also makes them much harder to decipher. Tools exist that can reconstitute almost exactly the Java source code that was used to produce a given class file; but if the names have been dramatically shortened (a form of obfuscation) this source code will be of little use.

When the Java compiler compiles a Java package, it does not know if additional classes will subsequently be added to the package. JDistill operates on the assumption that they will not. That means that anything within the package that is not referenced now will never be referenced and can be deleted.

You might imagine that anything JDistill can delete out of a package could also be deleted from the Java source code by a human user. This is not true for a number of reasons.

  • When you declare a constant in Java, like this: 
        static final int MAX_VALUE = 32767;
    
    the compiler must leave this field in the generated class file, so that other classes compiled later will be able to obtain its value. But no compiled class will ever refer to the field by name, because instead of generating a reference toSomeClass.MAX_VALUE the compiler will just copy the constant value 32767. JDistill will see that there are no references to MAX_VALUE in existing class files (because of this replacement) and will know that nothing outside the package can subsequently refer to MAX_VALUE(because it is not public), so it can delete it.
  • A product might exist in several configurations, selected at compile time. It is possible to eliminate code within methods when it belongs to a configuration that is not enabled, by doing something like: 
        if (Config.DEBUG) {
            // debug code...
        }
    
    But this technique cannot eliminate methods themselves or fields or even classes that are only used when Config.DEBUG is true. JDistill can.
  • Renaming methods and fields can save a small amount of space and can render class files very difficult to decipher. But it would be counterproductive to do the renaming in the Java source files!
  • As mentioned above, the Java compiler generates bulky code to initialize constant arrays - the same code as it generates for non-constant arrays, in fact. If you write: 
        static final int[] powers = {1, 5, 25, 125, 625, 3125};
    
    the compiler will generate code almost as if you had written: 
        static final int[] powers;
        static {
            powers = new int[6];
            powers[0] = 1;
            powers[1] = 5;
            ...
            powers[5] = 3125;
        }
    
    (The code is slightly more efficient than that because it can keep a reference to the powers array on the operand stack.) You can write code that takes up less space to initialize the array, by storing the initial values in a constant String and pulling them out in a loop, but it is unnecessary work to do so. JDistill does it for you automatically.

Using JDistill

2011 Éamonn says: you might want to skip to the section on technical details.

The command line for using JDistill looks like this:

java ORG.opengroup.jdistill.Main [-options] directory [...]

At its simplest, all of the .class files in the named directory are read, transformed to make them smaller, and written back over the originals. It is assumed that the class files form a package and that that package is closed, meaning that no new classes will be added to it after distilling, or that it will be redistilled after new classes are added.

Several directories can be named instead of just one; currently the effect is the same as if JDistill had been run on each directory individually. Files can also be named instead of directories, whereupon they are treated separately without the "closed" assumption.

Invoking JDistill without any arguments or with incorrect arguments produces a summary of the available options. The options are as follows. Each option can be abbreviated to any unambiguous prefix.

-output-replace
Followed by a pattern that says where output files should be written. The pattern should have two parts separated by=, and each part should contain exactly one% character. Each input file should match the pattern on the left of the =, with % being a wildcard that matches any sequence of characters. The name of the corresponding output file will be determined by the pattern on the right of the =, with the % being replaced by the same sequence of characters. Thus for example the option 
    -out /tmp/classes/%=/tmp/shrunkclasses/%
will cause a class file read from/tmp/classes/java/lang/String.class to be written to/tmp/shrunkclasses/java/lang/String.class

The % syntax is inspired by GNU make. The more obvious * character was avoided to prevent problems with Unix shell expansion.

If this option is not provided, distilled class files are written over the original undistilled ones. In that case classes that are found not to be referenced are not deleted; it would be more accurate to say that they are not copied when-output-replace is specified.

-option-file
Followed by the name of a file from which further options are to be read. Typically there will be -keep-* options in this file, kept up to date as needed. The syntax of options in the file is the same as on the command line except that they can be provided on several lines, and lines beginning with a# character are ignored.
-verbose
Print a line for each class file that is read to be distilled, each class that is loaded without being distilled (to provide inheritance information, for instance), and each class file that is written.
-no-warnings
Do not print messages about constructs found in the class files that might cause problems with the distillation. Currently the only warnings are associated with classes loaded dynamically usingClass.forName(String). If the argument is not a constant string, or if it is a string naming a class that cannot be found at distill time, a warning is issued.
-class-path
Use the class path immediately following this option instead of the standard class path when looking for classes that are not included in the set to be distilled but are referenced from it. Thus for instance JDistill will almost always want to loadjava.lang.Object to find out if any methods in that class are overridden by the classes being distilled; if those classes belong to a different system than the one where JDistill is being run then it would not be right to load the samejava.lang.Object that is being used for the JDistill code itself.
-keep-class
Do not delete the class following this option even if it is apparently unreferenced. The Java syntax for a class is used, not the name of the class file. So 
    -keep-class java.io.Thing
might cause the file /unshrunk/java/io/Thing.class to be copied to /shrunk/java/io/Thing.class even ifjava.io.Thing is not a public class and there are no references to it within java.io

This option does not affect deletion or renaming of methods and fields within the named class.

-keep-field
Do not delete or rename the field following this option even if it would apparently be all right to do so. This is especially useful for fields that are accessed only from native code. The field is given in its fully-qualified form,i.e., the full name of its class followed by the name of the field, such as java.awt.Font.pData

If a field is accessed from native code but not using JNI, all preceding fields should also be kept, or javah should be run on the distilled classes rather than the originals.

-keep-method
Do not delete or rename the method following this option even if it would apparently be all right to do so. There is no way to differentiate among overloaded methods with the same name using this option: all methods with the given name will be preserved. Again this is useful for methods that are accessed only from native code. The method is given in its fully-qualified form but without parameters, for instance java.io.File.exists.
-keep-constructor
Do not delete any constructors for the class following this option even if they are apparently unreferenced. The class name is spelt out as for -keep-class.
-no-main
Do not consider methods with the signature 
    public static void main(String[])
as entry points, unless they are in public classes, in which case they would be entry points anyway. This option allows main methods introduced for debugging to be automatically deleted from production code.
-closed-program
Assume that no classes other than those being distilled will ever be loaded. This means that anything that is not referenced within the distilled classes is not referenced at all and can be deleted.
-tree
Treat each directory argument as the root of a directory hierarchy and shrink every file in that hierarchy whose name matches *.class.
-make-directories
Make intermediate directories as needed when creating output class files.
-disassemble
Print a disassembly of the structure of each class file as it is read, including a symbolic representation of the bytecode of each method. The information printed is similar to that printed by the standard javap tool, but more detailed.
-show-deletions
Show fields, methods, and classes that are deleted because they are unreferenced. The information displayed can be very instructive. It may reveal parts of a program that should have been deleted during some earlier modification; it may reveal faulty program logic that means some code is never reached; or it may indicate things that are only referenced from native code and so need to be protected using the -keep options. If the program being distilled was compiled with -O, short methods will be inlined where they are called, so if they are not visible outside their package JDistill will legitimately delete them.
-debug-deletions
Also show constants from the constant pool that are deleted. These will typically include the names of deleted fields and methods, and also constant strings or integers that were only referred to from within deleted methods. If fields and methods are renamed, the old names will normally be deleted too.
-show-renames
Show fields and methods that are renamed to shorter names. You may want to keep this information in case you ever have to debug a stack trace through methods that are now called A orB.
-debug-renames
Show the analysis that is done to determine what methods can be renamed. The existence of method overriding and hiding (Java Language Specification §8.4.6) means that this analysis is complex, as described later. With this option, if a method is not renamed then JDistill explains why, and it also shows sets of methods that were found to be required to have the same name.
-no-renames
Do not rename any methods or fields.
-show-array-shrink
Show all array initializations that were considered for shortening, with the savings obtained for those that were indeed shortened and the reason why not for those that were not.
-debug-array-shrink
Show additional information about array shortening. This information is principally of interest for debugging JDistill itself.
-no-array-shrink
Do not shorten any array initializations.
-show-references
Show the results of the JDistill reference analysis before anything is deleted. Essentially, the list of things that are unreferenced is the same as the list of things that will be deleted, so the principal interest of this option is to see how many references there were to certain things. Currently, only constants in the constant pool are counted in this way; for other entities the information is just whether they are referenced or not.
-trace
Print a detailed trace of the flow of execution as references from bytecode are determined. This option produces a great deal of output and is mainly of use for debugging JDistill.

Measurements

These measurements were made on an HP/UX machine running version 1.1.5 of the JDK. All of the standard classes of the JDK were distilled using all available shrinking. A few -keep-*options had to be provided to prevent incorrect deletion of fields and methods used only by native code. With these options in place, the distilled classes get the same results in the JCK as the original classes. 

The overall reduction is a spectacular 32%, but a great deal of that is attributable to two factors: shortened array initializations, and classes compiled without optimization. Some classes, for instance java.lang.Character and almost all of the sun.io package, contain big arrays which the javac compiler turns into huge bytecode. A later section will mention some problems that the shortened array initializations may cause. The standard packages distributed as part of the JDK ought to be compiled with optimization (-O flag), which as a side-effect leaves out line-number information, but some of them are not, which makessun.io even larger than it would be.

The standard packages measured here are not typical of Java applications, in that they export many individual methods but there is less cooperation between classes in individual packages. The fact that most of the package information is exported greatly limits what JDistill can do, since publicly-visible fields and methods cannot be deleted or renamed.

As another example, the JDistill program itself, compiled with optimization (-O flag), was distilled. Here the saving was 19%, a figure probably more typical of applications. There were no large arrays to be shrunk so the figures with and without that optimization were essentially the same.

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
PackageOld sizeNew sizeSaved
java.lang1600309779162239 (38.9%)
java.lang.reflect1150410990514 (4.5%)
java.util64858601094749 (7.3%)
java.util.zip37163316145549 (14.9%)
java.io1117831033858398 (7.5%)
java.math21682176224060 (18.7%)
java.text229856102778127078 (55.3%)
java.text.resources37654432344753097 (14.1%)
java.net60327524677860 (13%)
java.security35759291636596 (18.4%)
java.security.acl32332358875 (27.1%)
java.security.interfaces15611015546 (35%)
java.awt25234220843443908 (17.4%)
java.awt.peer976067393021 (31%)
java.awt.image30732241326600 (21.5%)
java.awt.datatransfer616049791181 (19.2%)
java.awt.event31704264295275 (16.6%)
java.applet39313124807 (20.5%)
java.sql36624321364488 (12.3%)
java.rmi16475126013874 (23.5%)
java.rmi.registry30232489534 (17.7%)
java.rmi.dgc25932010583 (22.5%)
java.rmi.server24504201124392 (17.9%)
java.beans541404379010350 (19.1%)
sun.tools.java1508331457795054 (3.4%)
sun.tools.javac67557662301327 (2%)
sun.tools.debug1154999095324546 (21.3%)
sun.tools.asm43774413762398 (5.5%)
sun.tools.tree31094229318917753 (5.7%)
sun.tools.jar31606265365070 (16%)
sun.tools.util982182731548 (15.8%)
sun.tools.javap21708176654043 (18.6%)
sun.tools.javadoc496131524534368 (69.3%)
sun.tools.ttydebug30014242415773 (19.2%)
sun.tools.native2ascii50774228849 (16.7%)
sun.tools.serialver69285969959 (13.8%)
sun.misc541844004814136 (26.1%)
sun.io535026431408142209450 (41.3%)
sun.net13079105152564 (19.6%)
sun.net.ftp859972011398 (16.3%)
sun.net.nntp688056771203 (17.5%)
sun.net.smtp453135291002 (22.1%)
sun.net.www27227215375690 (20.9%)
sun.net.www.content.text14411062379 (26.3%)
sun.net.www.content.image19261485441 (22.9%)
sun.net.www.protocol.file60015160841 (14%)
sun.net.www.protocol.http16302130413261 (20%)
sun.net.www.protocol.doc38713400471 (12.2%)
sun.net.www.protocol.netdoc14841261223 (15%)
sun.net.www.protocol.verbatim34682897571 (16.5%)
sun.net.www.protocol.ftp928580841201 (12.9%)
sun.net.www.protocol.gopher856970851484 (17.3%)
sun.net.www.protocol.mailto45043852652 (14.5%)
sun.net.www.protocol.appletresource29162506410 (14.1%)
sun.net.www.protocol.systemresource749962911208 (16.1%)
sun.net.www.http15538131322406 (15.5%)
sun.net.www.httpd65895685904 (13.7%)
sun.audio1883096009230 (49%)
sun.awt365192644410075 (27.6%)
sun.awt.motif17346813892934539 (19.9%)
sun.awt.image591734587713296 (22.5%)
sun.awt.tiny14766211750030162 (20.4%)
sun.applet923177504517272 (18.7%)
sun.applet.resources14299129831316 (9.2%)
sun.security.acl13307105662741 (20.6%)
sun.security.util22505180724433 (19.7%)
sun.security.x50944805374447361 (16.4%)
sun.security.pkcs25451220403411 (13.4%)
sun.security.provider725785799314585 (20.1%)
sun.rmi.registry12711107331978 (15.6%)
sun.rmi.server19483166662817 (14.5%)
sun.rmi.transport565854591110674 (18.9%)
sun.rmi.transport.proxy40943328518092 (19.8%)
sun.rmi.transport.tcp45573371818392 (18.4%)
sun.rmi.rmic45814382697545 (16.5%)
sun.beans.editors17552150002552 (14.5%)
sun.beans.infos14771235242 (16.4%)
sun.jdbc.odbc15215712568826469 (17.4%)
sunw.util537266271 (50.5%)
sunw.io231100131 (56.7%)
Total (80 packages)909779461600532937741 (32.3%)
Average1137227700136722 (32.3%)

Technical details

JDistill does three essentially independent tasks when working on a package: 
  1. It traces the bytecode to find out what is referenced and deletes whatever is not;
  2. It shortens the names of fields and methods whose existing names are not visible outside the package;
  3. It shortens array initializations.

Finding references

The algorithm for tracing all of the bytecode in a package that can ever be reached is conceptually simple: start from the public entry points and trace through instructions; every time you see an instruction that references a field, mark that field as referenced; every time you see an instruction that invokes a method, mark that method as referenced and trace it too. 

The public entry points of a package are determined as follows:

  • Every public method in a public class is an entry point because it can be called directly by Java code outside the package.
  • Every protected method in a publicclass is an entry point because it can be called by a subclass of the class outside the package (this is not true if the class isfinal but we don't currently check that).
  • Every public or protected method in a class that is not public but that is subclassed by apublic class in the same package is an entry point because it is visible through the subclass.
  • A method that looks like 
        public static void main(String[] args)
    
    is an entry point because a Java program can start there, even if it is in a class that is not public. Since such a method often contains test code for the containing class that is not supposed to be included in the final product, JDistill has a command-line option -no-main to tell it not to consider this kind of entry point unless it is covered by one of the other rules.
  • Every method, even a private one, in a class that contains native method declarations is an entry point. Native code can typically access any field or method of any class regardless of Java-language visibility. We can't know what fields or methods are accessed in this way, and we don't want to keep all fields and methods just in case they might be accessed by native code, so we assume that classes with native methods are the only ones whose methods or fields are accessed in this way. When this assumption is wrong, it can be corrected as explained in the next paragraph.
  • Every method that was explicitly named by a -keep-method or -keep-constructor command-line option is an entry point. You can use this to preserve methods called only from native code and not included in the heuristic described just above.
A constructor is considered to be a method like any other for the purposes of these rules. 

If the -closed-program option is given, only the last two rules are applied, but then if a class containing native methods is determined to be referenced, all of its methods are considered to be entry points.

While we are looking for public entry points, we also look forvisible fields, that is fields that can be accessed directly or indirectly from outside the package. The rules here are the following:

  • Every public or protected field in apublic class or a class that has a publicsubclass in the package is visible. This is essentially the same rule as for methods.
  • Every field in a class that has native methods is visible, for the same reasons as explained for methods above.
  • Every field in a class that implementsjava.io.Serializable (directly or by inheritance) is visible unless it is static or transient, because changing the field in any way would break compatibility with previously-serialized objects of this class.
  • If a class that implements java.io.Serializablehas a field called serialVersionUID then that field also participates in serialization even if it isstatic (in fact it always isstatic).
  • Every field that was explicitly named by a -keep-field command-line option is visible. Again this can be used to prevent the deletion of fields that are only referenced from native code and which are in classes that do not themselves contain native methods.

Here again, the public rule does not come into play when -closed-program is given, and the other rules except the last are only applied to classes that are found to be referenced.

Tracing bytecode

We examine the instructions of every method that is an entry point as explained above, and every method that is called by such a method, and so on recursively. The following sorts of instructions have to be noted: 
  • A checkcast, instanceof,anewarray, getstatic,putstatic, or invokestatic instruction causes the class it references to be visited statically, as explained below.
  • A new, getfield,putfield, invokevirtual,invokespecial, or invokeinterfaceinstruction causes the class it references to be visited dynamically, as explained below.
  • The various invoke* instructions also cause the methods they invoke to be traced if they have not already been.
  • If an invokestatic instruction turns out to invoke the java.lang.Class.forName(String) method then we have detected a dynamically-loaded class. If the class being loaded is known (the argument to forName is a constantString) then it is visited statically. Furthermore, its no-arg constructor is traced, because it might be invoked through the newInstance()method in java.lang.Class. (An earlier version only traced the no-arg constructor if a call tonewInstance() was seen within the package, but this was wrong because the Class reference could easily be passed outside the package and newInstance() invoked there.) 

    If the argument to Class.forName is not a constant String, a warning is issued, because JDistill's assumptions about reachability might be violated.

    You will need to use the various -keep-*command-line options if the package being distilled contains classes, or fields and methods within classes, that are only referenced through dynamic loading and reflection.

  • The field-access instructions (getstatic etc) also cause the fields they name to be marked as referenced.
  • The ldc, ldc_w, andldc2_w instructions cause the constants they reference to be marked as referenced.

A class is visited staticallywhen we know that it will be initialized (in the sense of the Java Language Specification, §12.4.1). In this case the class is marked as referenced if it was not already, and if it has a static initializer method (<clinit>) that method's bytecode will be traced.

A class is visited dynamically when we know that there must be at least one instance of it for the program to be correct. In this case we have further information about the class:

  • If it implements the java.io.Serializableinterface and contains one or both of the methods 
        void readObject(java.io.ObjectInputStream in)
        void writeObject(java.io.ObjectOutputStream out)
    
    then those methods are potentially referenced and must be traced.
  • If it overrides methods in an ancestor class that are referenced then the overriding methods must also be considered to be referenced. This is further explained in the next section.
We do not explicitly visit a parent class dynamically when one if its children is visited dynamically, but if the child is really instantiated then we should end up visiting at least one of its constructors, and this constructor will start with an invocation of one of the parent's constructors, which will provoke a dynamic visit. 

Handling overriding methods

Once all of the public entry points and all of the methods they invoke have been traced recursively, a further important way to reach methods must be considered. A method that is never itself directly invoked, but that overrides a method in an ancestor class that is, must be traced, provided that the class it is contained in is actually instantiated. 

Finding indirectly-referenced overriding methods is not completely straightforward. Here is how we tackle it. We loop over all the classes in the package looking for non-staticmethods that are currently not referenced. For each such method, we recursively scan the ancestors (superclasses and superinterfaces) looking for methods with the same name and parameters, which are therefore overridden; we thus construct a set containing the first overridden method found on each path through the ancestor graph. If we find a referenced overridden method, then we mark the overriding method as referenced too and trace it. After we have scanned all the classes in the package and if we found any overriding methods, we repeat the whole process, until no further overriding methods are found.

In this way we are sure to propagate the referencedness all the way down a possibly long chain of overrides, and we are sure to pick up new chains of overrides that start at a method that is itself only referenced in an overriding method.

A more straightforward approach would be possible if we linked classes to all their subclasses and then methods to all their overriders, but building that information would not be much simpler than what we do here.

In the diagram, Tracing indirect method references   the method "0" is the only one explicitly referenced in the program. The methods labeled "1" are in subclasses that override it, and the methods labeled "2" override those. The first time we loop through all the classes we will find that the methods "1" are unreferenced but override a referenced method, so we will mark them referenced and trace them. Then either later in that scan through the classes or on the next scan, depending on the order in which we visit the classes, we will do the same thing for the methods labeled "2".

Renaming fields and methods

JDistill renames fields and methods in the package being shrunk so that they are shorter. Shortening the name of a field or method reduces space not just in the class where it is defined, but in every class that references it. If a member (a field or method) is referenced in a class there will be a single constant naming the class containing the member and the name of the member itself. This constant will be shared between all the references from the same class, so renaming will have most effect when there are many small classes rather than a few big ones. 

Renaming fields

Renaming fields is straightforward. Any field that is not visible can be renamed. JDistill renames fields in a given class starting at A and counting up toZ, then a to z, then_, then longer names starting with these characters but also including the digits 0 to 9. This means that all new names are valid Java identifiers. The specification of class files does not actually require this, but it seems best to avoid possible problems with incorrect implementations that do. 

The only other constraint on chosen field names is that they must not conflict with existing field names in the same class, specifically those that are visible. This is easily done using a hash table that is needed anyway in order to follow field references to the corresponding field.

Renaming methods

Renaming methods is much harder because of overriding. The field names in a class are independent of those in its ancestors and descendants; class files specify exactly which class's field is being referenced. The names of methods in a class on the other hand are intimately tied to the names of methods in its ancestors and descendants. If a method has the same name as a method in an ancestor, then both names must still be the same after renaming. Contrariwise, if the name of a method is different from the name of another method in the same class or in any ancestor or descendant, then it must remain different after renaming. 

Overloaded methods

Overloading also complicates method renaming slightly. Where we talked about names being different in the previous paragraph, we should really have talked about signatures. That is, the combination of name and parameter types must be the same or different according as there is an overriding relationship between two methods. 

Methods that must be renamed together

Before we can rename any methods, we must group together all methods that have an overriding relationship so that they will all continue to have the same name after renaming. This is straightforward. With every method in every class, we associate a set of other methods. This is usually a set of methods that must have the same name; but if a method must not be renamed, either because it is an entry point or because it has an overriding relationship with an entry point, then it will belong to a special set called the fixed-name set

To construct the sets of methods we proceed as follows. We visit the classes in ancestor-first order. For each method in each class we visit, we first check if it can be renamed at all. If not, it is added to the fixed-name set. If so, it is added to a newly-created set. Then, for every method that the method we are visiting overrides, we merge together the sets for the overriding and overridden methods. In this way, if any method in the group of methods that must have the same name as the current method turns out not to be renameable, all of the methods in the group will end up in the fixed-name set. If all methods in the group can be renamed then they will all end up in the same set.

Finding new names

We can consider finding new method names as a graph-colouring problem. In the graph, the nodes are methods and an edge joins two nodes if the corresponding methods must not have the same name, because they have the same parameters and either they are in the same class or one of them is in an ancestor class of the other. (Methods for which this is true and that already have the same name already have an overriding relationship and can be coalesced into the same node.) The graph is actually a collection of disjoint graphs, one for every possible parameter signature. These graphs can be considered to be coloured when we start, where the colours are method names and no two nodes connected by an edge can have the same colour. What we want to do is to recolour those nodes where this is allowed so as to use more efficient colours (shorter names). 

Building this graph would be expensive and unnecessarily complicated. Instead we use some heuristics to achieve a result that is theoretically non-optimal but will in fact almost always be as good as optimal (not use longer names than necessary). The heuristics are:

  1. We never recolour with a colour that was already in the graph before we started. This means that we only have to check the colours of adjacent nodes we have already coloured. (Translation: we build a big hashtable of every signature of every method and avoiding renaming a method to something that gives it the same signature as one of these.)
  2. We always visit the methods of a class before those of any of its descendants. This means that we can find the adjacent nodes already coloured by searching upwards in the inheritance dag. There are currently no downward links in it.

The reason this is nearly always as good as real colouring is that the short names we will be using as new colours hardly ever exist already as method names, so our first heuristic rarely makes any difference. The exception is when some but not all of the classes being distilled were generated by programs that make short names (such as JDistill itself). In that case we will typically find that we can't make those names any shorter than they were already, and we can shorten names in the non-generated classes less than we would be able to with an optimal algorithm.

Shortening array initializations

As we mentioned earlier, the standard JDK Java compiler from Sun takes source code like: 
    static final int[] powers = {1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024};
and generates bytecode almost as if you had written: 
    static final int[] powers;
    static {
    powers = new int[10];
    powers[0] = 1;
    powers[1] = 2;
    ...
    powers[10] = 1024;
    }
More exactly, the code it generates in the static initializer<clinit> looks like this (the comments show what is on the stack after each instruction): 
    bipush 11       // 11  (size of array)
    newarray int    // int[]
    dup         // int[], int[]
    iconst_0        // int[], int[], 0
    iconst_1        // int[], int[], 0, 1
    iastore     // int[]  (now array[0] == 1)
    dup
    iconst_1
    iconst_2
    iastore
    ...
    dup         // int[], int[]
    bipush 10       // int[], int[], 10
    sipush 1024     // int[], int[], 10, 1024
    iastore     // int[]  (now array[10] == 1024)
    putstatic powers    // (empty; now powers == the initialized array)
    return
In cases like this where the range of values is relatively small, we can instead initialize the array as if the code had been this: 
    static final int[] powers;
    static {
    powers = new int[11];
    final String powersString =
        "\u0001\u0002\u0004\u0008\u0010\u0020\u0040\u0080\u0100\u0200\u0400";
    for (int i = 0; i < 11; i++)
        powers[i] = powersString.charAt(i);
    }
That is, we construct a String where each character is a value to be put in the array, and extract those values when the array is created. This operation is slower than the original unrolled bytecode, which is why we only do this operation in static initializers; since they are only executed once presumably the small extra time will make no difference. (It would be useful to have an option to perform this operation everywhere big arrays are seen, notably in constructors, despite the small speed penalty; but currently there is no such option.) 

The bytecode we construct to unpack the values from the String into the array looks like this:

    bipush 11       // 11  (size of array)
    newarray int    // int[]
    bipush 10       // int[], 10 (which is the array index i)
  loop:
    dup2        // int[], i, int[], i
    dup         // int[], i, int[], i, i
    ldc "\u0001..." // int[], i, int[], i, i, "\u0001..."
    swap        // int[], i, int[], i, "\u0001...", i
    invokevirtual String.charAt(int)
            // int[], i, int[], i, <character at i>
    iastore     // int[], i  (now array[i] has been set)
    iconst_1        // int[], i, 1
    isub        // int[], i-1  (new value of i for next iteration)
    dup         // int[], i, i
    ifge loop       // int[], i
    pop         // int[]
    putstatic powers    // (empty; now powers == the initialized array)
    return
We do all the operations on the stack rather than using local variables, which would yield more straightforward bytecode, because we don't want to have to allocate a local variable or try to determine if one is free at the point where we are inserting this code. Similarly, we add a constant to the currently-declared maximum stack depth rather than computing what its real value is after the introduction of the new code, because we have no need to know anything about the stack usage of instructions anywhere else in the distiller. 

If some of the array values are negative or greater than 65535 then they will not fit in the char values of aString. In this case, provided that the spanof values is not more than 65535, we can encode in the same way but add a (possibly negative) constant to each value.

String constants are encoded in class files usingUTF8 encoding, which has the property that a character cwill be encoded using:

                  
1 byte,if 0 < c < 128;
2 bytes,if c = 0 or 127 < c < 2048;
3 bytes,otherwise (2047 < c < 65536)
So in fact we try to determine if adding a constant to each value in the encoded string will make for a string that takes up less space in the class file, even if the span of values is such that this is not strictly necessary. There are only two possible values that we may want to add: either the smallest value in the array (so that values near that smallest value are encoded in the efficient 1 - 128 range), or one less than the smallest value in the array (for the same reason, but additionally the smallest value itself is encoded as 1 rather than the less efficient 0). These are the only two possibilities because we can't add a number biggerthan the smallest value in the array (since characters in theString cannot be negative), and the fact that the cost function does not decrease except at 1 means that there is no point in adding a number smaller than the smallest value in the array minus 1. We try both possibilities mentioned to see if either of them makes the encoded UTF8 sufficiently smaller to compensate for the extra instructions needed to add the constant. 

After computing the optimal String encoding for the array, we estimate how much space it will take up in the class file, taking into account the instructions in the loop and the fact that it may be necessary to add a reference tojava.lang.String.charAt(int). If it looks as if the rewritten code will be more expensive than the original, then no change is made. The calculations here are not completely accurate so sometimes we may fail to save space when we could have, but the loss is small, typically a dozen bytes or so, and at worst roughly 50.

Why you might not want to shrink arrays

The Java Language Specification guarantees (http://www.javasoft.com/docs/books/jls/html/3.doc.html#101083

 Josh Bloch's Effective Java popularized the Builder Pattern as a more palatable way of constructing objects than constructors or factory methods when there are potentially many constructor parameters. The formulation in Effective Java makes for a particularly readable construction, like this:

new Rectangle.Builder().height(250).width(300).color(PINK).build();

The advantage over a constructor invocation like new Rectangle(250, 300, PINK); here is that you don't have to guess whether 250 is the height or the width. More generally, if the constructor allowed you to specify many other parameters — such as position, opacity, transforms, effects, and so on — it would quickly get very messy. The builder pattern stays clean.

But one question that arises is: how does it work in the presence of inheritance? For example, suppose you have an abstractShape class that represents an arbitrary graphical shape, with a set of properties that are common to all shapes, such as opacity and transforms. And suppose you have a number of concrete subclasses such as Rectangle,Circle, Path and so on, each with its own properties, like Rectangle's height and width.

As a reminder, here's what the builder pattern looks like in the absence of subclassing:

public class Rectangle {
    private final double opacity;
    private final double height;
    ...

    public static class Builder {
        private double opacity;
        private double height;
        ...

        public Builder opacity(double opacity) {
            this.opacity = opacity;
            return this;
        }

        public Builder height(double height) {
            this.height = height;
            return this;
        }
        ...

        public Rectangle build() {
            return new Rectangle(this);
        }
    }

    private Rectangle(Builder builder) {
        this.opacity = builder.opacity;
        this.height = builder.height;
        ...
    }
}

Subclassing

Now what happens if we want to introduce a Shapesuperclass, and move some of the properties ofRectangle into it? Let's just concentrate on opacity and height. We'll move opacity into Shape (all shapes have opacity) and leave height in Rectangle (a circle for example is defined by its radius rather than its height).

Here's a first attempt. Obviously we are no longer going to be able to keep our constructors private, at least not inShape, since a subclass needs to be able to invoke its superclass's constructor. So we'll make it protected, and we get this:

// First attempt at Shape/Rectangle separation.  This does not work!
public class Shape {
    private final double opacity;

    public static class Builder {
        private double opacity;

        public Builder opacity(double opacity) {
            this.opacity = opacity;
            return this;
        }

        public Shape build() {
            return new Shape(this);
        }
    }

    protected Shape(Builder builder) {
        this.opacity = builder.opacity;
    }
}

public class Rectangle extends Shape {
    private final double height;

    public static class Builder extends Shape.Builder {
        private double height;

        public Builder height(double height) {
            this.height = height;
            return this;
        }

        @Override
        public Rectangle build() {
            return new Rectangle(this);
        }
    }

    protected Rectangle(Builder builder) {
        super(builder);
        this.height = height;
    }
}

That looks pretty simple. Rectangle.Builder extendsShape.Builder, so it inherits the opacitymethod, and adds its own height. It overrides thebuild method to return its ownRectangle(Builder) constructor. That constructor passes its Builder to the superclass, soShape can set the opacity of the new object, andRectangle sets the height. So what's the problem?

Say we write:

Rectangle r = new Rectangle.Builder().opacity(0.5).height(250).build();

It doesn't compile. The reason why is slightly hidden by the reuse of the name Builder in bothShape.Builder and Rectangle.Builder.Rectangle.Builder inherits its opacitymethod from Shape.Builder. That method is declared to return Shape.Builder. But Shape.Builderdoes not have a height method. Even though the actualRectangle.Builder object that we are using does have aheight method, the compiler is using the declared type of opacity(0.5), which as we just saw isShape.Builder.

Suppose we required our callers to put the methods in order from subclass to superclass (which would be a very unpleasant requirement). So here the caller would have to write:

Rectangle r = new Rectangle.Builder().height(250).opacity(0.5).build();

But that still doesn't help. opacity(0.5) still returns Shape.Builder, so therefore thebuild() at the end isShape.Builder.build(), which returns aShape not a Rectangle.

A nasty solution

One way out is for Rectangle.Builder toredeclare opacity:

public class Rectangle extends Shape {
...
    public static class Builder extends Shape.Builder {
        ...
        @Override
        public Builder opacity(double opacity) {
            super.opacity(opacity);
            return this;
        }
        ...

That does fix our problem. But it's very nasty. It means that every time you add a property to Shape you have to visit all its subclasses, and all its subclasses' subclasses, and so on, to add a new color(c) or whatever method to all their Builder classes. It's hardly worth using inheritance at all if you end up doing things like that.

A better solution

There is a better solution, which allows us to do what we want without having to pollute subclasses with their superclass's properties. The main drawback is that it uses mindbending generics declarations similar to Enum<E extends Enum<E>>. But it works. Here's the code:

public class Shape {
    private final double opacity;

    protected static abstract class Init<T extends Init<T>> {
        private double opacity;

        protected abstract T self();

        public T opacity(double opacity) {
            this.opacity = opacity;
            return self();
        }

        public Shape build() {
            return new Shape(this);
        }
    }

    public static class Builder extends Init<Builder> {
        @Override
        protected Builder self() {
            return this;
        }
    }

    protected Shape(Init<?> init) {
        this.opacity = init.opacity;
    }
}

public class Rectangle extends Shape {
    private final double height;

    protected static abstract class Init<T extends Init<T>> extends Shape.Init<T> {
        private double height;

        public T height(double height) {
            this.height = height;
            return self();
        }

        public Rectangle build() {
            return new Rectangle(this);
        }
    }

    public static class Builder extends Init<Builder> {
        @Override
        protected Builder self() {
            return this;
        }
    }

    protected Rectangle(Init<?> init) {
        super(init);
        this.height = init.height;
    }
}

The idea is that instead of hardwiring opacity() to return the type of the Builder that defines it, we introduce a type parameter T and we returnT. The self-referential definition Init<T extends Init<T>> is what allows us to make the type of the inherited opacity() inRectangle.Builder be Rectangle.Builderrather than Shape.Builder.

We can no longer simply return this fromopacity(), since at the point where it is defined,this is an Init, not a T. So instead of this, we return self(), and we arrange for self() to be overridden so that it returns the appropriate this. This is pure ceremony to keep the compiler happy: all of the Builder classes have identical definitions of self(). (This is what Angelika Langer calls the getThis() trick, citing Maurice Naftalin and Philip Wadler for the name and Heinz Kabutz for the first publication.)

Why do we need to split our previous Builder class into separate Init and Builder classes? Because we still want the caller to be able to write:

Rectangle r = new Rectangle.Builder().opacity(0.5).height(250).build();

If Builder were the class with the self-referential type (Builder<T extends Builder<T>>) thennew Rectangle.Builder() would be missing a type argument. And even if we were prepared to bother every caller with having to supply such an argument, what would it be? We cannot write new Builder<Builder>() because then the second Builder is missing a type argument! That's why we need one class that has the self-referential <T> parameter (so opacity() can return T) and another one that doesn't (so callers can make instances of it).

Better still: static factory instead of constructor

It isn't very nice that users can see both theBuilder and Init classes. Is there a way to hide one of them?

The answer is yes, if we say that the way to build a rectangle is this:

Rectangle r = Rectangle.builder().opacity(0.5).height(250).build();

That is, the caller gets a builder from the static methodRectangle.builder() rather than with new Rectangle.Builder(). Here's what the modified code looks like:

public class Shape {
    private final double opacity;

    public static abstract class Builder<T extends Builder<T>> {
        private double opacity;

        protected abstract T self();

        public T opacity(double opacity) {
            this.opacity = opacity;
            return self();
        }

        public Shape build() {
            return new Shape(this);
        }
    }

    private static class Builder2 extends Builder<Builder2> {
        @Override
        protected Builder2 self() {
            return this;
        }
    }

    public static Builder<?> builder() {
        return new Builder2();
    }

    protected Shape(Builder<?> builder) {
        this.opacity = builder.opacity;
    }
}

public class Rectangle extends Shape {
    private final double height;

    public static abstract class Builder<T extends Builder<T>> extends Shape.Builder<T> {
        private double height;

        public T height(double height) {
            this.height = height;
            return self();
        }

        public Rectangle build() {
            return new Rectangle(this);
        }
    }

    private static class Builder2 extends Builder<Builder2> {
        @Override
        protected Builder2 self() {
            return this;
        }
    }

    public static Builder<?> builder() {
        return new Builder2();
    }

    protected Rectangle(Builder<?> builder) {
        super(builder);
        this.height = builder.height;
    }
}

 

This is my preferred version.

A shorter, smellier variant

There is a way to avoid having to define a second builder class for every class you want to construct, but it is a bit nasty. I won't spell it out like the other cases, but here's the gist:

public class Shape {
    ...
    public static class Builder<T extends Builder<T>> {
        ...
        @SuppressWarnings("unchecked")  // Smell 1
        protected T self() {
            return (T) this;            // Unchecked cast!
        }
        ...
    }

    @SuppressWarnings("rawtypes")       // Smell 2
    public static Builder<?> builder() {
        return new Builder();           // Raw type - no type argument!
    }
    ...
}

public class Rectangle extends Shape {
    ...
    public static class Builder<T extends Builder<T>> extends Shape.Builder<T> {
        ... no need to define self() ...
    }

    @SuppressWarnings("rawtypes")
    public static Builder<?> builder() {
        return new Builder();
    }
    ...
}

Some notes

I showed the opacity and height fields as final because fields should be final whenever possible, and because it demonstrates that this pattern works correctly with final fields. But of course it works with non-final fields too.

If the Shape class were abstract, we would omit its builder's build() method, and the staticbuilder() method in the variant that has one, but otherwise everything would be the same.

If you have several hierarchies of classes using this pattern, you might want to extract the self() method into a separate interface or abstract class, such as
public interface Self<T extends Self<T>>.

You can have required constructor parameters by putting those parameters in each Builder's constructor, and in the static factory method in that variant. Of course then you lose the ability to name those parameters.

You can have default values for parameters by providing initializers in the builder (not in the class itself!), for example:

public class Shape {
    ...
    public static abstract class Builder<T extends Builder<T>> {
        private double opacity = 1.0;
    ...

Can we do better?

In my preferred solution, every class in the hierarchy has to duplicate the code in blue. Is there a way to reduce this boilerplate code without resorting to smelly hacks? I have not found one, but I'm open to suggestions!

 I feel a bit guilty saying bad things about Apache Ant. It's free, it's available everywhere, and a lot of volunteers have put a lot of work into making it what it is. You can very quickly and easily make a build file for simple Java projects. But. It seems to have been more accreted than designed, and if you try to use the core system to accomplish anything outside the most basic stuff you can look forward to hours staring alternately at those horribly illegible XML files and that huge and frustratingly disorganized online manual. So I was pleasantly surprised recently to realize that you can use JavaScript directly inside your build.xml files to avoid having to fight with Ant's enormous but haphazard collection of concepts and tags.

It could well be that I am the last person in the world to learn this and that everyone else has been using JavaScript to work around Ant's deficiencies since 2001. Perhaps not, though, because it used to be the case that if you wanted to use a scripting language with Ant then you needed to go off and get the relevant jars and put them where Ant could find them. What's changed is that you can now pretty much assume that you will have at least JDK 6 everywhere (it came out in December 2006), which means you automatically have a JavaScript engine. If Ant is running on JDK 6, it can find and use this engine without any configuration.

This is not the only solution when you run up against the limits of what you can straightforwardly do with Ant. Several other approaches exist. One is to use XSLT to generate your build.xml files. I think that should only ever be the very last of last resorts. Imagine someone coming upon your system for the first time and trying to understand how it works. Where do they start? How do they find out what will actually happen when the system is built? Where do they start if they need to change something?  If there's one thing I've learnt over the years it's that an extra degree of "meta-ness" dramatically increases the difficulty of learning and using a system. Certainly it can also make it more powerful, but in this case there are less meta-ish alternatives that are no less powerful.

A second approach is to write your own Ant tasks using Java code. Aside from the constant nagging question "why I am having to do this?" as you do so, the result is again a build system that is hard to understand. Part of your build file will be devoted to compiling the custom tasks; and of course the source of those tasks, showing what they actually do, is off in some other files somewhere.

A third approach is to use ant-contrib to add to Ant things that should already be there, in particular the <if> and especially <for> constructs. (I'm sure I'm not alone in finding it completely inexplicable that Ant still does not come with these tasks as standard.) The right way to use ant-contrib isnot to plonk it in your Ant install directory, because then you're using a non-standard Ant and your projects will not work if you transfer them to another machine. Instead, you should copy ant-contrib-1.0b3.jar (or whatever the version is) into the lib directory of your project, and explicitly activate it in your build.xml with a line like this:

<taskdef resource="net/sf/antcontrib/antlib.xml" classpath="${basedir}/lib/ant-contrib-1.0b3.jar" />

Using ant-contrib is actually a fine solution to many of the most annoying Ant deficiencies, and you may prefer it to the final approach, which is my subject here: using JavaScript.

Using JavaScript in Ant tasks

Here is a complete Ant project that says hello using JavaScript:

    <?xml version="1.0" encoding="UTF-8"?>
    <project name="hello" default="default">
        <target name="default">
            <script language="javascript">println("hello, world")</script>
        </target>
    </project>
   

Now, the first thing to say is that I am living dangerously here. If ever my script contained a character with meaning to XML (basically < and &) then the whole Ant build would fail with a mysterious parse error. So to avoid that danger, I need to add some extra XML gobbledygook (you can tell I love XML very very much):

        <script language="javascript"><![CDATA[
            println("hello, world")
        ]]></script>
   

Obviously I could have achieved as much with the <echo> task, so let's look at a real problem.

Checking classpath dependencies

Probably Zig and Zag and Dustin the first time I realized that I didn't like Ant very much was when I tried to solve the following problem. I was working on a big application that had been split into several Ant projects. One of these projects (call it Big) depended on a number of other projects (let's say Zig and Zag). We were using NetBeans, which is quite good at managing inter-project dependencies. It automatically set things up so when I built Big it would first build Zig and Zag. The problem is that Big needed the resulting Zig.jar and Zag.jar, and it used them in a very expensive processing step that would take about 30 seconds. So I really wanted to ensure that that processing step would only happen if a new Zig.jar or Zag.jar had been built. If building Zig and Zag had made no change, because Zig.jar and Zag.jar already existed, then I didn't want to reconstruct Big for nothing.

The obvious way to solve this problem would be to insert a simple <uptodate> check in Big's build.xml that would test whether Big.jar was more recent than Zig.jar and Zag.jar. I didn't much like that, though. What if I added a new dependency, on project Zog? Would I have to remember to add Zog to the <uptodate> check? Surely I could write things in a way that didn't require me to list the upstream jars explicitly. Surely I could derive that list from something that NetBeans would set automatically when the project dependencies changed. Surely.

Let's state the problem a bit more formally. I have a property ${javac.classpath} that contains a list of jars and directories separated by File.pathSeparator. I want to set a property ${jar.up.to.date} only if Big.jar is more recent than every jar mentioned in ${javac.classpath}. I think it should be straightforward to do that in Ant, and it might even be that there is a straightforward way to do it, but even after spending hours puzzling over filesets and selectors and filterchains and resources and patternsets I never succeeded in finding one. Something is wrong when you have to spend endless time trying to see if it is possible to fit the random available jigsaw pieces together to make up the picture you want. I basically ended up writing ${javac.classpath} to a file and performing a series of transformations on that file so that it ended up being the required <uptodate> definition, which I could read in using <import>. Ugh.

Here's how I could solve that problem with the <for> construct which ought to be in Ant but is at least in ant-contrib:

    <for list="${javac.classpath}" delimiter="${path.separator}" param="jar">
        <sequential>
            <condition property="jar.out.of.date">
                <and>
                    <matches string="@{jar}" pattern=".*\.jar" />
                    <not>
                        <and>
                            <available file="@{jar}" />
                            <uptodate srcfile="@{jar}" targetfile="${dist.jar}" />
                        </and>
                    </not>
                </and>
            </condition>
        </sequential>
    </for>
   

(It's easier to compute "not up to date" and use "unless=" here; each loop iteration gets a chance to set jar.out.of.date and it only remains unset if no iteration sets it. Once it's set a property stays set, so there's no easy way to achieve the opposite effect, though of course I could then set jar.up.to.date based on jar.out.of.date.)

But maybe I don't feel like fighting against Ant and XML today, searching through the manual to find the magic <sequential> and <condition> and <available> tags I need. What would I write if I used JavaScript instead?

    <script language="javascript"><![CDATA[
        importClass(java.io.File)
        var base = new File(basedir)
        var targetJarName = project.getProperty("dist.jar")
        var targetJar = new File(base, targetJarName)
        var classpath = project.getProperty("javac.classpath")
        var parts = classpath.split(File.pathSeparator)
        var uptodate = targetJar.exists()
        for each (var part in parts) {
            if (part.endsWith(".jar")) {
                var jar = new File(base, part)
                uptodate &= jar.exists() && jar.lastModified() < targetJar.lastModified()
            }
        }
        if (uptodate) {
            project.setProperty("jar.up.to.date", "true")
        }
    ]]></script>
   

That's arguably no simpler than the <for> version, though it is vastly simpler than my solution with writing out files and <import>ing them again. But the big win is reallyfamiliarity: every Java developer should know enough JavaScript syntax and enough of the basic Java APIs to understand what is going on here. Relatively few developers are so unfortunate as to have had to master enough Ant arcana to be able to plunge without fear into the <for> version.

You do need to know a few basic things about the environment in which scripts execute:

  • The script variable project is set to Ant's Project object, and you can use it to get and set Ant properties with getProperty and setProperty (or, better, setNewProperty), as above, as well as executing other tasks and even adding new targets.
  • Every property, target, and resource collection (such as filesets and dirsets) whose name is a valid JavaScript identifier appears as a script variable of the same name. That's how we can reference ${basedir} as above.
  • As in the first example, println() is available if you need to debug your script.
  • You cannot assume that you are executing in any particular directory so in general you will need to make absoluteFile values using basedir as above, orproject.getBaseDir() which is already a File.

(Incidentally I share Attila Szegedi's mystificationat the absence of online Ant API docs at apache.org.)

You have the full power of the standard Java APIs available to you, but you also have access to Ant's APIs if you prefer to use Ant's internal logic to solve the problems it's best at, such as dependency checking. For example, another way to write the script above, which is more like the <for> version, is this:

    <macrodef name="check-against-jar">
        <attribute name="jar" />
        <sequential>
            <condition property="jar.out.of.date">
                <not>
                    <and>
                        <available file="@{jar}" />
                        <uptodate srcfile="@{jar}" targetfile="${dist.jar}" />
                    </and>
                </not>
            </condition>
        </sequential>
    </macrodef>
    <script language="javascript"><![CDATA[
        importClass(java.io.File)
        var classpath = project.getProperty("javac.classpath")
        var parts = classpath.split(File.pathSeparator)
        for each (var part in parts) {
            if (part.endsWith(".jar")) {
                var checkAgainstJar = project.createTask("check-against-jar")
                // println(checkAgainstJar.getClass()) to discover that it's a MacroInstance
                checkAgainstJar.setDynamicAttribute("jar", part)
                checkAgainstJar.execute()
            }
        }
    ]]></script>
   

So now when I'm faced with a problem that doesn't seem to be easy to solve with plain Ant, I know I have two alternatives to banging my head against the wall of Ant's obtuseness: ant-contrib and Blue Meanies JavaScript.

But what do I know?

 I've hacked my fair share of buildfiles but I wouldn't consider that I'm any kind of expert on Ant. Am I being unduly mean in my opinions? Is it actually really easy to solve problems like mine but I am too dim to see? Have all the cool kids abandoned Ant long ago to move on to a shiny new alternative? Let me know in the comments.

Here is the text of the message I recently sent to the JSR 255 Expert Group, in my capacity as Specification Lead.

Dear experts,

I'm sure that you saw some months ago that our work on JSR 255 will not be part of the JDK 7 release (http://openjdk.java.net/projects/jdk7/features/). This decision was made here at Sun in order that some of the higher priority features could be properly resourced, in particular for the TCK work.

So, we need to retarget our work for JDK 8. This note is to give you an update on how I plan on handling that properly.

Over the next month or so, I'll be setting up a new OpenJDK project called "JMX2" where the Reference Implementation work for JSR 255 will live. This means that all our specification and RI work on JSR 255 will continue to be available to all to evaluate, critique and try out, and it will be up to date with the stable JDK milestone releases.

In order to do this, I will prepare a big changeset (call it the "undo changeset"), which will change the JMX API back to what it was in JDK 6, and apply that to JDK 7. Then, I will kick off the new workspace as a clone of the JDK 7 repository, including the undo changeset. I will immediately apply a "backout" changeset that undoes the undo (using Mercurial's backout command), so JMX2 is where JDK 7 was before I started.

I hope to have better news about making progress with the TCK as the JDK team starts planning JDK 8, which is likely to be towards the end of the JDK 7 cycle (http://openjdk.java.net/projects/jdk7/calendar/), at the beginning of next year.

I've mentioned in the past that one of the new features in version 2.0 of the JMX API is "client contexts", which will allow a client to communicate context information to a server, and a server to adjust its behaviour accordingly.

The most obvious example is locale, where for example the client says that it is in the French locale and the server translates its messages and descriptions into French.

A slightly less obvious example is transaction ids. Here the client sets up a transaction (perhaps related to a database), performs a number of requests on the server in the context of that transactions, then commits or aborts the transaction. Updating configuration is the most obvious example where this is useful, especially if the configuration is controlled by several JMX MBeans. You want the update to have the usual transaction properties, like atomicity and consistency, and for this to be possible the participating MBeans must know their common transaction id.

(I should add that we don't have explicit support for transactions in the JMX API, but we do now have everything you need to build the transactional solution that is appropriate for you.)

Generalizing, a client context is aMap<String,String>, where the keys name context items such as locale or transaction id, with the corresponding values. For example, the locale item is named by the standard string "jmx.locale", so if the context includes the French locale then the Map will have the key"jmx.locale" with associated value"fr".

So what does all this look like in code? Let's take the French example to see.

Client side

Suppose the client has created a connection to the server in the usual way:

    JMXServiceURL url = ...server address...;
    JMXConnector connector = JMXConnectorFactory.connect(url);
    MBeanServerConnection connection = connector.getMBeanServerConnection();
 

All the new stuff related to client contexts is in the unimaginatively named ClientContext class. We can make afrenchConnection that is the same asconnection except that it addsjmx.locale=fr to every request:

    MBeanServerConnection frenchConnection =
        ClientContext.withContext(connection, "jmx.locale", "fr");
 

For the particular case of jmx.locale, there is also a special-purpose method, so another way to do it would be:

    MBeanServerConnection frenchConnection =
        ClientContext.withLocale(connection, Locale.FRENCH);
 

In either case, you can do everything withfrenchConnection that you could do withconnection, such as:

    String message = (String)
        frenchConnection.getAttribute(objectName, "Message");
 

Server side

Now suppose we want to write the MBean named byobjectName above. We want it to return a string from its Message attribute that is translated into the client's locale. Here's what the MBean would look like without any exotic context stuff:

    public interface ExampleMBean {
        public String getMessage();
    }

    public class Example implements ExampleMBean {
        public String getMessage() {
            return "My hovercraft is full of eels";
        }
    }
 

Here's a simplified context-aware version:

    public class Example implements ExampleMBean {
        public String getMessage() {
            Locale locale = ClientContext.getLocale();
            String language = locale.getLanguage();
            if (language.equals(Locale.FRENCH.getLanguage()))
                return "Mon aéroglisseur est rempli d'anguilles";
            else
                return "My hovercraft is full of eels";
        }
    }
 

(In reality you would want to use ResourceBundles and the like, rather than hard-coding translations like this.)

Like ClientContext.withLocale on the client side, on the server side ClientContext.getLocale() is a special-purpose version of the more general-purposeClientContext.getContext(). That method returns the completeMap<String,String> that makes up the context. So if you are using "com.example.xid" as the context key for transaction ids, then an MBean could get the current transaction id with:

        String xid = ClientContext.getContext().get("com.example.xid");
 

Context for local clients

A client in the same Virtual Machine as the server may have a direct reference to the MBeanServer object. In that case it has an alternative way of setting the context. Instead of usingwithContext to get an MBeanServer object where a context item has been set, it can use doWithContext to execute some code with the desired context:

    Map<String,String> newContext =
        Collections.singletonMap("jmx.locale", Locale.FRENCH.toString());
    String message =
    ClientContext.doWithContext(
            newContext,
            new Callable<String>() {
                return (String)
                    mbeanServer.getAttribute(objectName, "Message");
            }
        );
 

If you want to add items to the context rather than replacing it, you can do that straightforwardly:

    Map<String,String> newContext =
        new HashMap<String,String>(ClientContext.getContext());  // make a copy of the context
    newContext.put("jmx.locale", Locale.FRENCH.toString());      // change the copy
    ...doWithContext(newContext, ...)...
 

The Map returned by ClientContext.getContext() is unmodifiable. The only way to set it is usingdoWithContext to execute some code with the new context. As soon as that code returns, the previous context is restored. This means that you can safely call other code without worrying that it might change your context.

No need to care after this point

That's basically all you need to know to understand contexts. If you're not interested in the gory details, you can stop reading now.

How it works: the ugly truth

If we had designed contexts into the JMX Remote API from the very beginning then this would all work in an obvious way. Each protocol request from the client to the server would include the context, if any. On the server side, the context would be decoded from the request and attached to thread handling the request.

While we could have modified the JMX Remote API to work this way in the 2.0 API, it would have posed interoperability concerns. Contexts would only have been available if the client, server, and connector were all running the latest version. Pre-2.0 clients would have had no way to send contexts to servers; pre-2.0 servers would have had no way to receive contexts from clients; and pre-2.0 connector protocols (which is all connectors today) would have been unable to communicate contexts.

Instead, based on an idea by Simone Bordet, we defined a way to encode the context of a request into the target ObjectName of that request. We do this with a special pseudo-namespace called jmx.context//. When you access the ObjectName com.example:type=Foo in the French locale, you are actually accessingjmx.context//jmx.locale=fr//com.example:type=Foo. Since this looks like a namespace,ClientContext.withContext can reuse the narrowToNamespace mechanism (which is similar to a shell "cd" command). Pre-2.0 clients can manually insert thejmx.context//jmx.locale=fr// prefix. Pre-2.0 servers can install an MBeanServerForwarder to remove this prefix and use it to establish the thread context. And connector protocols do not need to be modified in any way, so all existing connectors support contexts.

I called jmx.context// a pseudo-namespace because even though it looks like one it is not actually implemented as one. There are a few reasons for this, but basically it boils down to a question of ordering with respect to other things you might want to add in the path between the client and the server. You might want to add an MBeanServerForwarder to localize the descriptions in the MBeanInfo returned by getMBeanInfo, but for this to work the locale would already need to be set when this localizer intercepted thegetMBeanInfo call. (In fact, the new API includes such a forwarder.) Or, you might already have anMBeanServerForwarder that performs security checks based on the ObjectName in the requests it sees; but if you now start adding jmx.context// into thatObjectName you'll have to change your checking logic.

For these reasons, we simulate ajmx.context// namespace in a specialMBeanServerForwarder that is deployed by default in the RMI connector and can be deployed in any other connector. This works well in practice, and in particular solves the problems described just above. The only drawback is that a local client of the MBean Server does not see the jmx.context//namespace by default. This usually does not matter, because a local client can use ClientContext.doWithContext, which will work. Client code that sometimes needs to operate with a remoteMBeanServerConnection and sometimes with a localMBeanServer should be rare. When it does arise, the solution is to wrap the local MBeanServer inside thecontext namespace forwarder. Since anMBeanServerForwarder is-an MBeanServer, you can just do this once and use the wrappedMBeanServer everywhere you used the originalMBeanServer before.

French locale encoded into ObjectName and decoded in MBeanServerForwarder 

[Tags: jmx jsr255]

The MXBean framework gives you a way to define MBeans with custom types, and map those types to standard ("open") types so generic clients can deal with them. Sometimes you want to do this mapping outside an MBean. For example, we recommend that the custom payload ("userData") of a Notification should use open types, but open values can be a bit painful to construct. It would be nice to use the MXBean framework to map from a custom payload type to an open type. Is there a way to do that?

Let's take an example. Suppose you are tracking coordinates read from some device, and you have a notification that indicates that an (x,y) value has been read that was outside the expected bounds. You'd like the notification to include the value that was read.

In the JMX 2.0 API, you can do this easily, using interfaces I described before:

    public class Coordinate {
        @ConstructorProperties({"x", "y"})
        public Coordinate(int x, int y) {...}
        public int getX() {...}
        public int getY() {...}
    }

    MXBeanMapping coordinateMapping =
        MXBeanMappingFactory.DEFAULT.mappingForType(Coordinate.class, MXBeanMappingFactory.DEFAULT);

    Coordinate outOfBounds = ...;
    Notification notif = new Notification(Coordinate.class.getName(), source, seqNo++);
    notif.setUserData(coordinateMapping.toOpenValue(outOfBounds));
 

Though you don't have to, you can also make the corresponding OpenType available to clients, by putting it in the Descriptor of the corresponding MBeanNotificationInfo. The Descriptor field openType is defined with this meaning, so the code would look like this:

    OpenType<?> coordinateOpenType = coordinateMapping.getOpenType();
    Descriptor notifDescr = new ImmutableDescriptor(
            new String[] {
                "openType",
            }, new Object[] {
                coordinateOpenType
            });
    MBeanNotificationInfo mbni = new MBeanNotificationInfo(
        new String[] {"com.example.bounds.exceeded"},
        Notification.class.getName(),
        "Read a coordinate value outside the expected bounds",
        notifDescr);
 

That's all very well, but the JMX 2.0 API will only be available in the JDK 7 platform. What if you want to do this today?

The good news is that it is possible, although it requires writing some throwaway classes (or generating them at runtime, as I discussed previously). I'll show some general-purpose code later, but for now here's a simple way you could do it. Create an MXBean with an attribute X of type Coordinate, register it in an MBeanServer (perhaps one created specially for the purpose), then arrange for the original value of X to be the Coordinate you want to convert. CallinggetAttribute(objectName, "X") should then give you the converted value, and you can get the OpenType by looking atgetMBeanInfo(objectName).getAttributes()[0].

But actually, that's overkill. You don't need to register an MXBean at all. You can simply use the StandardMBean class to create a private MXBean, and do the same trick. For example, you could write this:

    public interface CoordinateWrapperMXBean {
        public Coordinate getX();
    }
    public class CoordinateWrapper implements CoordinateWrapperMXBean {
        public Coordinate x;
        public Coordinate getX() {
            return x;
        }
    }
    CoordinateWrapper coordinateWrapper = new CoordinateWrapper();
    StandardMBean coordinateMBean = new StandardMBean(
        coordinateWrapper, CoordinateWrapperMXBean.class, true);

    Coordinate outOfBounds = ...;
    coordinateWrapper.x = outOfBounds;
    Object userData = coordinateMBean.getAttribute("X");
 

That's kind of unwieldy, and it's not thread-safe as written, but you can see the general idea.

A general-purpose MXBean conversion class

To reduce the unwieldiness, we can generalize this for any type, in such a way that you only need to write this...

    public interface CoordinateCaller extends Callable<Coordinate> {
        public Coordinate call();
    }
 

...and you can feed it to a simple conversion framework that I'll show below, like this:

    MXBeanConverter coordinateConverter =
        MXBeanConverter.getInstance(CoordinateCaller.class);
    Coordinate outOfBounds = ...;
    Object userData = coordinateConverter.toOpenValue(outOfBounds);
 

You can also get the OpenType like this:

    OpenType<?> coordinateOpenType = coordinateConverter.getOpenType();
 

And you can convert in the opposite direction (for example, when you receive the Notification and you want to reconstruct the original Coordinate object) like this:

    Coordinate outOfBounds =
        coordinateConverter.fromOpenValue(notification.getUserData());
 

If you only have a handful of types that you need to convert, having to create a FooCaller interface for each one shouldn't be too big a burden. If you have a huge number of types, then it would be worth looking at generating each FooCaller interface at runtime, for example using ASM. (Actually, I'd recommend generating an interface with getX and setX methods instead.)

By the way, if you look closely at the CoordinateCaller interface...

    public interface CoordinateCaller extends Callable<Coordinate> {
        public Coordinate call();
    }
 

...you might reasonably wonder why you need to define thecall() method when aCallable<Coordinate> already has one. Ideally you would not have to, but this is a shortcoming of the current MXBean framework.

The approach used here is a variant on thegetAttribute("X") approach I explained above. Since we're using a call() method and not agetX() method, we need to use invoke and not getAttribute on the StandardMBean.

To do the reverse conversion, we make an MXBean proxy for the CoordinateCaller, and arrange for its call to MBeanServerConnection.invoke to return the open-type value, so that the proxy will convert it back into a Coordinate that it returns from its call() method. (Don't worry if you didn't understand that; you don't need to in order to use the converter.)

OK, so here's the code:

package mxbeanconverter;

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.util.concurrent.Callable;
import javax.management.JMX;
import javax.management.MBeanServer;
import javax.management.MBeanServerDelegate;
import javax.management.StandardMBean;
import javax.management.openmbean.OpenType;

public class MXBeanConverter<T> {
    private final StandardMBean mbean;
    private final Callable<T> proxy;
    private final ThreadLocal<Object> threadLocal = new ThreadLocal<Object>();

    public OpenType getOpenType() {
        return (OpenType) mbean.getMBeanInfo().getOperations()[0]
                .getDescriptor().getFieldValue("openType");
    }

    public Object toOpenValue(T javaValue) {
        threadLocal.set(javaValue);
        try {
            return mbean.invoke("call", null, null);
        } catch (RuntimeException e) {
            throw e;
        } catch (Exception e) {
            throw new IllegalArgumentException(e);
            // Should not happen
        }
    }

    public T fromOpenValue(Object openValue) {
        threadLocal.set(openValue);
        try {
            return proxy.call();
        } catch (RuntimeException e) {
            throw e;
        } catch (Exception e) {
            throw new IllegalArgumentException(e);
            // Should not happen
        }
    }

    private MXBeanConverter(Class<? extends Callable<T>> intf) {
        mbean = makeMBean(intf);
        proxy = makeProxy(intf);
    }

    public static <T> MXBeanConverter<T> getInstance(Class<? extends Callable<T>> intf) {
        return new MXBeanConverter(intf);
    }

    private <I extends Callable<T>> StandardMBean makeMBean(Class<I> intf) {
        I impl = intf.cast(Proxy.newProxyInstance(
                intf.getClassLoader(), new Class<?>[] {intf}, threadLocalIH));
        return new StandardMBean(impl, intf, true);
    }

    private <I extends Callable<T>> I makeProxy(Class<I> intf) {
        MBeanServer threadLocalMBS = (MBeanServer) Proxy.newProxyInstance(
                MBeanServer.class.getClassLoader(),
                new Class<?>[] {MBeanServer.class},
                threadLocalIH);
        return intf.cast(JMX.newMXBeanProxy(
                threadLocalMBS, MBeanServerDelegate.DELEGATE_NAME, intf));
    }

    // An InvocationHandler that returns threadLocal.get() from every method.
    // This means that we can make an implementation of FooCallable where
    // the call() method returns threadLocal.get(), and it also means that
    // we can make an implementation of MBeanServer where
    // invoke(...any parameters...) returns threadLocal.get().
    private class GetThreadLocalIH implements InvocationHandler {
        @Override
        public Object invoke(Object proxy, Method method, Object[] args) {
            return threadLocal.get();
        }
    }
    private final InvocationHandler threadLocalIH = new GetThreadLocalIH();
}

[Tags: jmx mxbean ]

You can't construct a JMX ObjectName without handling MalformedObjectNameException, which is a checked exception. Here's why that is a pain, how to relieve that pain, and what we're doing to make it less painful in the next version.

If you've done any programming at all with the JMX API, you'll have noticed that all of the ObjectName constructors, and all of the getInstance methods, are declared as throwing MalformedObjectNameException.

MalformedObjectNameException is a checked exception, which means that you have to either declare it in the throwsclause of your method, or catch it within the method. Having this exception be a checked one was a mistake, in the light of modern practice. For example, Effective Java 2 says: Use checked exceptions for recoverable conditions and runtime exceptions for programming errors. Clearly a malformed ObjectName is a programming error.

When the ancestor of the JMX API was being designed about ten years ago this was all much less clear, and there were plenty of misleading precedents like MalformedURLException. If the API were being designed over again, MalformedObjectNameException would surely be a subclass of IllegalArgumentException, or perhaps the constructors would just throw IllegalArgumentException itself.

As it stands, this is painful, especially when you want to initialize a static final field to be an ObjectName. For example, you might want to write something like this:

    public interface EventClientDelegateMBean {
        public static final ObjectName OBJECT_NAME =
            new ObjectName("javax.management.event:type=EventClientDelegate");
        ...
    }
 

Alas, that doesn't compile, because field initializers can't throw checked exceptions. You might try to write this:

    public interface EventClientDelegateMBean {
        public static final ObjectName OBJECT_NAME;
        static {
            try {
                OBJECT_NAME =
                    new ObjectName("javax.management.event:type=EventClientDelegate");
            } catch (MalformedObjectNameException e) {
                throw new IllegalArgumentException(e);
            }
        }
        ...
    }
 

But that doesn't compile either, because interfaces can't have static initializers. Don't ask me why not.

So the solution today is to write a simple staticnewObjectName method that wraps the MalformedObjectNameException as an IllegalArgumentException, park it in some class, say Util, and call that every time you need to construct an ObjectName:

    public class Util {
        ...
        public static ObjectName newObjectName(String s) {
            try {
                return ObjectName.getInstance(s);
            } catch (MalformedObjectNameException e) {
                throw new IllegalArgumentException(e);
            }
        }
        ...
    }
 

Thus our troublesome interface becomes:

    public interface EventClientDelegateMBean {
        public static final ObjectName OBJECT_NAME =
            Util.newObjectName("javax.management.event:type=EventClientDelegate");
        ...
    }
 

That does compile, and it does work. (Although if the string really is incorrect you will get bashed with an ExceptionInInitializerError when you first reference the interface.)

My favourite tiny change in the JMX 2.0 API is that we do now provide standard methods to do this, so you no longer need to define them. The hardest part was to decide what to call them! As of JDK 7, you can write:

    public interface EventClientDelegateMBean {
        public static final ObjectName OBJECT_NAME =
            ObjectName.valueOf("javax.management.event:type=EventClientDelegate");
        ...
    }
 

[Tags: jmx java.]

We have released a new snapshot of the Web Services Connector being defined by JSR 262. Jean-François Denise has the details on his blog.

[Tags: jmx jsr262 ws-management.]

The JMX Namespace feature has now been integratedinto the JDK 7 platform. You can read about it in detail in the online documentation for javax.management.namespace. Here's my quick summary.

Namespaces add a hierarchical structure to the JMX naming scheme. The easiest way to think of this is as a directory hierarchy. Previously JMX MBeans had names likejava.lang:type=ThreadMXBean. Those names are still legal, but there can now also be names like othervm//java.lang:type=ThreadMXBean.

The othervm "directory" is a JMXNamespace. You create it by registering aJMXNamespace MBean with the special nameothervm//:type=JMXNamespace. You specify its contents via the sourceServer argument to theJMXNamespace constructor.

There are three typical use cases for namespaces. First, if you have more than one MBean Server in the same Java VM (for example, it is an app server, and you have one MBean Server per deployed app), then you can group them all together in a higher-level MBean Server. Second, if you have MBean Servers distributed across different Java VMs (maybe on different machines), then again you can group them together into a "master" MBean Server. Then clients can access the different MBean Servers without having to connect to each one directly. Finally, namespaces support "Virtual MBeans", which do not exist as Java objects except while they are being accessed.

There's much more to namespaces than I've described here. Daniel Fuchs is the engineer who did most of the design and implementation work on namespaces, and I expect he will have more to say about them in the near future on his blog.

[Tags: jmx jdk7 jsr255]

Shanliang Jiang talks about one of the main use cases of the Event Service from the new JMX API in this blog entry. You can use a wildcard ObjectName to subscribe to a set of MBeans. The Event Service tracks the arrival and departure of MBeans, so it will subscribe to new MBeans matching the pattern as they appear. The Event Service is already present in the latest draft of the JDK 7 API and you can try it out by downloading the latest snapshot.

 

[Tags: jmx jdk7 jsr255.]

Yesterday I cleaned up the umbrella bugthat lists the various things we are planning for version 2.0 of the JMX API, which is the version that should be included in Java SE 7. Here's a list of the things we were thinking of doing but are not now planning to.

In some of these cases, we realized after discussion in the Expert Group that the proposed feature was either not practical or too hard to nail down. For example, support for persistence would be good. But what would it look like? Would we give you some way to take a snapshot of every MBean? How would that snapshot be represented? How would it be restored? Or would we have a way for an MBean to define its own persistence explicitly? Then what could we usefully define that an MBean can't already do today?

In other cases, a reluctant engineering decision was taken because, though the feature would be useful, it would not be useful enough to justify the engineering work to specify it, implement it, and write spec conformance tests for it.

Here's the list, with explanations:

  • 4718350Wildcard support in notification filter

    This is superseded by the new class QueryNotificationFilter, which is more general.

  • 5108792 Add abstract base class to make coding DynamicMBeans easier

    Although this would occasionally be useful, the main thing it could do would be to implement getAttributes in terms ofgetAttribute and setAttributes in terms of setAttribute, and in practice people don't code Dynamic MBeans often enough for that to justify a new class.

  • 4718269 Model MBean should be extended to support EJB handles

    This one was never on the cards, because this API is part of the Java SE platform, and EJBs are part of Java EE.

  • 4718298ModelMBeanInfo.getConstructor(String) could be added

    ModelMBeanInfo is an interface so adding a method to it would be an incompatible change. (Existing classes that implement this interface would not have the new method.)

  • 6176837TabularDataSupport.entrySet() should prevent calls to Map.Entry.setValue on returned set elements

    I logged this one a few years ago for aesthetic reasons. The spec basically says "don't do this or things will break". A good spec should never say that. It should say "if you try to do this operation that would break things, you will get an exception". But in practice this is in a pretty obscure corner of the API and making it generate the exception would be a nontrivial amount of work, and probably imply a performance penalty, so the change was hard to justify.

  • 5031385 m-let file format should be XML

    Again MLet files are not a core part of the API so making this change would be hard to justify. What's more we'd probably have to work out how to support the previous almost-XML format and the new really-XML format at the same time, and that would be hard.

  • 6323896Support aggregation of multiple MBean Servers

    This is certainly something that we often see people needing, but the Expert Group discussion concluded that it would be very hard to define something that would be general enough to be useful while still simple enough to be understandable.

    I think this is still an interesting area for future work, but it will unfortunately not be part of the JMX 2.0 API.

    We will still support federation, meaning that you can group several MBean Servers into a single "master" MBean Server. But we will not support for example a predefined way to have an MBean in the master that shows the average of a certain attribute across each of the federated MBean Servers. (Users should still find it easy to address any given case by coding it explicitly.)

  • 6323795Metadata repository for JMX API

    The idea here is that you can see the metadata for every JMX MBean that currently exists (for example, its operations and attributes). But you can't see the metadata for every MBean thatmight exist. It would be great to be able to browse the MBean class hierarchy independently of instances.

    Although this is not planned for the 2.0 API, it would be an interesting project to use the new Namespace support to implement a namespace with one MBean instance for each class of MBean that exists (discovered perhaps by looking through the jars in the classpath). The MBeanInfo of each of these "template MBeans" would be the MBeanInfo that each instance of the corresponding class would have, or as near to that as we could deduce. (This idea is due to Nick Stephen.)

  • 5108724 Add support for persistence to JMX API

    I talked about this above.

  • 6323764 Add support for client sessions to JMX API

    The idea here was that a client could create an object on the server that represented some part of its state, for example a transaction or a set of notifications that the client is interested in or a long-running operation. However, the Expert Group considered that most of these cases could be addressed using "client contexts", a new feature where the client can communicate a context to the server with every operation. This context could be a locale or a transaction id or whatever.

None of these decisions is completely irreversible of course. But at this stage I think anyone with a very strong desire to see any of the departed features brought back would need to volunteer to do all the relevant engineering work!

[Tags: jmx jdk7 jsr255]

The new Event Service that is part of version 2.0 of the JMX API is available in the latest snapshot of the JDK 7 platform. The package description for the newjavax.management.event package summarizes what it is for and how to use it.

The description there starts like this:

This package defines the Event Service, which provides extended support for JMX notifications.

The Event Service provides greater control over notification handling than the default technique using MBeanServer.addNotificationListener or MBeanServerConnection.addNotificationListener.

Here are some reasons you may want to use the Event Service:

  • To receive notifications from a set of MBeans defined by an ObjectName pattern, such ascom.example.config:type=Cache,*.
  • When the notification-handling behavior of the connector you are using does not match your requirements. For example, with the standard RMI connector you can lose notifications if there are very many of them in the MBean Server you are connected to, even if you are only listening for a small proportion of them.
  • To change the threading behavior of notification dispatch.
  • To define a different transport for notifications, for example to arrange for them to be delivered through the Java Message Service (JMS). The Event Service comes with one alternative transport as standard, a "push-mode" RMI transport.
  • To handle notifications on behalf of MBeans (often virtual) in a namespace.

The Event Service is new in version 2.0 of the JMX API, which is the version introduced in version 7 of the Java SE platform. It is not usually possible to use the Event Service when connecting remotely to an MBean Server that is running an earlier version.

As with everything else in the new JMX API, we're always keen on hearing feedback, which you can add as a comment here or send to jmx-spec-comments@sun.com.

The principal designer of the new Event Service is Shanliang Jiang, and he'll probably have some interesting things to say about it, which I'll link to from here.

[Tags: jmx jdk7.]

Version 2.0 of the JMX API is being defined by JSR 255. I've written about some of the features in the new API before. They are trickling into the JDK 7 sources over time, so you can now play with some shiny new things. Here's a description of what they are, and how you can even access them from JDK 6 if you are brave.

As you probably know, the JDK 7 platform is being developed in open source, which means that you can see the changes as soon as the JMX team commits them.

You can browse the current mainline JDK 7 sources directly from the Mercurial repository, at http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/tip.

These sources correspond to the Javadoc snapshot at http://download.java.net/jdk7/docs/api/index.html, and to the JDK 7 snapshot binary at http://download.java.net/jdk7/binaries/.

If you download the JDK 7 snapshot binary, you can use it the same way you would use any other JDK installation. Of course, this is work in progress, so don't use it for anything critical. But it tends to be surprisingly stable, and I often use it to run NetBeans, for example.

Recent JMX API changes in the JDK 7 sources

You can also see all of the changes that have gone into the JDK 7 sources at http://hg.openjdk.java.net/jdk7/jdk7/jdk/shortlog. At the time of writing, one interesting thing you will see there (if you find the JMX API interesting) is this:

                                                         
4 weeks agoemcmanus6323980: Annotations to simplify MBean developmentchangeset | manifest

I wrote about this feature in detail before, and here it finally is. Have a look at the Javadoc for the new @MBean annotation for example, or for MBeanRegistration, where resource injection is described.

You can also see:

2 months agoemcmanus6562936: Support custom type mappings in MXBeanschangeset | manifest

I wrote about that one before as well.

And much further back, you can see:

5 months agoemcmanus6602310: Extensions to Query API for JMX 2.0changeset | manifest

You probably won't be surprised to learn that I wrote about that before too.

There's a small related change...

4 months agoemcmanus6610917: Define a generic NotificationFilterchangeset| manifest

...which I only wrote about in passing before, but you can read all about it in the specification for the new QueryNotificationFilter class.

Changes that are on the way

For a project as big and important as the JDK 7 project, there's a lot of extra engineering that has to happen between the time the JMX team pushes a change into a publicly-visible repository and the time that change actually shows up in the snapshots. I think of the changes as being "in the pipes" during this time. You can see our changes as soon as we push them by looking at http://hg.openjdk.java.net/jdk7/tl/jdk. (Here, tl stands for "tools and libraries", which is the subproject where the JMX API lives.)

At the time of writing, this will show the change
5108776: Add reliable event handling to the JMX API
which is the new Event Service. You might have already seen this in the Early Draft Review of the JMX 2.0 API, but if not it will eventually appear in the Javadoc snapshots in the packagejavax.management.event, or if you can't wait you could always read the source of the javadoc for that package.

There are two other chunks of functionality that should be showing up in the next couple of months, one big and one somewhat smaller. The big one is namespaces. The smaller one is the stuff that will build on namespaces to support "client contexts" (also described in the slides at that link) and localization of descriptions in MBeanInfo.

Once those are in we will be looking at a smallish subset of the various RFEs open against the JMX API. Now would be a good time to agitate for anything you particularly want to see.

Using the JMX 2.0 API with the JDK 6 platform

Using the JDK 7 sources, you can in fact build a jar with just the JMX classes and run that with a JDK 6 installation. Of course this is even less supported than running a JDK 7 snapshot, but once again assuming you're not using it for anything critical here's how you would go about it.

First download the JDK 7 sources, either using Mercurial if you have it, with a command like this...

hg clone http://hg.openjdk.java.net/jdk7/jdk7/jdk myjdk

...or by downloading the sources from http://download.java.net/jdk7/. The "JDK 7 Source under the JRL license" link on that page is a jar which you download and run using a command like this...

java -jar jdk-7-ea-src-b32-jrl-04_aug_2008.jar

Now cd to the jdk subdirectory of the sources, if you downloaded the bundle, or into the Mercurial repository you cloned if you did that (you only cloned the jdk subdirectory). Run this command...

ant -f make/netbeans/jmx/build.xml

If it works, it'll make you a jmx.jar indist/lib and that's what you can use with your JDK 6 installation. (If it doesn't work, tell me what went wrong and I'll update the instructions here.)

Now you can use your new jmx.jar in either of two ways. The simplest is to make a directory calledendorsed inside the jre/lib subdirectory of your JDK installation (if there isn't one already), and then drop jmx.jar into it. (I would recommend renaming it too, to something like jmx-2.0-snapshot-2008-08-06.jar.) Butbeware that if you do this you are replacing the JMX implementation for all users of that JDK installation!

The second way is to run your programs like this...

java -Xbootclasspath/p:/path/to/jmx.jar ...

Then only programs that are explicitly run in that way will be using the new JMX implementation.

Let us know how you get on

If you play with the new JMX API, or even if you are just browsing through it, we're very interested in hearing about your experiences. Feel free to add your comments here, or send mail tojmx-spec-comments@sun.com, or write a blog entry with tag"jmx". Thanks!

[Tags: jmx jdk7.]

This is the fifth and last installment in my summary of the sessions I attended at JavaOne this year.

 

The previous installments covered Java futures, Java programming practice, concurrency, and JMX stuff.

Once again, the titles here are not always the ones that appear in the conference programme. I tried to undo the mangling that the Trademark Police typically impose on talk titles, so let me just state up front that Java, JMX, and JVM are trademarks of Sun Microsystems, Inc., and any other abbreviation beginning with J has a sporting chance of being one too.

Here are the sessions summarized in this installment:

The Garbage-First Garbage Collector
Jamming with Java: Making Music with JFugue and JFrets
Semantic Web for the Working Ontologist
General Session: Extreme Innovation (the Toy Show)


TS-5419, The Garbage-First Garbage Collector, Paul Ciciora, Antonios Printezis. Antonios (Tony) presented the "Garbage-First" Garbage Collector, nicknamed G1, which will be showing up in an update of JDK 6. This is an improved version of the CMS collector (also originally designed by Tony). The new collector is better because it does not fragment memory as CMS can, and it does not require tuning the sizes of separate memory areas as CMS does. The performance should also be better. What made this presentation work was the painstakingly-constructed diagrams, which really helped understand the underlying ideas.

Tony talks more about this in an interview on the JavaOne website.

TS-5263, Jamming with Java: Making Music with JFugue and JFrets, David Koelle, Matt Warman. JFugue is a very nifty system that allows you to represent MIDI sequences with strings. This is hugely easier to work with than the javax.sound.midi API. JFugue adds useful notions like repeated patterns, which can be submitted to arbitrary transformers. It supports tapping out rhythms using arbitrary characters and then applying a substitution that says for example that *means a snare drum eighth-note and . means an eighth-note rest, so ..*...*. is a pattern that hits the virtual snare drum on the weak beats in 4/4 time.

Pedant moment. The melody shown for Frère Jacques in the slides had a D instead of a C as the fourth note, which means it wouldn't work as a round.

JFrets builds on JFugue to provide an app that does guitar tablature. I don't have much to do with tablature but JFrets seemed pretty useful for those who do.

TS-5555, Semantic Web for the Working Ontologist, Dean Allemang. I generally try to fit in at least a couple of sessions that are completely unrelated to my professional activities, and this is one. I chose this one because, like many people, I'd heard the term "semantic web" being bandied about but I didn't have any real idea of what it meant. Also, I know Dean from way back.

I think this area suffers from frightening terminology such as "ontologies" and indeed "semantic web" itself. It sounds a lot more complicated than it actually is. Which is not to say that it is simple.

The basic idea as I understand it is that today we have this fantastic web of human-readable information. We'd like to build on that infrastructure to create a web of machine-readable information as well.

Some questions are rather easy for people to answer by browsing the web. For example: who were the speakers at JavaOne 2008? But programs can't easily answer such questions because the human-readable information on the web isn't formally structured. The Semantic Web would allow web pages to be structured in a way that allows a program to understand them, without needing to build logic into the program for each information set.

There's an obvious parallel here with web forms intended to be filled out by humans, versus web services APIs intended to be called by programs. I don't know enough about the area to be able to say whether this might mean there is a sort of rivalry between web services and the semantic web.

I found it possible not to be scared by the word "ontology" by mentally replacing it with "schema". An "ontology" represents a set of concepts and the relationships between them. Well if that isn't a schema, it's near enough to one for me.

One thing Dean stressed is that it isn't necessary for some standards body to create all the world's schemas. The semantic web includes support for mapping between the concepts of different schemas, for example to say that "address" in my schema is the same as "location" in your schema. Of course there could be subtle differences in the semantics (perhaps "location" can represent the North Pole while "address" can't), but you can still get a lot of useful work done even with these mismatches at the edges.

And that's enough about the Semantic Web to be able to avoid sounding stupid when talking about it at parties. This talk, and the fact of writing it up, has led me to discover some promising starting points for further exploration, of which the best I've found is an article called Giant Global Graph by Tim Berners-Lee.

Sun General Session, Extreme Innovation, James Gosling and guests. Also known as the Toy Show, this is the annual overdose of geekery where James shows all the latest nifty stuff. It also doubles as a good way for the conference organizers to persuade people to stay around for the last day of the conference.

Unusually, I was right there at 8:30am when the show was due to start, because I knew that the first item would be VisualVM, being demonstrated by Tomas Hurkaand my teammate Luis-Miguel Alventosa. Equal measures of pride and stress for them, I think, as they stood up and demoed their stuff in front of thousands of people in a huge dark auditorium. In the event, the demo went off perfectly, as did the other demos from this session. (Unlike the first general session of the conference, but the less said the better.)

If you use JConsole, you should check out VisualVM.

Among the many other nifty things in this session, I liked the contraption that Greg Bollella built to show off Real-Time Java.

http://weblogs.java.net/blog/emcmanus/archive/rtjcontraption.jpg

Little coloured balls circulate through the plastic tubes, occasionally encountering pinball flippers or a bean machine. Eventually each ball falls past a colour detector and into a sorter. The Real-Time Java system has to flick the right switches while the ball is in free-fall so that it ends up in the right bin. Plain Java wouldn't be up to the job (what if a garbage collection happened at just the wrong time?), but Real-Time Java is. The box at the bottom of the picture is Project "Blue Wonder", a collection of Sun technologies suitable for use in industrial control processes. You can see more details in the video of this part of the session.

But the thing that impressed me most, along with most of the people I was talking to, was the Livescribe smartpen. It records your pen movements as you write and can upload them to a computer to reproduce what you wrote. It records ambient sound at the same time and can upload that as well. If you go back to something you wrote and touch it with the pen, the pen can play back the sound from the time you were writing. You can also search through your writing on the computer (handwriting-recognition algorithms seem to have improved without bound since the last time I was paying attention) and play back there the sound from what you find. There were some other nice demos: drawing a toy piano keyboard, then having the pen sound the notes from the keys you touch; or touching a word with the pen and having it pronounce the equivalent word in Arabic. But even without these I thought the pen would be invaluable to anyone who has to take notes often, for example students. It's not even very expensive: $149.

Jeff bought one for his wife (one can't buy such trivia for oneself of course) and she's apparently very happy with it. The pen needs special paper, but I noted with approval that Livescribe will be making software available so that you can print it yourself if you have a good-quality printer.


Well that's it from me about JavaOne for this year. Back to more mundane blogging...

[Tags: javaone javaone2008 garbage_collection garbage_first jfugue jfrets semantic_web visualvm real_time_java rtsj livescribe smartpen.]

This is the fourth installment in my summary of the sessions I attended at JavaOne this year.

 

The previous installments covered Java futures, Java programming practice, and concurrency. This one covers JMX stuff.

Once again, the titles here are not always the ones that appear in the conference programme. I tried to undo the mangling that the Trademark Police typically impose on talk titles, so let me just state up front that Java, JMX, and JVM are trademarks of Sun Microsystems, Inc., and any other abbreviation beginning with J has a sporting chance of being one too.

Here are the talks summarized in this installment:

JMX Technology Update
Practical JMX Security Options
Designing Manageable Java EE Applications in a Clustered Environment

In the remaining installment, I'll cover all the other stuff that didn't fit into any of my neat categories.


TS-5199, JMX Technology Update, Jean-François Denise, Éamonn McManus. Hey, that's us!

  http://weblogs.java.net/blog/emcmanus/archive/eams-thinks-at-podium.jpg 

Jeff(Jean-François) and I have been a double act for the last few JavaOnes and we repeated that here. I presented JSR 255, the JMX 2.0 API, much of which I have described to death elsewhere on this blog. Jeff presented JSR 262, the Web Services Connector.

Since Jeff is a bit of a poker fiend he had the idea that this year's demo could be based on poker somehow. He found an open-source poker server called CSPoker, created by students at the Katholieke Universiteit Leuven. He encouraged them to create a JavaFX GUI for the server.

  http://weblogs.java.net/blog/emcmanus/archive/poker-table.jpg 

Then he concocted a scenario where I was an evil poker cheater found out through JMX instrumentation. Thanks to the Web Services Connector, this JMX instrumentation was accessible through the same web server that players use. Once my cheating was discovered, the Web Services Connector also allowed Jeff to write a VBScript program that would access the JMX instrumentation to detect my cheating and kick me off.

  http://weblogs.java.net/blog/emcmanus/archive/sunglasses.jpg 

We rehearsed this demo until nothing could go wrong, and foresaw every possible problem. Notice the yellow cable linking the two laptops - no way were we going to rely on the conference centre's network to function as it should. But of course something did go wrong. There was much less light in the room than when we rehearsed there earlier, and I couldn't see the keyboard with my sunglasses! Well, that was pretty minor as problems go, and I even got an audience laugh out of it.

We were pleasantly surprised at how well received this simple sketch was. Actual applause when I showed my cheater's hand of four aces! We'll be thinking about another instructive demo for next year's talk, if it's accepted.

(The photos here are from hundredstaken by Yuichi Sakuraba at the conference.)

BOF-5698, Practical JMX Security Options, James Gould, David Smith. This was a very good introduction to JMX security from people who have obviously worked closely with it. Unlike the technical sessions, I don't think the slides from the BOFs will be available on the JavaOne site, but if they show up on the web somewhere I'll certainly link to them.

The message I took away from this BOF was probably different from most attendees. Configuring JMX security is too hard. We designed everything around standard Java security, using a SecurityManager, policy files, and so on. But almost nobody uses a SecurityManager, and although we also provide ways of configuring security without one, they are either too basic or too hard. An area for us to work on.

BOF-4945, Designing Manageable Java EE Applications in a Clustered Environment, Jens Jensen, Peter Kristiansson. Jens and Peter are at Ericsson, and I've had some contact with them before on this subject. Peter (presenting alone) described their solution for handling configuration in a clustered app server.

In their solution, each instance (JVM) has an MBeanServer that exposes a read-only MBean view of a tree of configuration elements. These MBeans get their data from a configuration provider in each JVM. One of the configuration providers is the master and the others are replicas. Updating the configuration involves a console or other management client updating the configuration in the master provider, which transactionally updates the persistent store and forwards the changes to the other providers. When a configuration provider (master or replica) gets a change, it informs the corresponding MBean so the MBean can send a notification.

They use Shoal to manage the cluster of configuration providers. They use JSR 303 (Bean Validation) to express constraints on configuration items.


In the next and final installment I'll cover the remainder of the sessions I attended.

[Tags: javaone javaone2008 jmx shoal bean_validation configuration poker CSPoker.]

Filter Blog

By date: