Skip navigation
1 2 3 Previous Next

dwalend

34 posts
dwalend-JavaNet

Good-bye, Alpha Blog

Posted by dwalend-JavaNet Jul 23, 2008

I released SomnifugiJMS v22 a few weeks back. Not alpha-0-22. This subtlety is a big change in approach for a small change in code. I make fewer changes in each release, and the extremely stabile JMS specification is only so big. I've added most of the features I intend to implement. After eight years of work, six years in production systems, and 21 alpha releases it's starting to feel done. I'd like to claim that I came to this conclusion on my own, but others had a big influence.

I hadn't thought to do it at all until I read Kirill Grouchnikov's series of articles on his approach to open source projects. Kirill says point blank not to do alpha releases: "...in a virtual world this means stripping the alpha / beta / 0.* status from your project. You have very short time window to impress the potential users, and a solid release number with no greek letters is a must-have. This is the least you can do after six years of development." Kirill's approach is not like mine, but his words got me thinking.

Bruno Ghisi pointed out a remarkable survey on why we do open source work. I suspect Kirill is much closer to the "professionals" and "believers" groupings than I am. I straddle the "skill enhancers" and "fun seekers" groupings. I write most of my code after hours, even if it is for work instead of hobby. I pick projects that teach me about new technologies and keep my skills sharp. Further, I enjoy the meditative nature of coding the way other people enjoy watching TV. It's something I do to relax, clear my head and settle down at the end of the day. My only nod to the "professionals" is that I wait to register new open source projects until I have a first release ready to go. However, for me to stay focused on a project without pay, that project has to entertain me or at least teach me something.

Alpha labels on my projects are a warning to everyone: This code isn't ready for prime time. I'll change the API if I need to, and I think I'll need to. And I won't apologize for it.

SomnifugiJMS is easy for me to recognize as mature. The JMS specification has been stabile for six years. SomnifugiJMS supports most of it. I have no plans to support the missing parts. SomnifugiJMS is done except for maintenance releases.

JDigraph is a very different story. The project is, among other things, an experiment on how to create good, clear API. Splitting mutators off from accessors worked amazingly well. Adding type specifiers for nodes and edges taught me how to use generics well. That part of the code is now very stabile. Building generic graph minimization algorithms exposed some big weaknesses of the Java language change that no one was talking about. I want to fix that, although it really needs operator overloading to do well. (I'll probably use that project to teach myself Scala.) Eventually I'll snap a line, cut out the failed experiments, and change over to maintenance releases. But not just yet.

SalutafugiJMSis in between. I finished its second release a few weeks ago. I've covered about a quarter of the big challenges in JMS. It's JMS again, so others determined the API six or more years ago. However, I know I want to switch over to an open, pure Java ZeroConfig library underneath -- a big code quake. I marked the second release as alpha. Maybe release four will have enough of the JMS specification to drop the alpha label.

At JavaOne this year I did a short talk on using JMX in test-driven development. The talk was based on what I discovered working with JMX on another project, and delved deeper into while adding JMX supp ]]>ort to <a href="https://somnifugijms.dev.java.net/">SomnifugiJMS</a> as part of <a href="https://somnifugijms.dev.java.net/servlets/ProjectDocumentList?folderID=9103&expandFolder=9103&folderID=0">release 21</a>. Test-driven development worked extremely well when combined with JMX. JMX should help te

Test Driven Development

Test driven development maximizes the benefits of writing tests by demanding that we write the tests before the code that will attempt to pass those tests. It ensures some level of quality because all work completed will pass at least one test. Developers establish a rapid, highly iterative development loop. The tests provide an easy way to measure progress. Managers enjoy having the detailed insight that a list of tests gives them. When I've got a little time away from the day job to do open source work, I can write a small test, then get back to making the test pass later.

Test code and JMX monitoring code have a shared programatic problem. We promise ourselves that we'll go back and write test code later, and will add JMX monitoring code before we need it. Deadlines loom large, we get things working, and move on to the next project without doing either job well. Test driven development breaks that vicious cycle for tests. Using JMX in the tests breaks the cycle for both. (Thanks to Chris for pointing this out after my talk.)

The test driven development loop is pretty simple. First, decide what feature to implement next. Design the API for that feature. Next, write a test for that feature and observe that all the other tests still pass. Finally write code to make the new test pass. Repeat until tests of all the features pass. Like John says, you can write the tests just after the code so long as you create tests in the same flow and are done before your work escapes your head and your desktop. Your code will have tests before it leaves the cradle.

Test driven development presents some challenges. You have to have some clue about what features to develop before starting the loop. You need to have some idea about the classes and methods that will form your API. Test driven development works best for black-box testing, so you'll have to expose enough of your system to test your work. In Java, that often means adding public access to classes and methods, hacking in via reflection, or excising test code out of packages and classes after you run your tests. JMX can help with these challenges somewhat, by providing an alternative way to get information out.

JMX

JMX is the Java Management eXtension API (from the late 1990s -- the X dates it). JMX provides a standard way for Java systems to report their state and health to the outside world, and to allow themselves to be adjusted and tuned via standard tools. It provides two main styles of interacting with the running system: MBeans expose select getters, setters and methods to outside control. Notifications push information out from the Java system to listeners.

MBeans expose select methods on a class to the JMX system. Java classes follow a special design pattern -- the MBean -- and registering instances of the class with an MBean server. An MBean interface defines the exposed methods. You could use this interface to expose useful attributes to make them observable in tests, and to set attributes and call methods in order to drive tests. For testing, the MBean interface doesn't provide any great advantage over just using public methods. However, I found I could use the MBean's existence in the MBean server to provide a simple test of an object's lifecycle.

JMX also specifies using Notifications to signal lifecycle and state changes from inside objects via the speaker/listener pattern. Add NotificationListeners to objects that implement the NotificationEmitter interface. When something of note happens within the MBean object, have it send a Notification to the listeners. This speaker/listener pattern works well in testing. It provides a good alternative to exposing state because the exposure is controlled by your code sending the Notifications, not via public method calls to your object. Further, it lets you write tests of lifecycle changes instead of tests based on methods. Lifecycle events are easier to get right early in development, generally easier to change, and are less likely to change over time than method signatures. Your tests will be less brittle and you'll spend less time rewriting them.

JMX-Based Tests

I found a few key features to make the code easy to test, and a few key features to test. MBeans will be easier to test if each MBean knows its ObjectName. Tests should check that expected MBeans exist. Because the MBean server will hold on to stray references, tests should check that MBeans are cleaned up at the end of their useful life. Notifications need to be machine-understandable; I subclassed Notification and added an enum field. Define the MBean interface and the Notification help in the same file to make it easier for others to find the Notifications. Tests should check that the code sent the expected Notifications exercised during the test run. These code examples are from SomniMessageConsumer.java, SomniMessageConsumerMBean.java and MonitorTest.java.

Each MBean Knows its ObjectName

To make each MBean know its object name, put the method in the MBean interface:

  public interface SomniMessageConsumerMBean { public ObjectName getObjectName(); ... }  

and fill it in:

  public abstract class SomniMessageConsumer implements MessageConsumer, SomniMessageConsumerMBean, NotificationEmitter { ... private final ObjectName objectName; protected SomniMessageConsumer(String name, Takable feed, SomniExceptionListener somniExceptionListener, String destinationName, SomniSession session) { ... try { objectName = new ObjectName("net.walend.somnifugi:type=" +this.getClass().getName()+", name="+name); registerWithJMX(); } catch(MalformedObjectNameException mone) { throw new SomniRuntimeException("JMX problem with "+this.getName(),mone); } }  

Using the MBean in your test code becomes easy; you aren't constantly writing queries to find the right MBean.

  expectedMBeans.add(subscriber.getObjectName());  

Tests Check for Expected MBeans

I check what MBeans were registered in the server at least twice per test, so I carved out a method to make sure that everything was there, with nothing extra, using an MBean query.

  @SuppressWarnings("unchecked") void checkExpectedMBeanNames(Set<ObjectName> expected) throws MalformedObjectNameException, MBeanException, AttributeNotFoundException, InstanceNotFoundException, ReflectionException { MBeanServer server = ManagementFactory.getPlatformMBeanServer(); ObjectName query = new ObjectName("net.walend.somnifugi:*"); Set<ObjectName> result = (Set<ObjectName>)server.queryNames(query,null); Set<ObjectName> remainder = new HashSet<ObjectName>(result); remainder.removeAll(expected); assertTrue("Remainder is not empty: "+remainder,remainder.isEmpty()); assertEquals(expected,result); }  

Then I fell into a regular pattern in the tests -- The code makes all the JMS parts, then makes sure they are all in the server.

  SomniTopicConnection connection = (SomniTopicConnection)SomniJNDIBypass.IT.getTopicConnectionFactory().createTopicConnection(); SomniTopicSession session = (SomniTopicSession)connection.createTopicSession(false,Session.AUTO_ACKNOWLEDGE); String topicName = "testNotifyTopicSubscribers"; SomniTopic topic = SomniJNDIBypass.IT.getTopic(topicName); SomniTopicPublisher publisher = (SomniTopicPublisher)session.createPublisher(topic); SomniTopicSubscriber subscriber = (SomniTopicSubscriber)session.createSubscriber(topic); connection.start(); MBeanServer server = ManagementFactory.getPlatformMBeanServer(); Set expectedMBeans = new HashSet(); expectedMBeans.add(connection.getObjectName()); expectedMBeans.add(topic.getObjectName()); expectedMBeans.add(session.getObjectName()); expectedMBeans.add(publisher.getObjectName()); expectedMBeans.add(subscriber.getObjectName()); checkExpectedMBeanNames(expectedMBeans);  

Tests Check that MBeans are Cleaned Up

The second call to checkExpectedMBeanNames() makes sure that the test cleans out all of the MBeans at the end of the test. If the code doesn't clean them out, the server holds onto a reference, which prevents the garbage collector from reclaiming the memory. Some of these objects hold many references, so a leak can use a lot of memory.

 connection.close(); SomniJNDIBypass.IT.removeTopic(topicName); expectedMBeans.clear(); checkExpectedMBeanNames(expectedMBeans); } 

Machine-Understandable Notifications

For a finer lifecycle test, I created machine-understandable Notification subclasses. Notification comes with a String type, Object source, and a long sequenceNumber. The source and sequence number are pretty straightforward to use. The type String offers very little. I made a subclass to carry a machine-understandable enum and to help with consistency in the type string. This enum has to implement SomniNotificationType. There's no way to enforce that in Java, but it's at least javadoced. getSomniNotificationType() provides a way to check the enum in code.

 @SuppressWarnings("serial") public class SomniNotification extends Notification implements Serializable { private final SomniNotificationType noteType; private final String partName; public SomniNotification(SomniNotificationType noteType, Object source,long sequenceNumber, String partName) { super(noteType.getTypeString(),source,sequenceNumber,partName + noteType.getVerbPhrase()); this.noteType = noteType; this.partName = partName; } public SomniNotificationType getSomniNotificationType() { return noteType; } public String getPartName() { return partName; }  /** To be implemented by an enum defined in the MBean interface. */ public static interface SomniNotificationType { public String getTypeString(); public String getVerbPhrase(); }  } 

Collocate the MBean interface and Notification Enum

I put the notification enum in the same file, as a static inner class of the MBean interface. This grouping means there's only one place to look in source code to understand how the MBean interacts with a JMX monitor.

  public interface SomniMessageConsumerMBean { ... @SuppressWarnings("serial") public static enum SomniMessageConsumerNotificationType implements SomniNotification.SomniNotificationType, Serializable { CONSUMERCLOSED("net.walend.somnifugi.ConsumerClosed"," closed"), MESSAGECONSUMED("net.walend.somnifugi.MessageConsumed"," consumed"), MESSAGELISTENERCHANGED("net.walend.somnifugi.MessageListenerChanged"," changed"), MESSAGETIMEDOUT("net.walend.somnifugi.MessageTimedOut"," timed out"); private final String typeString; private final String verbPhrase; private SomniMessageConsumerNotificationType(String typeString,String verbPhrase) { this.typeString = typeString; this.verbPhrase = verbPhrase; } public String getTypeString() { return typeString; } public String getVerbPhrase() { return verbPhrase; } }  

Tests Check Notifications Sent During the Run

I created TestNotificationListener to test that the system was producing the right notifications. It takes a list of expected notifications in the constructor, and verifies that all the notifications came in the right order inside a check() method.

  private static class TestNotificationListener implements NotificationListener { private List expected; private List received = new ArrayList(); TestNotificationListener(List expected) { this.expected = expected; } public void handleNotification(Notification notification, Object handback) { SomniNotification somniNotification = (SomniNotification)notification; received.add(somniNotification.getSomniNotificationType()); } void check() { assertEquals(expected,received); } }  

In the test, I give the TestNotificationListener a list of notifications, do things during the test, and make sure the right notifications were sent. This approach lets me define the expected lifecycle steps within the class, and observe them from outside, without using public methods.

  List expected = new ArrayList(); expected.add(SomniMessageConsumerMBean.SomniMessageConsumerNotificationType.MESSAGECONSUMED); expected.add(SomniMessageConsumerMBean.SomniMessageConsumerNotificationType.MESSAGECONSUMED); expected.add(SomniMessageConsumerMBean.SomniMessageConsumerNotificationType.CONSUMERCLOSED); TestNotificationListener listener = new TestNotificationListener(expected); Object handback = null; server.addNotificationListener(name,listener,null,handback); ...//Lots of code for the test listener.check();  

Where This Technique Will Work

Using JMX for test-driven development will work very well in some cases, but doesn't fit all problems. If you want to support JMX MBeans and Notifications, using this technique is an easy right decision. If objects in your system have nontrivial lifecycles, it is a very good fit. If you aren't sure about the methods your system should support, but know more about the lifecycle of the objects, it should work well. SomnifugiJMS was a particularly good fit for JMX-based tests. I wanted to add JMX support. The Java Message Service specification defines a horde of parts used to ship message around. Each part has its own lifecycle. Plus I wanted to remove non-JMS public methods, and to use JMX to support more complex tests.

Some systems may have trouble using this technique. JMX has some overhead; if you are concerned about how much CPU and memory goes into creating and shuffling strings for logging, JMX may be too heavyweight. It's not a very good fit for data structure or algorithm code, where the public API is the main focus. Systems with no lifecycle won't get much benefit. JMX-based tests written for systems with extremely complex, changing lifecycle may be too brittle.

References

Wikipedia has a non-volatile description of test-first development. I found this when I was looking for something with minimal hype and political agenda. The JMX reference from Sun contains examples of how to use JMX in your source code. The bias in the document is more toward monitoring existing systems -- it does a great job of telling how to hook up monitors via different protocols, but a skilled programmer see how to use stock JMX classes in an existing system via delegation. Eamonn McManus' Eamonn McManus' blog is the best source for what's coming next in JMX.

My day job is software architect for a bunch of algorithm scientists who "feel the need for speed." We work to keep the system safe from the evils of premature optimization while finding hot spots that really speed things up. I won't bore you with the details, but we are computing a lot of solutions to a problem in advance, and looking up the answer in a big multidimensional boolean array. For this particular puzzle, the size and accuracy of the problem we can solve depends on how big a monster boolean array we can pack into memory.

We were blowing through the top of the JVM's heap violently via OutOfMemoryErrors, so I went digging with the profiler, and digging deeper with Google. From Sun's tutorial on primitive types, "This data type represents one bit of information, but its "size" isn't something that's precisely defined." That's really freaky, given that bytes, shorts, ints, and longs are precisely defined, that floats and doubles are precisely defined by IEEE 754, and that chars are precisely defined by Unicode.

Stack Frames

A peak at bytecodes showed me that booleans are bigger than a bit. In a stack frame, booleans take 32 bits. That makes sense; the usual JVM is a 32 bit machine. From a discussion forum posting, you can see that the operations are all integer operations (starting with 'i') in the bytecodes, with 31 unused bits in the register:

0 iconst_0
1 istore_1
2 iconst_0
3 istore_2



In the Heap

Arrays of booleans are also bigger than I thought. The profiler and cheezy experiments were telling me booleans were about 8 times bigger than they should be. Google came through on this one, too, in another forum. In my JVM, boolean array elements are bytes.

Other People's Bits

A little more digging showed me that a BitSet uses a long, and an EnumSet uses a long or arrays of longs to hold their bits, not boolean arrays. From BitSet.java:

    /**
     * The bits in this BitSet.  The ith bit is stored in bits[i/64] at
     * bit position i % 64 (where bit position 0 refers to the least
     * significant bit and 63 refers to the most significant bit).
     * INVARIANT: The words in bits[] above unitsInUse-1 are zero.
     *
     * @serial
     */
    private long bits[];  // this should be called unit[]



Arrays of booleans aren't java.util's choice. They won't be mine.

More Questions

Remaining questions: Why use longs and long[]s instead of ints and int[]s? Are the java.util programmers anticipating 64-bit machines? Is there something inherently more efficient about longs, which are usually less efficient? Or are the library writers trying to cover 33-64 members more efficiently and giving themselves twice the maximum upper size limit?

Does the weak definition of a boolean rate a bug or a request for enhancement? It seems to break write-once-run-anywhere because I can't predict how much memory my application requires. Should the compiler convert boolean[]s into the appropriate primitive or primitive array without us having to think about it? Do we want a sizeof() method?

SalutafugiJMSis a peer-to-peer implementation of the Java Messaging Service specification that uses ZeroConf DNS-SD discovery and TCP sockets to communicate in a distributed computing system. I built it after seeing Daniel Steinberg's JavaOne talk on ZeroConf. SalutafugiJMS usesSomnifugiJMS as a skeleton and Apple's Bonjour implementation of ZeroConf for muscle inside special SomnifugiJMS FanOuts and Channels. SalutafugiJMS eliminates the central (or federated or clustered) message broker found in traditional JMS implementations and the need to manage specific services required by traditional DNS-SD systems. Name the JMS Queues and Topics for information your system needs to exchange. Your system consumes what your system needs. Your system sends out what it chooses. SalutafugiJMS takes care of the rest, leaving your system very loosely coupled.

Broker-Free JMS

In a traditional JMS application, message producers and message consumers don't have to know about each other. That feature createshighly decoupled systems. However, message producers and consumers all have to get connections to the same JMS message broker. That shared knowledge about the JMS broker usually comes from magic connection url strings to summon objects out of JNDI servers. Getting those magic strings correct is tedious; many elaborate projects focus on abstracting away those strings, sometimes to the point of programming in XML. SalutafugiJMS is a peer-to-peer system -- producers send messages directly to consumers -- while remaining highly decoupled. The only magic string for you to wrangle is the name of the JMS Topic or Queue. No central broker exists; the system can handle failures in any single service and keep going.

Anonymous ZeroConf

In a traditional ZeroConf system, ZeroConf provides service discovery; a service registers itself so that applications can discover and use it. Those applications browse for the services they need, and resolve them by name. The system is loosely coupled at build time, but becomes very tightly coupled as soon as the application resolves the services at runtime. If the application depends on a service that fails midway through a run, that application suffers. SalutafugiJMS adds JMS' anonymous endpoints to ZeroConf to eliminate that tight coupling. Behind the curtain, SalutafugiJMS consumers register what Queue or Topic they wish to receive messages from. SalutafugiJMS manages browsing and resolving the services inside the producers. Your applications simply produce and consume messages in the system when they need to.

Building SalutafugiJMS

I spent about thirty hours' total reading, designing and coding to get messages flowing between JMS Topic Publishers and Subscribers. Bonjour's programmers' interface is a little call-back heavy. Stopping all the services' monitor threads took some thought and a little reworking of SomnifugiJMS. Daniel and Stuart Cheshire's book was very good; everything worked like they said it would. Getting to the initial release was easy and fun. I'd hoped to be able to mine the effort for half-a-dozen blogs, but I think I'll only get two -- one on Thread.sleep(), and one comparing ZeroConf, JXTA and JINI.

The only aggravating part was getting the Bonjour dns_sd.jar from Apple. I do most of my work on an older G4 Mac (amazingly long battery life). Apple's Bonjour page provided a windows .exe installer to get at that dns_sd.jar. I got the jar from a friend with a newer Intel Mac. Daniel tells me I could have pulled in theBonjour source code and built the dns_sd.jar myself after all. I've gotten very used to Java's jar-based library distribution. I normally only pull in source code to help fix problems and add features. I didn't even think to look.

SalutafugiJMS' first release is very much alpha-0-1, JMS Topics-only, and missing many features. SalutafugiJMS seems a little slow and I haven't looked into why. I haven't used SalutafugiJMS in any of my own applications yet, so downloader beware. Alpha-0-2 will focus on logging and fault management, alpha-0-3 on filling in features for Topics.

If you're more interested in broker-free JMS now and can swallow a GPL (or maybe MPL) license, have a look at MantaRay. I think they use a proprietary multicast protocol, not ZeroConf. They've had a working peer-to-peer JMS for a few years.

JavaOneis an amazing conference. About 15,000 of us get together to exchange ideas, hear about new things, meet internet friends face-to-face, and explore the boundaries of Geekdom. I use the conference to learn about things far afield from my work, things I can't just read about, and tips for handling the crises I'll wade back into after being gone for a week. Midway through Tuesday, the ideas start mixing in my head. By Wednesday morning, new project ideas and solutions to old problems start percolating in my head. JavaOne is also a test of endurance. I typically loose five pounds and only get about five hours of sleep each night. Consider that carefully before taking my advice.

Getting There

If you're flying in, Oakland is as good as SFO. It's easy to get to the downtown hotels from the other side of San Francisco Bay via BART. Phone the hotel clerk to ask which stop is closest to them. Jet Blue had the best flights for me this year, possibly due to an embarrassing episode this winter. Travel light enough to skip the car. There's no room for cars in San Francisco and it's a great city for walking.

Sleeping There

All I really do at the hotel is sleep, shower and monitor what the crew is up to back at work, so I pick an inexpensive hotel close to the convention center. Hotel service in San Francisco has always been pretty good. The people who work the desk tend to live somewhere in town, and know the place. I found hotel prices via the conference to be a little better, but hotel prices are pretty variable. Every year someone seems to have revamped an older hotel with funky art and all new furniture. For the first year after the revival these hotels have the best price, too. That's why I've never stayed in the same place two years in a row.

Pick a hotel that's a good walk to Moscone. JavaOne has busses, but a walk feels good after days underground.

JavaOne is not a great time to sleep on a friend's couch for a week, unless your friend is also going to the conference. You won't be around enough to be a decent guest, and will be keeping some pretty intolerable hours. I usually fly out Friday or Saturday to visit with friends the weekend before the conference.

Things to Bring

The hotter the interior of California, the colder and soggier San Francisco gets. Two years ago, the conference sold out of sweat shirts. So did the Old Navy near Moscone, and every tourist vendor in town. Early afternoons in May tend to be deceptively perfect, evenings are cool and damp.

Moscone's floors are concrete or industrial carpet over concrete. I've got goat hoofs from a lifetime of hiking, but if your feet are tender you will want cushy shoes. My one caveat to traveling light: I take two pairs of shoes, and alternate days. I don't normally wear shoes for 18 hours straight.

JavaOne is guys to two sigmas, and coders to three sigmas, so you can pretty much come as you are. I shave most days out of habit, but I doubt anyone cares.

Every employer I've ever had has given me at least one box of 1000 business cards per job, promotion and move -- maybe 13000 cards so far. I have no idea why. The only place I've ever run out is JavaOne. Bring at least 50, maybe 200.

Gear for me: Notebook, 100 business cards, one lap top (my old power-savvy Mac G4, not the employer's twelve-pounder), two batteries for lap top, cell phone, charger, empty water bottle (filled after clearing airport security), book, shaving kit, swimsuit, towel, fleece jacket, two pairs shoes, umbrella, carry-on backpack, checked duffle bag, two pairs kakis, good shirt for my presentation, enough underwear, socks and T shirts.

Carried into the conference: Notebook, business cards, lap top, cell phone, backpack, umbrella, fleece jacket, water bottle, one set clothes.

Yes, I bring a cell phone. No, I don't own it. It belongs to work.

The Weekend and Monday

Things happen over the weekend and Monday. Aside from frantic conference preparation, various expert groups and community groups meet Saturday and Sunday. Monday afternoon and evening were especially good to meet internet friends face to face. Try not to be shocked when we don't look like our internet pictures, or are especially short or tall, or no longer have a two-foot pony tail.

JavaOne University runs during this time. It sounds like a good way for beginners to come up to speed on some new technology. This year, on Monday JavaOne is sponsoring CommunityOne. The unconferencelooks most interesting to me. I'm not sure what to expect from it, but I hope it brings back some of the spontaneity of earlier JavaOnes.

Swag

When you register, they send you into the basement to pick up conference swag. Expect a lap top case of some kind, so you don't need to bring extra luggage. Some of the things in the bag are useful for life on the road, like the miniature mouse on a spring cable. Typically there's a tee shirt or two, and a lot of developer CDs. There's more swag on the pavilion floor and the alumni room, some of it entertainingly weird. It's been a bit more practical in recent years -- no Pladough or dancing cows, but Google gave away good messenger bags two years ago if you solved their puzzles. I used to harvest swag in the pavilion, but now I only take stuff I plan to use.

I'll miss the tee shirt hurling contest this year. In past years, it was just luck -- good luck -- that they missed me. A shirt from some kind of home made air cannon nearly parted my hair.

Food

JavaOne is a test of endurance for me. I typically loose five pounds during the week. At the conference I sleep about six hours a night and cover about four miles during the day. Your milage may vary.

I drink a lot of water during the conference. Staying hydrated lets me stay up longer and helps me hold my beer and salty hours devours. It won't save my voice; I'll be hoarse by Wednesday night.

Most of the alumni perks have to do with food. The fire side chat dinner is more of a tasty snack than a dinner (too many alumni). The alumni room gets its own supply of muffins, pastries, juice and coffee in the mornings. The pavilion dinner is good eating. Java community receptions are appetizers. This year it looks like we've got a longer lunch break. Lunch varies in quality day to day. More interesting things are happening, so I'll probably short lunch. It's fine to grab something and take it with you to a keynote or technical session, provided you can eat it quietly and neatly. Burritos, not tacos, for example

I try to use dinner to meet up with friends at the conference I don't see often enough. San Francisco has great restaurants. I'll typically let the friends pick out a place I've never been before. The dinner break is pretty short, so I try and stay within a few blocks of the conference.

Plan the Day

Start each day with a plan: Which sessions and BoFs will you attend? Go to a keynote or skip? What to do in the pavilion? Enter a contest? When will you touch base with work? When will you call your family?

Moscone is big. If its your first time there, take some time to study the map and know where things are. Don't waste time drifting about inside this combination conference hall and earthquake shelter.

Keynotes

First, the keynotes are about hype. They rev up the crowd for the day. Second, they're about information. You can learn a lot about what's coming in the next year just from the keynote. Last year Sun announced they were going to make Java open-source. The company has put tremendous work into that effort this year; surely they put off doing something else interesting (IMHO, probably more interesting). There's a lot of hype between the content. I enjoy the muffin, crank through work emails, and plan the day either before or at the keynote. If you need to skip something to catch up on work, skip the keynote and watch it later. Videos of the keynote are the first thing on line.

Technical Sessions and BoFs

There are ~400 talks at JavaOne. You only get to go to about 30 of them. You have to set priorities. I pick talks to ask questions for work or for my open source projects. I go to talks on topics about which I know very little. I go to talks on things I can't read about. Take the time to get some exposure to something you don't get to do at your current job.

Be aware that some talks have great abstracts, but will be weak. Other talks will be brilliant, but cover things you already know. Some sales pitch always sneaks through the selection process. Keep in mind that you are at JavaOne for your own benefit. You're at the talk to learn stuff. Have a backup plan of what to do if the talk is a bust. Slip quietly out the back, and head to another talk or to the pavilion.

I find Schedule Builder inadequate for planning my time at the conference. The one available a few years back let you pick out all the sessions you were interested in, then resolve all of the conflicts. The new version only lets you pick one session for each time slot and forces you to resolve conflicts immediately. (Circle with a red ! in it means you've got a conflict already.) The time-out on the application is far too short to read the descriptions of two talks and deciding between them. Schedule Builder tells you there are some repeated sessions (two arrows forming a circle), but doesn't tell you when the repeated session is. I resolve this problem by killing trees; I take notes on paper. My plan for the day, and the backup plan are on paper.

In the past, they'd used the food room and spare meeting halls as overflow rooms, piping in sound and video. It wasn't that good a fix -- you couldn't participate in the Q&A. Last year they started making conference attendees use Schedule Builder to sign up for sessions in advance, to guarantee seats to people who wanted to see specific talks. This year the conference planners are selling that as a feature that's gotten the appropriate response in the commentary. The obvious new problem is that some things (especially labs) filled up within 24 hours of their turning on Schedule Builder because they didn't get a big enough room. The less obvious problem with schedule builder is that it takes away the spontaneity and some of the social aspects of the conference. There are few conversations while waiting in line to get into a room other than, "waiting in line is lame." In the past, I've had great conversations in the overflow room.

Last year there was talk of a contest to improve Schedule Builder for 2007, but it didn't pan out. I think it'd make a good lab. Let the JSF people square off vs. the rich client crowd and give us something that benefits the conference.

Hands-on Labs

I'm not a great fan of labs at the conference. Other people love them. For me, two hours is enough time to learn a bit about something, but if I try to do something with it I spend too much time typing and not enough time learning. Some labs go more smoothly than others. I actually get a great deal out of sitting next to someone else and following along. The lab materials tend to be very high quality and are worth picking up to read on the flight home. I'll make two exceptions: Labs that let us play with specific hardware we don't always have access to are really worth it. And if I've got a specific question to ask about something, I'll duck into a lab session to ask it. The crews running the labs often have a different perspective and experience than the people who develop the technology.

I'm not sure what to make of JavaOne Campyet.

Exhibit Pavilion

Getting what you want out of the Pavilion takes some skill and daring, but can really pay off.

First, the pavilion caters to those of you looking for swag. I may never need to buy another tee shirt in my life. (Booths tend to run out of huge shirts first, so plan accordingly if you're big. The smaller you are, the less rush.) I don't pick up anything I can't use, unless it's really odd -- a foam rubber dancing cow, for example. Bring a messenger bag or a backpack for the swag.

Second, the pavilion is the best place to meet experts and talk to them face-to-face. Come with specific questions in hand, or just listen in on the conversations if you want to learn something. Sun's booth farm tends to be packed, but packed with the right people. If you've got a question, or a list of questions, there's probably someone there who can answer it. They can suggest workarounds for bugs, and give advice on general approaches. You have to work to find the right person; the people who work the Sun booths are a sampling of Sun employees. They do tend to be just one or two hops from the right person. The vendor booths are a little less useful -- they are trying to sell you something. Vendors are much more likely to have a single layer of flappers to hand out swag. Serious technical questions cut right through to the technical people.

A famous JEE application server's flapper: "Did you name your company that [BAE Systems] so that people would confuse you with us?"

The java.net people run a miniconferencein the pavilion, where different people talk about their open source work. The speakers are enthusiastic. The topics are solid, and are sometimes more daring than the official JavaOne talks. They also have a great couch.

The pavilion floor runs at about 80 decibels; it's OK to lean toward someone and put your ear next to his mouth to hear. That's why I'm hoarse by Wednesday.

Contests

The contest topics tend to be very good, but take away from time for other things. I'd rather the contest coding happened before the conference. (I focus when I code; I hit a groove and drop out of the world for a while. I'd be missing something interesting at JavaOne.)

Teams of people win contests. Teams of people with good process and simple ideas win contests. For example, the crew who won last year's slot car challenge had a very simple program, and a process for waiting in line to get lots of track time.

Organize the Follow-up

I take notes. I collect contacts to follow up with. I capture new ideas during the conference. Afterwards, I have a few weeks' follow-up work. The important thing here is to keep enough of a record so that you know why you want to follow up with someone. I typically write a note on the back of people's business cards to remind myself why I have it.

I finish the conference with a mountain of ideas for new projects. One of the jobs on the plane ride home is to sort the projects by work/hobby, easy/hard, smart/crazy, then set priorities for what to do first. That feeds back into the follow-up and sets the focus for my next few months of hobby code.

Burn Out

I don't recover from red-eye flights well anymore, so I stay on an extra night and go home Saturday. I'll typically spend Friday evening not thinking too hard. I try to meet up with friends from the conference for dinner, maybe a movie, Friday night. Six hours' quiet coding on the plane, a two mile walk and I'm home.

I finally found some time to get back to the generics saga. A comment in the feature request says that wildcards should help make Semirings easier to use. This article describes what I was able to do with them, and the effect they have on the code someone using JDigraph would write. It was a bit of a disappointment. I could only use wildcards where my code did not use the type parameters. That limitation prevented them from helping with all but the simplest examples, and only eliminated the easy-to-grok type parameters. I could eliminate the type parameters a developer would understand best -- Nodes and Edges -- but left him trying to puzzle through monsters like MutableFastNodeOverlayDigraph<TestBean, Boolean, IndexedMutableSimpleDigraph<TestBean>> to make the system hang together.

Mads Torgersen & company's JOT article gave me a good understanding of how wildcards worked. It might just be a matter of repetition, but I think I really needed to read what the compiler was up to to understand wildcard's use and limits. The core idea: if my code doesn't use a particular type parameter, I can leave it out. If my code has no warnings (and no dodgy @SuppressWarnings("unchecked") annotations), I can drop the type parameter and use a wildcard instead. The compiler knows enough to put the pieces together for me, or will tell me when I've blown it.

After some initial experiments, I set up a CVS branch (Subversion puts CVS to shame. Why'd we wait twenty years?) and cleaned out all of the warnings. That was no small undertaking; JDigraph's working code had eight warnings when I started, but the test code had about five hundred, mostly generics-related. My experiments had shown that warnings and wildcards don't mix at all. Cleaning out warnings ate my hobby code time for much of January.

Algorithms First

I started working from the end of the dependency chain forward. I started with the Floyd-Warshall algorithm example I'd used in the bug report. Floyd-Warshall is about the simplest graph minimization algorithm there is. Initially, the class declaration looked like this:

public class FloydWarshall<Node,
                                Edge,
                                Label,
                                BaseDigraph extends IndexedDigraph<Node,Edge>,
                                LabelDigraph extends IndexedMutableOverlayDigraph<Node,Label,Edge,BaseDigraph>,
                                SRing extends Semiring<Node,Edge,Label,BaseDigraph,LabelDigraph>>



and to use it in a test case, I had to type all of this:

        FloydWarshall<TestBean,
                      SimpleDigraph.SimpleEdge,
                      MostProbablePathLabel,
                      IndexedMutableSimpleDigraph<TestBean>,
                      NextStepDigraph<TestBean,MostProbablePathLabel,SimpleDigraph.SimpleEdge,
                             IndexedMutableSimpleDigraph<TestBean>>,
                      MostProbablePathSemiring<TestBean,
                                        SimpleDigraph.SimpleEdge,
                                        IndexedMutableSimpleDigraph<TestBean>>> floydWarshall 
            = new FloydWarshall<TestBean,
                                  SimpleDigraph.SimpleEdge,
                                  MostProbablePathLabel,
                                  IndexedMutableSimpleDigraph<TestBean>,
                                  NextStepDigraph<TestBean,MostProbablePathLabel,SimpleDigraph.SimpleEdge,
                                      IndexedMutableSimpleDigraph<TestBean>>,
                                  MostProbablePathSemiring<TestBean,
                                                    SimpleDigraph.SimpleEdge,
                                                    IndexedMutableSimpleDigraph<TestBean>>>();



Floyd-Warshall should be very well suited for wildcards. It consists of three nested for loops to walk the Nodes. The loops don't use Node, Edge or Label. The algorithm just marches along the indices, calling the Semiring's relax() method.

Using wildcards, I was able to reduce the class declaration to:

public class FloydWarshall<BaseDigraph extends IndexedDigraph<?,?>,LabelDigraph extends IndexedMutableOverlayDigraph<?,?,?,BaseDigraph>,SRing extends Semiring<?,?,?,BaseDigraph,LabelDigraph>>

To use it in the test code, I had two fewer type parameters to specify:

FloydWarshall<IndexedMutableSimpleDigraph<TestBean>,
                      NextStepDigraph<TestBean,MostProbablePathLabel,SimpleDigraph.SimpleEdge,
                                IndexedMutableSimpleDigraph<TestBean>>,
                      MostProbablePathSemiring<TestBean,
                                        SimpleDigraph.SimpleEdge,
                                        IndexedMutableSimpleDigraph<TestBean>>> floydWarshall
    = new FloydWarshall<IndexedMutableSimpleDigraph<TestBean>,
                          NextStepDigraph<TestBean,MostProbablePathLabel,SimpleDigraph.SimpleEdge,
                              IndexedMutableSimpleDigraph<TestBean>>,
                          MostProbablePathSemiring<TestBean,
                                            SimpleDigraph.SimpleEdge,
                                            IndexedMutableSimpleDigraph<TestBean>>>();



The Floyd-Warshall algorithm is pretty slow: O(nodes^3), not NP hard but like waiting for Santa. It solves general graph minimization problems, without taking advantage of any knowledge about the problem. Sometimes that's the best you can do.

However, in many problems a Semiring's summary() operator makes a choice between two paths. For those problems, Dijkstra's algorithm can do better -- O(edges + nodes log (nodes)). In many real-world applications, O(edges) ~ O(nodes), so Dijkstra will run in O(nodes log (nodes)) time. In my implementation Dijkstra's algorithm keeps a Fibonacci heap of Labels; the Labels can't be wildcarded away. My A* implementation hit a similar issue. The best I could do for these algorithms still had five type parameters.

public class Dijkstra<Label,
                        BaseDigraph extends IndexedDigraph<?,?>,
                        LabelDigraph extends IndexedMutableOverlayDigraph<?,Label,?,BaseDigraph>,
                        Comp extends HeapComparator<Label>,
                        SRing extends PathSemiring<?,?,Label,BaseDigraph,LabelDigraph,Comp>>



I kept those changes in JDigraph's main branch.

Semirings

I was able remove Nodes and Edges from the Semiring interface. but I wasn't really happy about removing either. It's removing domain information from the Semiring, stripping away part of its mathematical definition. The code survives it because the algorithms, especially the relax method, use int indices to work with arrays inside IndexedDigraphs.

It worked for the Semiring interface and an abstract parent class, but then hit a dead end. Reachability is possibly the simplest Semiring. Its operators can determine if one Node can be reached from another. In the createLabelDigraph() method, it constructs a MutableFastNodeOverlayDigraph in which to build the solution:

    /**
Creates an initialized LabelDigraph. The LabelDigraph has BaseDigraph's 
Nodes, initial Labels set to values from getInitialLabel().
    */
    public MutableFastNodeOverlayDigraph<?,Boolean,Edge,BaseDigraph> createLabelDigraph(BaseDigraph baseDigraph)
    {
        MutableFastNodeOverlayDigraph<?,Boolean,Edge,BaseDigraph> result 
            = new MutableFastNodeOverlayDigraph<?,Boolean,Edge,BaseDigraph>(baseDigraph);



But you can't use a wildcard in a constructor. One fix would be to change MutableFastNodeOverlayDigraph so that the Node doesn't matter there. Unfortunately, MutableFastNodeOverlayDigraph implements Digraph<Node,Edge>, and implements demands real type specifiers, too. Another fix would be to pass in the MutableFastNodeOverlayDigraph in which to build the solution. That change would move this step from encapsulated inside the Semiring to outside in the developer's code. It misses the point.

However, I found a small step forward. In OverlayDigraph, the underlying Digraph's Edge's, UnderEdge, really shouldn't matter. For Reachability, that's fine. Removing Edge from Semiring and Reachability made the test code using FloydWarshall look like this:

        Reachability<TestBean,IndexedMutableSimpleDigraph<TestBean>> reachabilitySemiring 
           = new Reachability<TestBean,IndexedMutableSimpleDigraph<TestBean>>();
        
        FloydWarshall<IndexedMutableSimpleDigraph<TestBean>,
                        MutableFastNodeOverlayDigraph<TestBean,Boolean,IndexedMutableSimpleDigraph<TestBean>>,
                        Reachability<TestBean,IndexedMutableSimpleDigraph<TestBean>>> floydWarshall 
                        = new FloydWarshall<IndexedMutableSimpleDigraph<TestBean>,
                                            MutableFastNodeOverlayDigraph<TestBean,Boolean,IndexedMutableSimpleDigraph<TestBean>>,
                                            Reachability<TestBean,IndexedMutableSimpleDigraph<TestBean>>>();
        
        IndexedMutableDigraph<TestBean,Boolean> labels = floydWarshall.computeLabels(reachabilitySemiring,baseDigraph);



When I worked on slightly more complex semirings, where Labels are Path<Node,Edge>s, the Edges showed up again. LeastPathSemiring and MostProbablePathSemiring need to specify Nodes and Edges for their versions createLabelDigraph(). If that constructor hadn't caused problems, I'd have had a hard time measuring the cost to cross an Edge from one Node to another using a PathMeter that didn't know the type to use for Node or Edge.

The best I was able to do to use AStar using LeastPathSemiring using wildcards was this:

        LeastPathSemiring<TestBean,SimpleDigraph.SimpleEdge,IndexedMutableSimpleDigraph<TestBean>> shortestLabelsSemiring 
            = new LeastPathSemiring<TestBean,SimpleDigraph.SimpleEdge,IndexedMutableSimpleDigraph<TestBean>>(new TestPathMeter());
        AStar<LeastPathLabel,
                      IndexedMutableSimpleDigraph<TestBean>,
                      NextStepDigraph<TestBean,LeastPathLabel,SimpleDigraph.SimpleEdge,IndexedMutableSimpleDigraph<TestBean>>,
                      LeastPathComparator,
                      LeastPathSemiring<TestBean,SimpleDigraph.SimpleEdge,IndexedMutableSimpleDigraph<TestBean>>> aStar 
                        = new AStar<LeastPathLabel,
                                    IndexedMutableSimpleDigraph<TestBean>,
                                    NextStepDigraph<TestBean,LeastPathLabel,SimpleDigraph.SimpleEdge,IndexedMutableSimpleDigraph<TestBean>>,
                                    LeastPathComparator,
                                    LeastPathSemiring<TestBean,SimpleDigraph.SimpleEdge,IndexedMutableSimpleDigraph<TestBean>>>(new LeastPathEstimator());



Wildcards just don't make that much difference when a developer tries to use the generic algorithms. I really want a way to encapsulate all the complexity inside the Semiring, then pull it out inside the algorithm code. I want something like this.

There is some hope, though. Peter Ah

The pronouns we use when we address computers and imagine them addressing us hides some profound insight. I haven't pinned down exactly what. Wikipedia has a nice grammar description on first, second and third person. I think the way we use these implied points of view in computing is really strange and not very consistent. Someone could write a decent master's thesis drawing some conclusions on these inconsistencies.

First Person

I've always found "I," "me," and "my" disconcerting in computing. Some of that unease is from growing up on Unix systems. /home/dwalend feels natural to me. Windows' "My Documents" clashes. The computer screen is referring to documents I wrote, so the file system's word is recognizing that I am part of the narrative. The computer's language implies that it wants to become an extension of me, as part of my internal narrative. Microsoft Visual Basic apparently has a My keyword to invite a programmer in. The computer drops into passive voice to avoid defining itself. (My coworkers have endured many of my tirades on how passive voice always hides something important.) "My Documents not found." I know where I left them. Not found by whom?

I've observed a very different use of first person prevalent in computer programming examples. Example writers often use "my" as a prefix in a reference name to imply unshared possession -- a filled UML diamond aggregate. Instead of the computer trying to merge with my subconscious, a class grows its own ego and takes possession of its parts.

http://www.jgraph.com/example/Tutorial.java

    public static JFrame myFrame;

    public static void main(String[] args) {
      myFrame = new JFrame("Tutorial");
      myFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
      JGraph graph = new Tutorial();
      myFrame.getContentPane().add(graph);
      myFrame.setSize(520, 390);
      myFrame.show();


However, "My" as a prefix to a class name implies that some person -- the writer or the reader -- possesses the class. "MyFrame" invites a third person into the dialog with my computer and me. It's like Who's on First but not funny.

Second Person

I've found myself using second person, -- "you" and "your" -- in my examples to refer to the person reading the examples: https://jdigraph.dev.java.net/servlets/ReadMsg?list=users&msgNo=2

    MutableDigraph yourDigraph = new MutableHashDigraph();
    
    yourDigraph.addNode(nodeA);
    yourDigraph.addNode(nodeB);
    yourDigraph.addEdge 
    (nodeA,nodeB,objectContainingDataAssociatedWithRelationship);


Second person works for me as a style here. It extends a dialog that I have with another person; we both know what "you" refers to.

However, second-person dialog between people and computers have mixed results at best. Clippy was the definitive failure and reduced to a target forspoofs. Bob the dog still digs pixels on the windows machine at work when I search. Ms. Dewey seems to focus more on, well, Ms. Dewey than getting anything useful done. She's entertaining on her own, but would you really want her to stay for dinner? Wildfire was better. She did something useful and told great jokes, but never quite caught on. Creating a first-person/second person dialog with a computer isn't so much science fiction as fiscal fiction. I did a little work with voice interaction in graduate school; I built a voice-interactive version of Cluedo. Voice interaction isn't that hard to do, provided your software can limit what a person might say. When people speak, they really want to talk to something with a sense of self.

Third Person

I grew up with unix, which is all about third person; /home/dwalend is where the heart is. The computer doesn't talk about itself as an entity beyond its host name, and every reference has to be third-person. The words that appear on the computer screen don't show a pretense of self-awareness. Java's "this" keyword is also third-person, referring to "this instance of a class." I find third-person the least jarring for our current window-interface mouse-pointer interactions. A dialog with a machine that has no sense of self is really uncomfortable. I've done a little work with voice user interfaces; third-person spoken dialog feels even more unnatural.

Wrapping Up

Microsoft created or paid for "My Documents," Visual Basic, Clippy, Bob and Ms. Dewey. I'm not sure what conclusions to draw about one company producing so many examples that make me cringe. I'll admire their bold public attempts, and leave the rest unsaid.

I'm not yet convinced that a computer with a sense of self is a useful thing. The computers we have now are mostly tools or for entertainment. The market place so far has rejected computers which pretend to have any sense of self. However, the examples I have sited for the most part had very assertive personalities relative to the help they provide; Clippy always injects himself at the least opportune times.

adamTaglet -- Architecture Driven Assisted Modeling Taglet -- is a taglet that generates class diagrams from a custom JavaDoc tag. The tag holds a list of the leaf classes for the diagram and a list of forbidden classes. adamTaglet walks the class and interface inheritance structures from the leaves, building up a directed graph. The taglet stops walking the graph whenever it reaches an uninteresting forbidden class like Object or Serializable. The taglet then feeds the resulting directed graph toGraphViz to generate a picture of the class relationships and a client-side image map. Finally, adamTaglet replaces the custom tag with the picture and image map. Click on a class or interface, and go to its JavaDoc.

I had to use the verboten com.sun API to build up the URLs for the links in ClassDiagramVisitor. I found it in com.sun.tools.doclets.internal.toolkit.Configuration after digging through a lot of the javadoc source code.

    /**
    @return a URL String to link to node.
    */
    private String urlForNode(Class node)
    {
        //If the node is in the same package
        if(node.getPackage().equals(packag))
        {
            return shortNameOfClass(node) + ".html";
        }
        else
        {
            //If the node is from some other JavaDoc repository
            int nameSize = node.getPackage().getName().length();
            String shortClassName = node.getName().substring(nameSize+1);

            String result = configuration.extern.getExternalLink(node.getPackage().getName(),"relativePath",shortClassName);
            
            //If the node is from some other package in this repository
            if(result==null)
            {
                result = pathToClass(node);
            }
            return result+".html";
        }
    }



The API I'm calling is pretty rickety. That"relativePath" magic String worries me. Worse, I'm interpreting a null return value, which is always a little scary. If someone changes that getExternalLink() method to return "No Link, Dude" then adamTaglet will make broken links. Plus there's my lingering fear that some developer at Sun will need to change Configuration's API to at least not expose the extern member variable. That'd lead to different compiles for different JDKs

I don't mean to be too discouragingly critical of the javadoc tool's code base, but I do need to warn others about what they'll find diving in. The javadoc code has characteristics of code written by a series of developers over time -- new strata of API wrapped over old layers, naming and style limited to certain strata, and magic Strings to bind the layers together. I suspect these developers didn't know each other and were afraid of breaking existing code by making it simpler. The resulting code is pretty much opaque. I had to rely on an IDE (Netbeans this time) to stumble through it. (At some point I'll write a scathing blog about our dependence on IDEs, but I haven't the courage just now.)

Taglets were new in JDK 5, but don't seem to have found a lot of use. To me it looks like most people doing interesting extensions to JavaDoc are using custom Doclets. There aren't any big changes for JavaDoc in JDK 6. JSR-260 promised a revamp of the javadoc tool and tags in 2004, but missed the deadline for JDK6. I wonder if others are waiting to see what this JSR delivers before starting new large projects.

Javadoc itself comes from the dawn of Java in 1995. JavaDoc has been remarkably useful over the years, and remarkably unchanging. The existence of JSR-260 suggests that we as a community may be outgrowing the existing javadoc tool. JDK 6 does provide the new compiler API, which might make a good starting point for a new javadoc tool.

Now that Java is going open source the Java community might finally be ready for cross-linked javadoc more sophisticated than -link. (Accept that Sun's lawyers are part of our community even if they seem to get underfoot more often than not.) The Java Tools community has talked about how to show which of our projects use each other's work. Hosting cross-linked javadoc would be a great pillar to add to java.net.

In my ears: Monteverdi Vespers of 1610

Back in May I blogged about simplifying my generics code with dot accessors to the type parameters, to simplify code that currently looks like this:

public class FloydWarshall<Node,
                            Edge,
                            Label,
                            BaseDigraph extends IndexedDigraph<Node,Edge>,
                            LabelDigraph extends IndexedMutableOverlayDigraph<Node,Label,Edge,BaseDigraph>,
                            SRing extends Semiring<Node,Edge,Label,BaseDigraph,LabelDigraph>>


into something more like this:

public class FloydWarshall<SRing extends Semiring>


The request for enhancement made it into Sun's database. Seven of you voted for it in as many hours. Thanks.

I also had what I think was a revealing back-and-forth with a Sun engineer. I'm a bit disappointed it ended as quickly as it did, but I hope my suggestion planted some seeds for further thought about how the Generics language features can mature. I think the dialog in the RFE makes interesting reading.

Back in May I blogged about simplifying my generics code with dot accessors to the type parameters, to simplify code that currently looks like this:

public class FloydWarshall<Node,
                            Edge,
                            Label,
                            BaseDigraph extends IndexedDigraph<Node,Edge>,
                            LabelDigraph extends IndexedMutableOverlayDigraph<Node,Label,Edge,BaseDigraph>,
                            SRing extends Semiring<Node,Edge,Label,BaseDigraph,LabelDigraph>>


into something more like this:

public class FloydWarshall<SRing extends Semiring>


The request for enhancement made it into Sun's database. That whooshing sound may not be a windmill after all.

If you can spare a bug vote, please vote for this RFE. Judging by the evaluation from Sun's engineer, this RFE needs some votes raising it up so the Sun language giants might spot it.

The RFE will also need some good rational discussion. I held back my irrelevant knee-jerk reaction -- "Didn't we all out-grow one-letter-variables when we traded our PETs for C-64s?" I could send a link to an old blog, but even that might distract from the cause. Please keep in mind that we want these folks to do us a favor. I'm working on a response that frames the RFE as "encapsulation vs. exposure," to dispel the "inference vs. explicitness" suggestion.

About a month ago there was a flurry of blogs about LINQ (Language Integrated Query. I'm not sure what the N stands for.) These articles are a very good description of LINQ, which looks like a useful add-on to the Microsoft CLR family of languages. LINQ would let me wind through layers of getters from the top level, to pluck out just the objects I care about. That's a nice way of saying I feel our usual C#-induced draw of language feature envy.

Deja vu followed that feeling of envy. Two years ago I was checking on what some old coworkers from VNP were doing. Drew Davidson and Luke Blanshard had created OGNL, the Object-Graph Navigation Language. OGNL is "an expression language for getting and setting properties of Java objects. You use the same expression for both getting and setting the value of a property." I've been looking for a nail for the OGNL hammer since I saw it. I think the main reason I haven't used OGNL is that writing a big chain of accessors is straightforward (for simple queries), and custom methods to select and build up collections of things inside of classes I control is not hard (for more interesting queries). So unless I force myself to do the (easy) overhead work of adding OGNL to a project, I'll continue to add the (not hard to write) one-off code to get the feature.

OGNL was inspired by frustration with commons beanutils' PropertyUtils; Java developers have been sharing this sort of thing for at least five years. We've no reason to feel envy. We just need to get some clue of what we already have, filter out the better ideas, share them and eventually convince the JCP to codify them.

I'll also have to regain my waning faith in Sun's process to evolve Java. We've still no bug parade number for my generics RFE.

I've finally added message selector support for Topics in SomnifugiJMS, a single-process implementation of a Java Messaging Service. It's SomnifugiJMS's 14th alpha release. (Message selectors for Queues will be more difficult, but is the last piece keeping SomnifugiJMS in alpha releases. I think I can get it to work with j.u.c.Conditions and some extra work, but haven't had the time to design it yet.)

At JavaOne, I met up with Linda Schneider, one of the key people working on https://mq.dev.java.net/, at JavaOne. She sent me a pointer to the message selector implementation in the JMS reference implementation. The JMS RI just switched to CDDL, which isa file-based license, so I picked up their code, edited it to play nice with my own, and now have message selectors for Topics. They even have a good set of tests in a main method; the JMS RI predates JUnit by a year or so.

The only snag I hit was that I use member fields in SomniMessage to hold information that every message should hold, but the JMS RI Selector API expects a Map.

public class SomniMessage 
    implements Message
{
...
    private String messageID;
    private long timeStamp=0;
    private String correlationID;
    private Destination replyTo;
    private Destination destination;
    private int deliveryMode;
    private long expirationTime;
    private int priority=0;
    private int producerCount=Integer.MIN_VALUE;
    private String producerConnectionClientID;
    private boolean readOnly=false;
    private boolean redelivered=false;
...
}



    public synchronized boolean match(Map properties, Map fields)
        throws SelectorFormatException
    { ... }



My first thought was to recode the JMS RI Selector to use javax.jms.Message-specific getters. That sounded tedious; I'm not familiar with the code, and am not overly interested in writing or modifying an SQL92 parser. Linda emailed, "[Selector.java is] huge because there is a lot of different items to parse," which seems to say it all. My second thought was to recode SomniMessage to use a Map to store this sort of information. In SomnifugiJMS these fields always have some value, so a Map just seems wrong. SomnifugiJMS is supposed to be faster and simpler than database JMS implementations, especially for small-scale work (which rarely needs message selectors). I didn't want to loose that simplicity to get the message selectors working.

I pondered a bit and came up with a third option: use reflection to implement a Map that calls getters when given a field name as a key:

class ReflectingMap
    implements Map
{
    private static final String GETTERPREFIX = "get";
    
    private Object hasGetters;
    
    ReflectingMap(Object hasGetters)
    {
        this.hasGetters = hasGetters;
    }
    
    private String getterNameForKey(String key)
    {
        return GETTERPREFIX+key;
    }
    
    public boolean containsKey(Object key)
    {
        if(!(key instanceof String))
        {
            return false;
        }
        try
        {
            hasGetters.getClass().getMethod(getterNameForKey((String)key));
            return true;
        }
        catch(NoSuchMethodException nsme)
        {
            return false;
        }
    }

    public Object get(Object key)
    {
        if(!(key instanceof String))
        {
            return null;
        }
        try
        {
            Method method = hasGetters.getClass().getMethod(getterNameForKey((String)key));
            return method.invoke(hasGetters);
        }
        catch(NoSuchMethodException nsme)
        {
            return null;
        }
        catch(IllegalAccessException iae)
        {
            throw new SomniRuntimeException(iae);
        }
        catch(InvocationTargetException ite)
        {
            throw new SomniRuntimeException(ite);
        }
    }
    
    public Object put(String key,Object value)
    {
        throw new UnsupportedOperationException("I haven't done this yet. Maybe someday when I think about setters.");
    }
  
    public Object remove(Object key)
    {
        throw new UnsupportedOperationException("This will never work. What are you doing?");
    }

    //All other methods throw new UnsupportedOperationException("I haven't done this yet.");
}



I could implement a general purpose put() method by slipping another Map inside this one, and delegating put()s and get()s to that map for keys that hasGetters does not have. I can't implement the whole Map interface this way because there's no way to remove arbitrary keys. What would removing a field even mean?

I think that's an interesting piece of code, especially given the talk of invokeDynamic's evil partner, "changing the members of an object," at a JavaOne talk. It's an interesting choice for sometimes-optional fields when designing data structures.

However, "Interesting" might mean that it'll be a source of horrible suffering.

First, code that works this way will have to haveif(containsKey()) or if(yourRef==null)forks everywhere, will have to handle NullPointerExceptions, or (worst and most likely) will spew NullPointerExceptions. I'll predict the same problems as I've seen using BeanShell's voids. Variables that don't exist in the a BeanShell name space are void (instead of null or some default value). Developers sometimes should check if(hisRef==void), but usually don't.

Second, the keys are Strings, so there's no great way to predict what might be a key. For message selectors, I know from the JMS specification that the keys will be a small, defined set of Strings that happen to match method names. Perhaps an Enum of possible keys would better define the universe of possible optional members, but that shifts back toward the unsatisfying "this member hasn't been set yet" pattern for accessors. I can live with these problems in my own code, but I want to know these land mines are out there, especially after a decade of trying to avoid using nulls as magic error codes.

In my ears: Beethoven's Missa Solimnis. Wow.

I gave a talk on the trouble I'm having using complex generics code at the java.net community pavilion at JavaOne. It was a great experience. About a dozen people dropped by to listen; most settled in for the whole twenty minutes. Chris Adamson and the java.net team put together a great facility, with couches, a video screen for slides and -- most important -- an amplification system. My voice doesn't carry well and I was a bit worried that people would not be able to hear me over the 85 decibel din. Inside my head I sound like James Earl Jones. During the talk I thought I was shrieking like a punk rocker. Instead I sounded like a deranged muppet. Chris recorded it for posterity. The amazing thing about the pod-cast is the lack of background noise.

I've linked to the slides and the pod-cast, but I think the original blog entry is more clear than me yelling at a slide projector. I had to dumb down the code just to make it fit on slides.

I've heard no word yet from Sun about the RFE. That's long compared to Sun's usual timeliness. My hope is that the person who reviews language RFEs was getting ready for JavaOne, then at JavaOne, and is taking a week or two off to recover. (Shortly after posting that RFE, I asked Sun to post a package-list file for Taglets' JavaDoc. Sun picked up that one after about 18 hours. I've gotten used to quick turn around.) I'll blog again when I've got a public tracking number.

I've got a quixotic streak that needs exercise every so often. It's time. I'm doing a talk on usability problems in generics that I think can be fixed at 2:30 pm on Thursday in the Java Community pavilion at JavaOne. That windmill aught to break a lance.

Just before JavaOne 2005 there was a huge dust-up over generics as people started using them, and trying to teach how they work. Ken Arnold blogged "Generics are a mistake... A design should have a complexity budget to keep its overall complexity under control. Generics are way out of whack." Trashing generics became the season's fashion. Chris Adamson retorted that we were experiencing "Buyer's remorse for Generics." After all, JSR-14 -- the generics proposal -- started in 1999, but did not appear in a release until JDK5 in late 2004. Any of us could play with generics in Java using the gj and pizzac compilers. We had five years to point out problems we found using the syntax. For example, borrowing C++ template syntax would look too much like XML. (If we get direct manipulation of XML, it won't look like XML. I'm not sure that's a bad thing. XML is pretty hard on the eyes. The angle brackets do make writing html about Generics or XML really painful -- &lt;s everywhere.) That's water over the dam at this point. I'd like to start a conversation about finding ways to fix the stumbling blocks we've found, targeted for Dolphin.

I like generics. I use them (and abuse them. See below) in my JDigraph project. I wrote JSDK 1.4 code in a Digraph interface that looked like this:

    public Object addEdge(Object fromNode, Object toNode, Object edge) throws NodeMissingException;


which caused frequent NodeMissingExceptions from calls like

        hisDigraph.addEdge(hisFromNode, hisEdge, hisToNode);


. The compiler couldn't point out that the second argument was supposed to be a node and the third was supposed to be an edge. Autocompletion in IDEs, combined with some developers' habit of grabbing the top argument in the autocomplete list led to NodeMissingExceptions, confusion and frustration. In JDK 5, the code looks like this:

    public Edge addEdge(Node fromNode, Node toNode, Edge edge) throws NodeMissingException;


Just as Keys and Values in Maps are rarely of the same class, Nodes and Edges rarely share an ancestor closer than Object. The compiler helps the developers get things right. My code is much clearer to me (easier to figure out what I did) and much clearer to people trying to use it (fewer agitated developers at my office door).

One of Ken's specific complaints was about the complexity ofEnum<E extends Enum<E>>, the abstract parent of all Java enums. This particular class interacts closely with the compiler, jumping through some high hoops to give us enums of Objects instead of short integers. It's powerful, but impossible to explain without guessing the java compiler developers' intent. The compiler will specify using your enum class for E. So

public enum YourEnums


becomes something like

public class YourEnums
    extends Enum<YourEnum>


so that YourEnums is always an Enum of YourEnums. To define the Enum class, it'd be great if someone at Sun could just code

public abstract class Enum<E is this.getClass()>


and remove the guesswork. It's a corner case. I ran into something similar while I was generifying JDigraph's HasState interface. HasState provides a method to compare two objects with different representations, but share a defining principle interface that defines the interesting part of the state. For example, Digraphs and Subgraphs have the same principle interface, Digraph. Two Subgraphs are not equal if they are Subgraphs of different Digraphs. However, if the Subgraphs have the same nodes connected by the same edges then they have the same interesting internal state.

I'd love to say

public interface HasState<HasS is this.getPrincipleInterface() which better extend HasState<HasS>>


but had to limit the generic type parameters to

public interface HasState<HasS extends HasState<HasS>>


and hope that a developer implementing HasState doesn't do something diabolically mismatched between the type specification and the filling for

        public Class getPrincipleInterface();


The compiler has all the pieces to sort out Enum and HasState puzzles, but there's no way in the language to direct the compiler to do the right thing in the code for the abstract class or the interface. Since this.getPrincipleInterface() andthis.getClass() return hard-coded Classes, maybe something could be done with annotations interacting with generics. However, I think these two examples really are corner cases and don't rate a change in the language. Others might disagree, citing last Summer's venting of the spleens.

In contrast, I do think we need something to help us avoid generics mismatch Hell. I think just having dot access to type parameters' type parameters would work well. It would look and feel about the same as using inner classes.

The JDigraph project is my demonstration of reusable directed graph algorithms. It uses a general purpose graph containers inspired by the Java collections kit, built off of the Digraph interface. Here's the illustration, with examples from JDigraph:

public interface Digraph<Node,Edge>


is fine. You can specify any class to be Node, and any class to be Edge. Generics actually make the code easier to follow, as people stop confusing nodes with edges. (IndexedDigraph is a Digraph with indexed access to the nodes and edges. You'll see that in a few paragraphs.)

Generics mismatch Hell starts innocuously enough. Subgraph is an interface for directed graphs that hold subsets of the nodes and edges of some Supergraph. Someone using Subgraph has to match up the Node and Edge type specifications for a Subgraph with the Supergraph.

public interface Subgraph<Node,Edge,Supergraph extends Digraph<Node,Edge>>
    extends Digraph<Node,Edge>


could be a bit simpler with dot access, because the compiler knows that Supergraph is a Digraph, and has type parameters for Node and Edge.

public interface Subgraph<Supergraph extends Digraph>
    extends Digraph<Supergraph.Node,Supergraph.Edge>


It's not a huge difference, and in fact takes more characters. However, Subgraph's definition could do the work to line up the type parameters. Also, Subgraph would only need one type parameter, not the three required above.

The complexity grows from there. OverlayDigraph is an interface for representing a Digraph that shares nodes with an underlying Digraph, but has its own edges. Think about it like someone laying a transparency sheet over a bunch of dots and connecting the dots her own way with a sharpie.

public interface OverlayDigraph<Node,Edge,UnderEdge,Undergraph extends Digraph<Node,UnderEdge>>
    extends Digraph<Node,Edge>


The compiler knows what Undergraph will be using for Node and Edge. This is just generics mismatch Heck. Someone using this interface has to get the Node and UnderEdge type specs correct. It'd be a lot nicer to be able to take advantage of the compiler's knowledge of Undergraph's type parameters with something like

public interface OverlayDigraph<Edge,Undergraph extends Digraph>
    extends Digraph<Undergraph.Node,Edge>


and use Undergraph.Edge instead of UnderEdge in the code. That's two type parameters to fill in instead of four. (You're about to see IndexedMutableOverlayDigraph, which is an OverlayDigraph with indexed access to the nodes and mutators to add and remove edges.)

Semirings define a set of useful operators for general graph labeling and graph minimization algorithms that work on Digraphs. The theory is right out of CLRS chapter 26.4. Semirings hold identity and annihilator label constants, and extension, summary and relax operators. The extension operator calculates the label to move from one node to another across an edge. The summary operator calculates the label combining two alternative labels. The relax operator calculates a new label by combining the existing label with a new, alternative label. These operators are the basic parts for building the Floyd-Warshall, Dijkstra, Prim's, Johnson's, Bellman-Ford and A* algorithms. I also made Semiring responsible for creating the initial OverlayDigraph of Labels. Encapsulating these things in Semiring has saved me writing the same tricky algorithm code for every new graph minimization problem.

Still with me? Hold on to your synapses. Semiring's declaration looks like this:

public interface Semiring<Node,
                            Edge,
                            Label,
                            BaseDigraph extends IndexedDigraph<Node,Edge>,
                            LabelDigraph extends IndexedMutableOverlayDigraph<Node,Label,Edge,BaseDigraph>>


All five of the type specifiers have to line up correctly to compile code using this vile beast. It's hard for me to tame, and I wrote the thing. This is a big part of why JDigraph only does alpha releases. That's generics mismatch Hell. If the compiler were a bit smarter, Semiring's declaration could just be

public interface Semiring<LabelDigraph extends IndexedMutableOverlayDigraph>


Five type specifiers could collapse to just one! You tell the compiler what to use for LabelDigraph. LabelDigraph knows Undergraph and Label. Undergraph knows Node and Edge. The Semiring interface and the compiler could do all the work to match up the types. Remember, Semiring's reason to exist is to encapsulate all those operators so I can write generic algorithms. Dot access to type parameters would let Semiring encapsulate all the gory details of type parameters, too.

The Floyd-Warshall is about the simplest graph minimization algorithm there is (three nested for() loops). Got any synapses left? Here's what its declaration looks like now:

public class FloydWarshall<Node,
                            Edge,
                            Label,
                            BaseDigraph extends IndexedDigraph<Node,Edge>,
                            LabelDigraph extends IndexedMutableOverlayDigraph<Node,Label,Edge,BaseDigraph>,
                            SRing extends Semiring<Node,Edge,Label,BaseDigraph,LabelDigraph>>


Six! Six type parameters! Ah ha ha! I feel like that vampire from Sesame Street. (Dijkstra's algorithm and AStar need a Comparator, for a total of seven!) Letting dots access type parameters would let Semiring encapsulate all the type complexity, as well as the operators.

public class FloydWarshall<SRing extends Semiring>


Using dots to access contained parameters would be a Get out of Hell free card for this sort of problem. As more people decide to use generics, it's likely to help a lot of people a little bit. It would help JDigraph immensely.

The next step is to figure out where in the community to propose the change. Graham Hamilton's blog on language changes in Dolphin is a bit dated. My own blog and a JavaOne community talk should be a good start.