Web Swinging Blog

Version 2


    Data is everywhere. It is on your computer, on the internet, squirreled away on some corporate server. We have telephone records, class schedules, blogs, personal websites, and our personal favorite websites. There're Flickr, MySpace, and online genealogical tools. This article introduces simple techniques and APIs available in theSwingX-WS project to help you write rich client applications in Java that take advantage of all these data sources for creating Swing mashups, orSmashups.

    Accessing Web Content

    While there are many great sources of online data, many of them do not provide public APIs for accessing that data. When was the last time you used a public API for accessing your bank records? And yet all this data is available, if only we could access it! Well, we can: it's called screen scraping, and it is d-i-r-t-y. Essentially, it means viewing the source of a website, figuring out what information in that page you want, and then parsing out the data.

    There be sharks in them there waters

    Before digging too deeply, I want to point out that relying on screen scraping for data access is extremely perilous. Web site authors might change their websites at any time, breaking production applications. For this reason, it isnot recommended to rely on screen scraping for mission-critical applications. Rather, it is a fun and interesting technique for creating small applications (often that won't be used by anybody beside yourself or your mom). This technique requires that if a website changes and breaks your parsing code, that you'll quickly fix your code and redeploy an updated application.

    However, there is a silver lining. In this article, I talk a lot about screen scraping, but these same techniques can be applied to quick scripting of Swing UIs that use XML as their data sources. Typically, we end up writing a SAX or StAX parser, or maybe a mound of code for parsing a DOM tree. This article introduces techniques that can be used to make parsing of XML files a loteasier.

    Accessing Web Content withURL

    The first step in writing a Smashup is accessing web content. There are several approaches we could take. Let's start with the easiest: java.net.URL. The humble URLclass has been a part of the JDK since version 1.0. For HTTPGET requests, URL is an excellent choice.

     //read and download the page text URL url = new URL("http://www.java.net"); InputStream in = url.openStream(); StringBuffer buffer = new StringBuffer(); byte[] data = new byte[1024]; int length = -1; while ((length = in.read(data)) != -1) { buffer.append(new String(data, 0, length); } System.out.println(buffer); //read an image Image img = ImageIO.read(new URL("http://java.net/images/header_jnet_new.jpg"));

    Clearly, reading page text could benefit from a helper class, but otherwise our faithful old friend URL is a simple, straightforward way to access web content.

    Apache Jakarta HttpClient

    But URL doesn't help at all when it comes to HTTP POST. Enter the Apache Jakarta HttpClient project. HttpClient is an excellent low-level library for working with HTTP servers (or in other words, nearly every server on the planet). Using some low-level constructs such as the HttpClient and HttpMethod classes, you can access web resources via the GET,POST, DELETE, PUT,HEAD, or TRACE HTTP methods. You can also authenticate, deal with proxy servers, manage cookies, manage headers, and manage parameters.

    For fairness in comparison, here is the minimal amount of HttpClient code that equals the functionality of the aboveURL code example:

     //read and download the page text HttpClient client = new HttpClient(); HttpMethod method = new GetMethod("http://www.java.net"); client.executeMethod(method); System.out.println(method.getResponseBodyAsString()); method.releaseConnection(); //read an image method = new GetMethod("http://java.net/images/header_jnet_new.jpg"); client.executeMethod(method); Image img = ImageIO.read(method.getResponseBodyAsStream()); method.releaseConnection();

    This code should be fairly self-explanatory. In some ways this is an improvement over URL:

    • No looping and byte arrays. Yah!
    • If I need to use HTTP POST instead ofGET, all I need to do is create aPostMethod instead of a GetMethod.
    • I can set a flag on GetMethod to automatically follow HTTP redirects.

    There are also some problems. All this configuration of theGetMethod (such as setFollowsRedirects, configuring proxy support, authentication, etc.) is begging for a higher-level framework to abstract away the tedious details of establishing a connection. There is also no support for HTML concepts such as HTML Forms, since HttpClient is, well, about HTTP. It was also more work to download an Image than it was to use good ol' URL.

    Enter SwingX-WS

    The org.jdesktop.httppackage of the SwingX-WS project provides a set of higher-level constructs for interacting with HTTP-based servers. These classes were written for two reasons: first, because HttpClient needs a higher-level framework for simplifying the common case; second, because SwingX-WS shouldn't be tied by API to a third-party library, especially one that is likely to be succeeded by another project in the not-too-distant future. Again, here is the sample code, which achieves the same end result as the previous examples:

     //read and download the page text Session s = new Session(); Response r = s.get("http://www.java.net"); System.out.println(r.getBody()); //read an image r = s.get("http://java.net/images/header_jnet_new.jpg"); Image img = ImageIO.read(r.getBodyAsStream());

    This code introduces two new classes: Sessionand Response.Session represents an HTTP session from the client's perspective. For example, if I were implementing a tabbed web browser, I would have one Session per tab. EachSession maintains its own cookie policy. Each may also support several simultaneous connections. Each Sessionalso maintains its own password/authentication state.

    Session has several convenience methods which make executing a GET, POST, or other request very simple. Note that all these methods block.

    • get(String uri): Executes a GETrequest on the given URI.
    • get(String uri, Parameter... params): Executes aGET request using the specified parameters on the given URI.
    • post(String uri): Executes a POSTrequest on the given URI.
    • post(String uri, Parameter... params): Executes aPOST request using the specified parameters on the given URI.
    • execute(Method method, String uri): Executes the given method on the given URI.
    • execute(Method method, String uri, Parameter... params): Executes the given HTTP method (such asMethod.POST, Method.GET,Method.DELETE, etc.) on the given URI, using the specified parameters.
    A Riddle

    So what happens if the following is executed?

     Session s = new Session(); s.post("http://mycompany.com/servlet?fruit=\"apple\"", new Parameter("color", "green"));

    Answer: what you would hope. The fruit="apple"parameter is actually extracted from the URL and included with the"color" parameter in the body of the post.

    SwingX-WS Requests

    In addition to these convenience methods, you can gain greater control over the HTTP request by using a Requestobject together with the execute(Request req) method of Session.

    Request encapsulates the details of making an HTTP request. It includes a flag for whether to automatically follow redirects, as well as properties for the method to use and the URL to request from. It also provides the API for reading and setting the parameters and headers for the HTTP request. Finally, there are several methods for setting the request body, including:

    • setBody(String)
    • setBody(byte[])
    • setBody(SimpleDocument) (a type of DOM document)
    • setBody(InputStream)

    When a Request is executed, a Responseis produced. In addition to providing access to the response body (as a byte array, String, InputStream, orReader), Response also provides the response status code, headers, and base URL (the base URL that originated this response).

    Asynchronous HTTP Requests

    The Session and associated APIs are very useful, and very powerful. However, although they support multithreaded access, all of the get, post, andexecute methods of the Session classblock. If you were to call these methods from a Swing event handler, the entire GUI would freeze while the call was being made!

    There are many ways to call a blocking method from your Swing code, including using SwingWorker, Spin, or Foxtrot. However, this problem is not unique to Swing. In the world of thin-client web browser applications, this has long been a usability problem. Each request from the web client to the server involved "blocking" the client while a new page was requested from the server. Recently, DHTML has been renamed Ajax, and with this came a whole new paradigm for web programming. Instead of blocking the entire web app while communicating with the server, Ajax allows the developer to execute long running tasks on a background thread by using the XMLHttpRequest JavaScript object. The interesting part of the story is that Ajax applications have essentially the same issues as rich clients: they both need to perform I/O on a background thread.

    In the SwingX-WS project, we recently introduced theXmlHttpRequest class. This class is based on the W3C working draft specification www.w3.org/TR/XMLHttpRequest, and is similar to the XMLHttpRequest object found in web browsers today. Those of you familiar with Ajax, rejoice!

    In addition to enjoying widespread use,XmlHttpRequest is incredibly easy to use:

     final XmlHttpRequest req = new XmlHttpRequest(); req.addOnReadyStateChangedListener(new PropertyChangeListener() { public void propertyChange(PropertyChangeEvent evt) { if (XmlHttpRequest.ReadyState.LOADED == evt.getNewValue()) { //update my Swing GUI here textArea.setText(req.getResponseText()); } } }); req.open(Method.GET, "http://www.java.net"); //called from the Swing event handling code, // this starts the background process req.send();

    First, create an XmlHttpRequest object. Then attach a PropertyChangeListener that will listen to the "readyState" property of the XmlHttpRequest object. The ready state indicates what state theXmlHttpRequest is in. States include:

    • OPEN
    • SENT
    • LOADED

    In this example, I listen for the LOADED ready state, indicating that the data is fully downloaded from the server. I then read the entire text by callinggetResponseText(). Because this is anXmlHttpRequest instance (rather than the superclassAsyncHttpRequest), I can also call thegetResponseXML() method to get a DOM document representing the response.

    There is also the JsonHttpRequest class, which has the getResponseJSON and getResponseMapmethods. These methods parse the response and construct either aJSONObject or a map containing the data.

    The XmlHttpRequest and associated classes make it trivial to access web content on a background thread, and update the Swing UI on the proper event dispatch thread.

    HTML Forms

    Earlier I mentioned that one of the downsides to usingHttpClient directly is that it doesn't support any HTML-isms, such as HTML forms. SwingX-WS contains a package,org.jdesktop.html.form, which does exactly that. It provides support both for parsing HTML forms out of HTML code, and for setting values in that form and submitting it back to the web server!

    The core API is the Forminterface. It defines the model portion of the HTML form element. For example, there is no id orname property in the Form interface, because they have more to do with HTML than the form concept itself.

    The Form interface has methods for retrieving the "action" (the URL to navigate to when the form is submitted) and the HTTP method to use on submit. A Form is composed of a set of Inputs. Each Input represents an HTML <input> element. An Inputhas a name and value. The name property is read only, but the value property is read/write and takes aString.

    HTML forms also support the concept of a group of inputs tied together by name (these are represented as radio buttons in the form), and a combo-box- or list-type selection component. TheRadioInput and Select interfaces are used to represent these constructs.

    Let's see how these pieces fit together. The following code fragment will log in to java.net using your username and password.

     Session s = new Session(); Response r = s.get("https://www.java.net/servlets/TLogin"); Form form = Forms.getFormById(r, "loginform"); form.getInput("loginID").setValue(username); form.getInput("password").setValue(password); r = form.submit(s);

    First I create the session and get the HTML page associated with the java.net login servlet. Next, I call into the Formsutility class to parse an HTML form from the response and create an implementation of the Form interface. The rest simply sets the input values and submits the form. By the way, when the form is submitted, java.net returns a session cookie. You can inspect this cookie in the response. The important point is that this is all handled for you automatically.

    The Forms class contains several utility methods to help you parse out an HTML form from an HTML document:

    • getForm(Response r, String expression): Uses the given XPath expression to find the <form> tag in the HTML document to use in creating a Form.
    • getFormByName(Response r, String name): Creates aForm based on the HTML form with the given name.
    • getFormById(Response r, String id): Creates aForm based on the HTML form with the given ID.
    • getFormByIndex(Response r, int n): creates aForm based on the nth form in the HTML file.

    Note: The Forms methods accept XMLand attempts to use malformed HTML. JTidy is used behind the scenes and attempts to fix the HTML prior to parsing it.

    These classes make it trivial to interact with existing HTML documents containing Forms.

    Simplified DOM and XPath

    I've shown how to very easily get access to web content including both XML and HTML documents. But actually accessing the content of these document types is no easy matter! HTML is often malformed, and even when using tools such as TagSoup or JTidy, parsing the resulting XML document can be very verbose.

    XPath was created for this very reason. Elliott Rusty Harold wrote a great article, "The Java XPath API," that briefly outlines how and why you would want to use XPath for evaluating DOM documents. Rather than recapitulate those arguments, if you aren't yet a believer in XPath, I suggest you read that article.

    Although XPath has simplified the job of extracting data from DOM documents, its usage in Java isn't as concise as I'd like. To use XPath in Java, you have to do the following:

     Session s = new Session(); Response r = s.get("http://www.google.com"); DocumentBuilder builder = DocumentBuilderFactory.newInstance().newBuilder(); ByteArrayInputStream in = new ByteArrayInputStream(r.getBody().getBytes()); Document dom = builder.parse(in); in.close(); XPathFactory factory = XPathFactory.newInstance(); XPath xpath = factory.newXPath(); XPathExpression exp = xpath.compile("//a"); //gets all the anchor tags in the document NodeList nodes = xpath.evaluate(exp, dom, XPathConstants.NODE_LIST); for (int i=0; i<nodes.length(); i++) { Node n = nodes.item(i); String href = xpath.evaluate("@href", n, XPathConstants.NODE) .getTextContent(); String text = n.getTextContent(); System.out.println(text + "(" + href + ")"); }

    By contrast, the following code snippet performs the same task as that above, but uses the helper classes of theorg.jdesktop.dom package.

     Session s = new Session(); Response r = s.get("http://www.google.com"); SimpleDocument dom = SimpleDocumentBuilder.simpleParse(r.getBody()); for (Node n : dom.getElements("//a")) { String href = dom.getString("@href"); String text = n.getTextContent(); System.out.println(text + "(" + href + ")"); }

    In the third line, I create a new SimpleDocumentby asking the SimpleDocumentBuilderto parse the response body. In the fourth line, I then iterate over all the elements of the DOM tree that match the specified XPath expression!

    The benefit of this code is not merely the reduced number of lines. Rather, it is in the conceptual weight of the code. How many different concepts are required to understand the traditional DOM/XPath code segment? There are two factories, a builder, an optional compilation step, the NodeList API, and XPathConstants. In the second code block, you need to know two new classes, and the DOM API.

    SimpleDocument can be created two ways. First, you can create it via its constructor, which requires a delegate DOM document. SimpleDocument simply wraps a normalorg.w3c.dom.Document object, and provides convenience methods atop it. Second, you can create aSimpleDocument by calling one of the staticsimpleParse methods onSimpleDocumentBuilder, or one of its non-staticparse methods.

    SimpleDocument andSimpleDocumentBuilder both extend the normal DOM API, and can therefore be used anywhere traditional DOM is required.

    Let's look a moment at some of the key API inSimpleDocument:

    • toXML(): A simple method to convert aSimpleDocument into the equivalent XMLString.
    • getElements(String expression): Returns all the DOM elements that match the given XPath expression.
    • getElement(String expression): Returns the element matching the given XPath expression.
    • getString(String expression): Returns aString (the textContent() of the DOM element) matching the given XPath expression.

    The getElements and getString methods on SimpleDocument are convenience methods that delegate to org.jdesktop.xpath.XPathUtils.SimpleDocument manages a cache of compiled XPath expressions, so you don't have to worry about compiling them. If you want to manage compilation manually, you can skip theSimpleDocument helper methods and instead use XPathUtils directly.


    Working with HTML and XML content in Java rich clients can be very productive. With the inclusion of a little bit of new API and leveraging existing open source projects, I've demonstrated how you can very easily download and work with various types of content, including XML, HTML, and JSON. Using XPath andSimpleDocument, you can easily extract data from XML documents and use this data in your Swing applications.