1 2 Previous Next

kellyohair

29 posts
kellyohair

Two Blogs Too Many Blog

Posted by kellyohair Jan 7, 2008

Just FYI...

Maintaining two blog sites is a bit problematic, I'll be doing most future Java blogs on blogs.sun.com: http://blogs.sun.com/kto/category/Java

-kto

Final Update

Hip Hip Hooray!

JDK7 Build 24 has been promoted (no raise, just a promotion :^). Remember, Build 24 should not differ from Build 23, in fact if they don't behave exactly the same, please report it as a bug. The only difference is that Build 23 was built from sources that lived in TeamWare workspaces, and Build 24 was built from those same sources living in Mercurial repositories.

The open source repositories used to create Build 24 are available at: http://hg.openjdk.java.net/jdk7/jdk7/. These are often called the jdk7 master repositories.

If you have Mercurial installed on your system, and you have it setup with the forest extension on, building the openjdk should be as easy as:

hg fclone http://hg.openjdk.java.net/jdk7/jdk7 yourjdk7 cd yourjdk7 make./build/*/j2sdk-image/bin/java -version

"Ha!" he says, like it can be so easy. :^)
Refer to the complete build instructions in the fileyourjdk7/README-builds.html. Hopefully, over the next few months, I can now fix some of the build problems people have been running into. I'll push my build changes into the build area at http://hg.openjdk.java.net/jdk7/buildalong with anyone else making build related changes, and then occasionally this build area will be integrated into the master repositories. The same integration procedure that many of the team areas will follow, although it may take some time for all the various pipelines to warm up, so to speak. Should be a fun ride, buckle your seat belts.

Speaking of fun rides, on Highway 101 in the Oregon Dunes area, you can ride in a sand rail (with a professional driver), definitely an "E Ticket" ride.
Rail.png

Google "sanddunes frontier" for more information, no cloning necessary. ;^)

-kto

Update 7

Ok ok, call us snails if you want... but things are progressing, really. We are currently going through a "dry run" event of having team integrators and developers learn Mercurial, clone the experimental repositories, push fake changes, and verify that the repositories work and the servers are stable. If you are on the build-dev@openjdk.java.net alias you may have seen the emails when the push events happened. So far so good, we haven't gone running back to TeamWare, yet, (that's a joke :^). Once Build 24 promotes and the repositories are declared official we can all get back to real work on the jdk.

So when will Build 24 promote? The Thanksgiving week created a bit of a problem, we came up with an unofficial perhaps slightly aggressive target of last week in November. So it's possible this slips into the first week of December, but we'll try and keep it in November. Remember we have literally hundreds of Sun JDK developers involved with this transition, this is not a minor event.

The experimental repositories are still available at: http://hg.openjdk.java.net. And I encourage anyone considering working with these repositories to try them out.

I know I said it before, but we are in the final lap, just keep stopping along the way to clean up the track... the "official" repositories should be available around the end of November.

-kto

Update 6

Build 23 has been promoted, so far there is at least one build bug found (see 6624808 or this openjdk build-dev email). But these sources will be used pretty much 'as is' to start the Mercurial repositories and Build 24, of which experimental repositories are available at: http://hg.openjdk.java.netright now. The control directory disappears and it's Makefile is made part of the enclosing repository. If you have the Mercurial forest extension in your Mercurial installation, you can simply:


And have your own forest clone of the OpenJDK to play with. 

Also see OpenJDK forest, OpenJDK Integration Wheel, and Working in a Mercurial World for more information.

We are in the final lap... "official" repositories are very close.

-kto

Update 5

Well we had a few snafus with Build 22. A hotspot bug 6616627managed to sneak in, and the jaxws workspace lost some GPL markings and we will need to back out the jaxws integration in Build 23 (this should be temporary, expect jaxws to be re-integrated in a few builds).

The split of the langtools, corba, jaxp, and jaxws workspaces has continued to cause some minor issues with regards to doing partial builds from the j2se workspace. Most of these have been resolved in build 23 coming up.

We did have a few changes to the plan. Build 23 is being used (in TeamWare) to hold changes like whitespace normalization, SCCS keyword removal, and also the j2se name change ("j2se" is being changed to "jdk"). No team integrations will be allowed in Build 23 with the exception of emergency fixes (like the above hotspot bug fix).

Build 24 will be in Mercurial and the files managed by TeamWare in Build 23 will be the same files that enter the initial Build 24 Mercurial repositories. There are a few restructuring things that will happen though as the Mercurial images are created. There will not be a "control" repository, instead there will be a Makefile at the top of the forest which is the same Makefile that used to be at control/make. So when you used to:

cd control/make && gnumake

Now you would:
gnumake

See OpenJDK forest for more information on the layout.

Build 23 is being worked on now and should be available within a week, doing the Mercurial conversion for Build 24 should be fairly straight forward, but getting everything all setup for the integrators and developers is not a trivial task, so once we have repositories it may take a few weeks to get it all setup. Hopefully, read-only MASTER repositories can be made available as soon as possible, and maybe sooner so that we can get help trying things out and looking for issues.

Stay tuned...

-kto

kellyohair

VisualVM Tool Blog

Posted by kellyohair Oct 17, 2007

If you like jconsole, go to the VisualVM pages at https://visualvm.dev.java.net/and try out the new VisualVM tool.

Very cool.

-kto

Update 4

We tried very hard to split out corba, jaxp, and jaxws in Build 21 but didn't make it, however they just now got integrated into Build 22. This splits out an additional 6,000 files or so from the primary j2se workspace. This would not have been accomplished at all without the dedicated hard work of Seema, thank you Seema.

So with Build 22 we will now have:

  • control
    Primary control build make files. Builds a complete or partial set of the components listed below.
  • langtools
    Independently buildable javac, javah, javap, javadoc, and apt sources. Uses a Makefile over an ant build script. Contains NetBeans projects and tests.
  • corba
    Independently buildable corba sources. Currently uses Makefiles only, which will change in the future. Requires a BOOTDIR jdk and optionally the langtools bootstrap tools.
  • jaxp
    Independently buildable jaxp sources. Uses a Makefile over an ant build script. Requires a BOOTDIR jdk and optionally the langtools bootstrap tools.
  • jaxws
    Independently buildable jaxws sources. Uses a Makefile over an ant build script. Requires a BOOTDIR jdk and optionally the langtools bootstrap tools.
  • hotspot
    Independently buildable hotspot sources, mostly native code but some Java code. It does need access to a BOOTDIR javac during a build.
  • j2se
    Independently buildable jdk, with access to a JDK_IMPORT_PATH. Requires a BOOTDIR jdk and optionally the langtools bootstrap tools. You can point it at the dist areas of langtools, corba, jaxp, and jaxws, for these classes, or it will pull the classes out of the JDK_IMPORT_PATH rt.jar and tools.jar.

The langtools component is built with the BOOTDIR jdk and creates the jdk contribution files (dist/lib/classes.jar and dist/lib/src.zip) for the resulting jdk image being built, plus bootstrap tools that can be run with the BOOTDIR jdk for building the complete jdk with the latest class versions etc.

The corba, jaxp, and jaxws components are given a BOOTDIR jdk to build with (and optionally the langtools bootstrap files when doing a full control build). All the components create contribution files (dist/lib/classes.jar and dist/lib/src.zip). Corba also contributes some special jdk image files (dist/bin.zip).

Build 22 continues to be our target for the last JDK7 promotion built via the TeamWare workspaces, the Build after this one would be done via Mercurial repositories. There is some risk to this target, and it's possible it might slip, but it continues to be out target, and the momentum is building up.

There will still be a few SCCS keywords in the Mercurial sources at first, but these should be harmless. We'll continue to clean up the source as we go.

-kto

Update 3

Build 20 now contains a separate "langtools" (javac, javah, javap, apt, and javadoc) directory in the Build 20 source bundles.

Build 21 (could be 22) will have separate corba, jaxp, and jaxws directories. Build 20 had some duplicate javac tests between j2se and langtools, this should get corrected in Build 21, which should be out soon.

Build 22 continues to be our target for the last JDK7 promotion built via the TeamWare workspaces, the Build after this one would be done via Mercurial repositories. Then my co-workers might want to burn me at the stake, so I may go into hiding for a while. ;^)

We have started to look at the various conventions we want to put in place for things like changeset comments, etc. The general feeling is that the changeset comment should always contain a bugid and synopsis, and not much else, pushing the bulk of the data about a bug and it's fix into the bug database. The Mercurial changeset itself serves as the exact changes for a bug, so there is no need to save diffs or webrevs, just the changeset global ID.

It has been agreed that the release engineering team will be creating global tags for the various promotions, something like "jdk7-ea-b23", so re-creating the sources as of a promotion should be easy. (The fact that Mercurial makes it so easy to re-create accurate source trees for any changeset or tag name may change the way we deal with saving source bundles, or even creating source bundles. Time will tell.

There may still be a few SCCS keywords in the sources at first, but these should be harmless. We'll continue to clean up the source as we go.

The current plan is to apply a source normalization script to the Java and Native C/C++ sources as they transition from TeamWare to Mercurial. The normalization procedure will attempt to clean up the whitespace uses in the sources, expanding TABS, converting characters, removing trailing blanks on lines, and making sure the file ends with a newline character. Hopefully we can put some hooks in place to prevent this kind of whitespace clutter from coming back.

(WHAT? NO TABS? ... yes, no tabs ... ARE YOU NUTS? ... at times ... WHY? ... because tabs create a source display problem ... BUT IT WORKS FINE FOR ME IN VI/EMACS! ... yes, but what about everybody else?)

-kto

Update 2

The work to create OpenJDK/JDK7 Mercurial repositories is still progressing, we had a hickup getting the langtools split off from the j2se and it did not make Build 19 as you probably well know. We integrated the split into Build 20 and have spent the last 2 weeks adjusting to the change. So Build 20 will contain a separate "langtools" (javac, javah, javap, apt, and javadoc) directory in the Build 20 source bundles, which will become a standalone Mercurial repository soon.

Build 20 also includes a great deal of Makefile changes and minor changes to many files to remove SCCS keywords in the sources and test files. We haven't fixed everything, but we are making quite a bit of progress.

In Build 21 (or Build 22 as a backup) we hope to split off corba, jaxp, and jaxws as separate openjdk repositories. (At some later date, we may be able to better interface with the corba, jaxp, and jaxws teams in terms of how they integrate and how we accept changes from these areas into the jdk product, so this is just step one). Other changes in Build 21 and 22 will be around SCCS keyword changes, and of course all the various teams are also integrating changes all around us, which is to some degree why this is taking a bit longer. Build changes often impact everyone, and we are dealing with a moving target.

We continue to target Build 22 (Build 23 as a backup) be the last promotion built via the TeamWare workspaces, the Build after this last TeamWare one would be done via Mercurial repositories. Then at some point after that, we can start working on providing those same OpenJDK repositories on the openjdk website.

In other areas, we are starting to look at the various conventions we want to put in place for things like changeset comments, etc. and how we will use Mercurial hooks to protect us from mistakes and verify the conventions are followed.

Build 20 Issues with regards to the langtools split

  • You need 'ant' to build langtools, and you need to make sure the environment variable ANT_HOME is set (especially on Windows).
  • When building from the control/make/Makefile, langtools is built first, then the resulting 'dist' directory is supplied to the j2se build via the variable ALT_LANGTOOLS_DIST. The langtools dist/bootstrap/javac.jar is then run with the BOOT jdk (ALT_BOOTDIR) to create all the resulting jdk classes for the build.
    IF langtools is not built or not found, then the javac launcher from the ALT_JDK_IMPORT_PATH is run (a partial jdk build). This ALT_JDK_IMPORT_PATH is assumed to be a recent jdk7 build, or at least recent enough to handle building the jdk7 itself.
    It's acceptable to set ALT_BOOTDIR and ALT_JDK_IMPORT_PATH to the same jdk7, and might even work if they both refer to the same jdk6, but no guarantees.
    ALT_JDK_IMPORT_PATH continues to be the place to import parts of a jdk that a partial build has chosen to not build, but it is also now a backup place to get a javac when the langtools dist area is not available.
  • Prior to the langtools split, the jdk was built in a somewhat bootstrap process, primarily because the javac sources were part of the rest of the jdk. Now that javac is separate, and can be run with a BOOT jdk, the thought was that the entire jdk could be built without actually being run during the build process, a great simplification and a benefit to anyone doing cross-platform builds. Unfortunately, it turns out that the build procedure relied on a few tools like rmic, idlj, native2ascii, and javazic be run from the freshly built classes of these tools in the jdk source. So for the time being, these tools will be run from the BOOT jdk or run via the BOOT jdk until we can resolve these issues. The actual jdk install image (e.g. rt.jar tools.jar etc.) is now created before it is run now, although the image is massaged a little after it's initial image is created.

-kto

The work to create OpenJDK/JDK7 Mercurial repositories is progressing, but before I tell you anything significant, I'll bore you with some basic details about JDK building. ;^)

JDK Build Promotions

First, let me explain a little about the JDK build promotions. The build promotion cycle for active releases is usually 1 or 2 weeks long, and currently for JDK7 it's 2 weeks. That means that roughly every 2 weeks, a promoted JDK7 build is made available at the jdk7.dev.java.netbinary bundle area (plus the JRL source bundle area), and also at the OpenJDK.java.net source bundle area. I assume people see the emails sent out on promotions to the announce@openjdk.java.net alias (subscribe at the openjdk mailing lists site).

Between promotions, individuals integrate their JDK7 changes into the various team integration areas, and the teams will then integrate their changes into the MASTER JDK7 source workspaces. The MASTER source workspaces are like the Pacific Ocean, with all the various teams sending their changes to the MASTER repositories like water flowing through the various rivers, lakes, etc. into the Pacific. At any point the changes might get held up or inspected by a "gate keeper" or "integrator", like the dam operators on rivers (that's "dam operators", not "damn operators"). Ok ok, strange analogy, but it kind of fits. In any case, sometimes a change looks like it might make it into the MASTER workspaces and into a build promotion, but it doesn't, some gate keeper might scoop it out of the river, or some bear might drink it or something (some bears might also make contribution to the river, but let's not go there, most of the pesky bears have been relocated north of here). Ultimately if a change doesn't make a promotion and isn't permanently diverted, it will make the next promotion, or the one after that, just depends on how far it has to go. (And yes, testing is part of this process, the water is indeed safe to drink).

The schedule for when build promotions will happen is fairly predictable, but anything can happen, and sometimes they get delayed. The current schedule is... HA, you thought I was going to give you hard dates! ... well ok, why not, but these are estimated dates, and sometimes you need to add a day to the final promotion day before the actual bundles are available on the jdk7.dev.java.net and openjdk.java.net sites:

  • JDK7 Build 19 final - estimated Aug 31
  • JDK7 Build 20 final - estimated Sept 14
  • JDK7 Build 21 final - estimated Sept 28
  • JDK7 Build 22 final - estimated Oct 12
  • JDK7 Build 23 final - estimated Oct 26

If a build slips, all the following build promotion dates have to be permanently adjusted. These build promotion dates are probably no big surprise, anyone could look at the past delivery of JDK7 build promotions and guessed this much. But I wanted to attach at least some preliminary dates to some build numbers so you understand the planning a bit.

Note that once we get into delivering Mercurial repositories, the whole "time frame" as to when you see source changes is drastically different. The traditional source bundles become less interesting, except for maybe archiving. And even after Mercurial, the JDK Build Promotions will continue, and the binary bundles and all deliveries to jdk7.dev.java.net won't change much.

The "langtools" Separation

Second, hopefully in Build 19 (backup plan is Build 20) we will be doing some major restructuring of the j2se workspace. This first restructuring is called the "langtools" separation. The "langtools" are the tools javac, javah, javap, javadoc, and apt. This change is significant because it removes the fairly complicated javac "recompile" cycle from the j2se build, and greatly simplifies the j2se build (many Makefiles get deleted). It introduces another tree of sources (about 3,000 files) with the name "langtools".

No, it doesn't mean the entire j2se build can convert to ant.

Other Changes

Third, in Builds 20 through 22 or 22, there may be additional restructurings and things like SCCS keyword removals.

The Transition

Lastly, we sincerely hope that Build 22 or 23 will be the last promotion built with TeamWare sources, meaning that at some point after those promotions we can start working on making the OpenJDK repositories available, probably fairly quickly in terms of a read-only state.

Mercurial Transition Details

Just to make sure people are aware of some of the technical details of how the transformation will happen:

  • A Set of Repositories 

    Keep in mind that we are talking about a set of repositories, not just one. The current repository set is: control, langtools, j2se, and hotspot. A one-to-one mapping from the TeamWare workspaces to the Mercurial repositories will be maintained at transition time. (Although we had plans for a Mercurial forest, and may use them eventually, for now the OpenJDK repositories will likely just be a set of separate repositories).

    And more important, we are probably talking many different sets of repositories, one set for the MASTER repositories which are read-only to everyone except the gate keepers or integrators, plus potentially a set of repositories for each of the team integration areas. The specific details remains to be seen. With a DSCMsystem like Mercurial or TeamWare, you usually end up with the Ocean and rivers analogy I mentioned above, with changes trickling in over time, from various team integration areas. Just looking at the MASTER level may be appropriate for some, but for others, particular team integration areas may be of more interest. The patterns we have used over the years with TeamWare will map pretty much the same way to Mercurial.

  • History of Past Changes 

    No past history of the sources will be included (for legal reasons we can't provide history prior to the launch, and for technical reasons we decided to not provide the TeamWare history between the launch and the transition). Effectively each file will be 'sccs edit'd out of the TeamWare workspace, and added to a Mercurial repository.

Conclusion

Hope this was helpful. This project has been extremely hard to predict, and I'm still not sure that some evil issue won't de-rail the plans, but hopefully this keeps you all informed.

It will be over soon, I hope, as you can tell from my writing I'm getting a little dingy (no I'm not getting a small boat). ;^)

-kto

OpenJDK Builds (Solaris & Linux)

Anyone building the new OpenJDK bundles from openjdk.java.net should find that this is an easier build procedure than the JRL building from jdk7.dev.java.net.

First off, it's just the basic JDK sources, no plugin or installer bundling logic has been included yet. Also the special version of Motif is not included for Linux builds, but only some of the include files are needed and they can easily be downloaded from various locations or even installed via official Linux distributions.

Second, there are a few small pieces that we still have legal issues with. So you'll see mention of "Binary Plugs" in the build process. This means that we have provided you the binary versions for these pieces, and they will be copied into your OpenJDK build during the build process. Note that you need the correct "Binary Plug" bundles from www.openjdk.org's Download area.

Third, the Rhino javascript classes are not in the OpenJDK, but are available from http://www.mozilla.org/rhino/(there were some license issues here).

Fourth, ... I'm sure I forgot something... I'll re-edit this post when I remember. :^(

-kto

P.S. Windows builds will not work at this time.

P.P.S. On Solaris you need the Sun Studio 11 compilers, but if you build on Solaris Express, the Sun Studio 11 compilers are already installed.

kellyohair

JDK Builds on Windows Blog

Posted by kellyohair Jan 25, 2007

I need to add this to the JDK build documentation, but it may be helpful to have it posted here for some people.

Building the JDK on Windows can be difficult at times, so if it hasn't been mentioned before, here are a few clues:

  • When using MKS, make sure that the PATH setting has the ${ROOTDIR}/mksnt and ${ROOTDIR}/bin directories BEFOREthe system paths. Ideally they should be the first items in your PATH. There are conflicts between the MKS tools and what is supplied in Windows, and you don't want a mixture. I cannot verify or reproduce this, but there might be some kind of issue regarding long PATH values on Windows. Even with MKS in the right order in PATH, if many paths are placed before MKS, sometimes this doesn't work. So my recommendation is is to place MKS very early in PATH.
    • When using CYGWIN, the same thing is true, make sure /usr/bin is before the system directories. Mixtures of tools will often not work.
    • Use an MKS shell when you start the gnumake.exe (GNU make built for MKS). Starting make in a Windows cmd.exe command window will often not work.
    • Use a CYGWIN shell when you start the /usr/bin/makeof CYGWIN, if you don't you might get an error like:
      /bin/sh: cannot duplicate fd 31 to fd 0: Bad file descriptor
    • The 3.81 version of make (in the latest CYGWIN) does not understand paths that use a drive letter like C:\ or C:/. Which means you cannot use this version of GNU make to build the JDK. It sounds like this is being fixed but it may take some time to get the latest CYGWIN fixed. You need make version 3.78.1 to 3.80, maybe 3.82 if this problem is addressed in tha version, although it sounds like it's a matter of how you build GNU make perhaps? So get an older version of the make command.
    • There have been some reports of the latest find command in CYGWIN is also broken (version 4.3.2 as reported by Dmitri), perhaps with the drive letter path names too. So get an older version (Dmitri recommends 4.3.0) of the find command.
    • If you are building JDK5, you may need to
      unset TMPDIR
      unset TMP
      in your environment. The JDK5 makefiles used these as make variables and they can cause conflicts with the environment variable version used by MKS. I don't know if this is also a problem with CYGWIN since JDK5 didn't build very reliably with CYGWIN anyway. In JDK6, the makefiles were changed to not use these names.
    • The Windows Visual Studio compiler you are using should be in your PATH. You can't run the cl.exe compilers with a full path like C:/.../Bin/cl.exe without also having that Bin in your PATH settings. If you get an error message that says something about not being able to get CC_VERSION, then try running cl yourself in the same shell, perhaps compile and link a small hello world program to verify that works from the shell command line.
    • When setting the JDK ALT_* variables in your environment use the pathname style of "C:/", not "C:\" or CYGWIN's "/cygdrive/C/". Ideally, you should also try and use the path names without spaces (see MKS dosname -s and CYGWIN cygpath -s -m). With JDK6, many ALT_* variables should not need to be set and the makefiles should figure it all out, so try using as few ALT_* variables as possible. The one exception is ALT_UNIXCOMMAND_PATH for CYGWIN, which is by default /usr/bin, but should be a CYGWIN style path if you need to set it for some bizarre reason.

    Hope these tidbits are helpful to somone.

    -kto

kellyohair

JVM TI Agents Article Blog

Posted by kellyohair Jan 17, 2007

Just a plug (and additional reference) on my December 2006 articleon JVM TI at http://java.sun.com/developer/technicalArticles/J2SE/jvm_ti.

I really had not expected this article to be very popular, but I was assuming that only people writing JVM TI agents would be interested. It appears there is quite a bit of general curiosity on VM agents. Of course the Java Lobbyand Sun System News links helped too. :^)

Big thanks to Janice for all her help with this article! It would never have read as well without her help.

Just for reference, there is also a JVMPI to JVM TI conversion article at: http://java.sun.com/developer/technicalArticles/Programming/jvmpitransition/.

-kto

Update for 1/23/2007, just a very short note on windows.

The findbugs target needs to add vmlauncher="false", so the line:

<exec executable="findbugs" failonerror="true">

changes to

<exec executable="findbugs" failonerror="true" vmlauncher="false">

It's not exactly clear why this is necessary, but this allows the findbugs target to work on windows, and also works everywhere else. The findbugs -version exec was also removed because this will trigger the GUI to start up if it runs findbugs.bat, and findbugs.bat doesn't seem to have a -version option.

In addition, the ant 1.7.0 install bits seem to have a windows bin/ant.cmd file that does not have execute permissions, by adding execute permissions to this file, ant can be run from a Makefile. Again, it's not exactly clear why this is needed, but it makes sense for it to have execute permissions, so this seems harmless.

Updated 1/22/2007, tried to mark changes in bold underline. Also added the actual build.xmlto download.

Now I have never liked ants (see my ants blog), but this story is about my adventure with the Apache Ant build system. I can safely say that I still hate ants, all ants, but even ants have a place in the world, ant build scripts included.

So I was crawling along with my little NetBeans Java project just happy as a clam using NetBeans 5.5 with the findbugs plugin, and the JUnit tests that NetBeans help me to create and run so easily. I never had to write an Ant script before, I just pointed NetBeans at my sources, set a few properties and let NetBeans handle it. Overall it was pretty easy, and simple.

But when it came time for a batch product build, I had used a Makefile and a separate build process. So the romantic picnic at the park was over, it was time for the ants to take over. :^) The Makefile worked, but wasn't ideal, and it was always a temporary thing, but it did create the bin scripts and run some bin script tests that I hadn't managed to get the NetBeans building to do.

First off, it's never good to have two different ways to build a product, you never know for sure if the two build mechanisms really built the same thing. So in general, when it comes to a build process, everyone should follow the Highlanderrule of "There can be only one". Granted, a "development build" will always be slightly different, probably a subset, but it should be using the same basic mechanisms as a more formal and complete product build.

Second, I was missing some of the things I wanted built into the batch build process, namely running findbugs and running all my JUnit tests. So I wasn't happy with my existing batch build process.

Ideally I want a batch build process to:

  1. compile with full javac error checking and treat all errors as fatal (javac -Xlint:all -Werror)
  2. construct the 'dist' area (the jar file and including any necessary bin scripts)
  3. run all the JUnit tests
  4. run the bin script tests
  5. run javadoc and treat all errors as fatal
  6. run findbugs with full checking and treat all errors as fatal

Lucky for me I've got a very clean set of source files, and I want to keep them that way, otherwise I'd have to back off on some of my "treat all errors as fatal" requirements. ;^) The first two items above represent the typical development build or "full build" while inside NetBeans, but it was important to me that all the steps should be manually runnable inside NetBeans too.

So I started my adventure to make this happen.

My first thought was about having the Makefile run findbugs and JUnit directly, but that seemed wrong and didn't solve my first problem above. So the answer was fairly obvious, if I needed a Makefile, it would be trivial to have it run the ant script, right? So I just needed to create an ant script. Of course it's never that easy. I did manage to get this to work, and I did learn enough about ant to create this standalone ant script, one that did everything I wanted, but it was NOT simple and easy, and the end result was not very satisfactory.

It turns out that when you create a NetBeans project with an existing ant script, some of the NetBeans features (like debugging a JUnit test) will not work. :^( So some NetBeans features (or what I perceive to be features) are tied to the specific ant build scripts that NetBeans creates.

Then there were the ant problems, which involved having a version of ant that would work with the latest findbugsant task and the junitant task. I finally settled on using ant 1.7 but since I allow for my project to build anywhere, I needed ant 1.7 everywhere, and that meant I had to manage my own version of ant (I could not trust all systems to have an ant that would work for me). I think the ant inside NetBeans 5.5 was 1.6 something, and it didn't work with the findbugs ant task for some reason, but that was ok because while inside NetBeans you really want to use the findbugs plugin for the best interaction with your sources. So I finally gave up on the findbugs ant task and just used:

  <condition property="findbugs.home" value="/Applications/findbugs">
        <os family="mac"/>
    </condition>
    
    <target name="findbugs-batch" depends="init"
            description="Run findbugs in batch mode">
        <exec 
            executable="${findbugs.home}/bin/findbugs"
            failonerror="true">
            <arg value="-textui"/>
            <arg value="-effort:max"/>
            <arg value="-low"/>
            <arg path="${dist.jar}"/>
        </exec>
    </target>

An UPDATE on this. Turns out that Windows is giving me no end of grief regarding pathnames, so this was changed to assume findbugs was in the PATH setting, overall this seems to make life easier. In general avoiding having ant deal with full paths is ideal, that also means avoiding the ant property${basedir} which WILL be a full path. The Makefile changed too, see below. I want this ant script to work on MacOSX, Solaris, Linux, and Windows, others may not have this requirement. I also wanted it to work no matter where the source was moved to, so again, avoid fullpaths. I also added the findbugs -exitcode option so that errors trigger a non-zero process exit code and also the -maxHeap 512 option to give findbugs lots of heap space (it seems to need it, and runs a bit faster this way). So now I use something more like:

   <target name="myfindbugs" depends="init"
            description="Run findbugs in batch mode">
        <exec 
            executable="findbugs"
            failonerror="true">
            <arg value="-maxHeap"/>
            <arg value="512"/>
            <arg value="-textui"/>
            <arg value="-effort:max"/>
            <arg value="-low"/>
            <arg value="-exitcode"/>
            <arg path="${dist.jar}"/>
        </exec>
    </target>

As it turns out, because using the findbugs ant task causes the ant VM to run out of memory, so you had to restart ant with more memory, what a pain. Using my above batch target the findbugs process uses it's own VM. This really doesn't change the performance much since findbugs takes a few minutes to run anyway, and just using the bin findbugs script must set the max memory up higher or something.

So now I had to deal with this loss of NetBeans Junit debug capability. I did like how NetBeans automatically created the menu for the targets in my build.xml file, but I could not live without the ability to quickly debug any JUnit test. So I needed to either configure my build.xml file better, or try another approach.

So I went back to letting NetBeans create it's own ant scripts and when I found where NetBeans placed these files (build.xml plus the nbproject directory), I just copied over all the nbproject directory and the build.xml file to my project's Mercurialrepository and reopened the repository as a pre-configured NetBeans project. That way I didn't have to worry about matching the NetBeans ant conventions to get the JUnit debug feature to work.

An update on this, first, do NOT include the nbproject/private directory in your repository. And second, if you tell NetBeans to use anything other than the default Java platform, things don't work very well. This was easy for me to avoid, but the way the Java platforms are defined in the ant scripts didn't make the files transportable. Maybe this was a Mac specific thing?

Then I edited the NetBeans generated build.xml file (which is now my primary product build.xml file) and added to it.

  • Some properties and hooks into the NetBeans ant scripts I needed to set for some reason. Some of these properties should not need to be set since they are set for you already in the nbproject files, so I'm puzzled why you have to set property values sometimes and not others. You will also notice the echo targets I have created, which is just my style, I like to know when targets are being run, and the ant verbose option is too verbose, so I added appropriate echo commands.
        <!-- project name (determines jar and script name) -->
        <property name="project.name"           value="jprt"/>
        
        <!-- top level package name -->
        <property name="package.name"           value="jprt"/>
        
        <!-- top level paths -->
        <property name="src.dir"                value="src"/>
        <property name="test.src.dir"           value="test"/>
        <property name="lib.dir"                value="lib"/>
        
        <!-- source paths -->
        <property name="bin.src"                value="${src.dir}/bin"/>
        <property name="sbin.src"               value="${src.dir}/sbin"/>
        <property name="doc.files.src"          value="${src.dir}/${package.name}/doc-files"/>
        
        <!-- build paths -->
        <property name="build.dir"              value="build"/>
        <property name="manifest.file"          value="${build.dir}/mainfest.mf"/>
        <property name="build.classes.dir"      value="${build.dir}/classes"/>
        <property name="build.test.classes.dir" value="${build.dir}/test/classes"/>
        
        <!-- dist paths -->
        <property name="dist.dir"               value="dist"/>
        <property name="bin.dir"                value="${dist.dir}/bin"/>
        <property name="sbin.dir"               value="${dist.dir}/sbin"/>
        <property name="dist.jar"               value="${dist.dir}/${project.name}.jar"/>
        
        <!-- path to test scripts  -->
        <property name="test.script"            value="${test.src.dir}/test.sh"/>
        <property name="test.system.script"     value="${test.src.dir}/test_system.sh"/>
        
        <!-- javadoc paths -->
        <property name="dist.javadoc.dir"       value="${dist.dir}/javadoc"/>
        <property name="doc.files"              value="${dist.javadoc.dir}/${package.name}/doc-files"/>
        
        <!-- classpath settings -->
        <property name="javac.classpath"        value=""/>
        <property name="run.classpath"          value="${build.classes.dir}"/>
        <property name="javac.test.classpath"   value="${build.classes.dir}:${lib.dir}/junit.jar"/>
        <property name="run.test.classpath"     value="${build.test.classes.dir}:${javac.test.classpath}"/>
        
        <!-- default system options -->
        <condition property="system.instance" value="testsystem-ant">
            <not> <isset property="system.instance"/> </not>
        </condition>
        
        <!-- always provide debugging information -->
        <property name="javac.debug"            value="true"/>
        
        <!-- the main class for the project -->
        <property name="main.class"             value="${package.name}.tools.Main"/>
        
        <!-- make sure netbeans knows we have a manifest file -->
        <property name="manifest.available"     value="true"/>
    
        <!-- hooks into netbeans ant files -->
        <target name="-pre-compile"             depends="mycompilestart"/>
        <target name="-post-compile"            depends="mycompileend"/>
        <target name="-do-jar-with-manifest"    depends="mymanifest"/>
        <target name="-pre-jar"                 depends="myjarstart,mymanifest"/>
        <target name="-post-jar"                depends="myjarend,mybinfiles,mysbinfiles,mypropfiles,myjartests"/>
        <target name="javadoc"                  depends="myjavadoc"/>
    
    
    
  • The special targets of my own:
       <!-- just used to add messages at certain points -->
        <target name="mycompilestart"> <echo message="COMPILING"/>           </target>
        <target name="mycompileend">   <echo message="COMPILING COMPLETED"/> </target>
        <target name="myjarstart">     <echo message="BUILDING JAR"/>        </target>
        <target name="myjarend">       <echo message="JAR COMPLETED"/>       </target>
        
        <!-- create my own manifest file -->
        <target name="mymanifest" description="Create manifest file">
            <echo message="Creating manifest file: ${manifest.file}"/>
            <manifest file="${manifest.file}">
                <attribute name="Built-By"   value="${user.name}"/>
                <attribute name="Main-Class" value="${main.class}"/>
            </manifest>
        </target>
        
        <!-- copy the bin files -->
        <target name="mybinfiles" description="Create bin files">
            <echo message="Populating bin files"/>
            <mkdir dir="${bin.dir}"/>
            <copy todir="${bin.dir}">
                <fileset dir="${bin.src}" casesensitive="yes">
                    <include name="*"/>
                </fileset>
            </copy>
            <chmod dir="${bin.dir}" perm="ugo+rx" includes="*"/>
        </target>
        
        <!-- copy the sbin files -->
        <target name="mysbinfiles" description="Create sbin files">
            <echo message="Populating sbin files"/>
            <mkdir dir="${sbin.dir}"/>
            <copy todir="${sbin.dir}">
                <fileset dir="${sbin.src}" casesensitive="yes">
                    <include name="*"/>
                </fileset>
            </copy>
            <chmod dir="${sbin.dir}" perm="ugo+rx" includes="*"/>
        </target>
        
        <!-- copy the property files into the sbin directory -->
        <target name="mypropfiles" description="Create sbin property files">
            <echo message="Populating ${dist.dir} with prop files"/>
            <mkdir dir="${dist.dir}"/>
            <copy todir="${dist.dir}">
                <fileset dir="${src.dir}" casesensitive="yes">
                    <include name="*.properties"/>
                </fileset>
            </copy>
            <echo message="System instance: ${system.instance}"/>
            <!-- Ant echo is Broken: <echo file="${dist.dir}/config-default.properties"
                  message="jprt.system.instance=${system.instance}"/> -->
            <exec executable="echo" failonerror="true"
                  output="${dist.dir}/config-default.properties">
                <arg value="jprt.system.instance=${system.instance}"/>
            </exec>
        </target>
        
        <!-- test the jar file -->
        <target name="myjartests" description="Test jar.">
            <echo message="Testing: java -jar ${dist.jar} "/>
            <java jar="${dist.jar}" fork="true" failonerror="true">
                <assertions> <enable/> </assertions>
            </java>
            <echo message="Testing: java -jar ${dist.jar} help"/>
            <java jar="${dist.jar}" fork="true" failonerror="true">
                <arg value="help"/>
                <assertions> <enable/> </assertions>
            </java>
            <echo message="Testing: java -jar ${dist.jar} usage"/>
            <java jar="${dist.jar}" fork="true" failonerror="true">
                <arg value="usage"/>
                <assertions> <enable/> </assertions>
            </java>
            <echo message="Testing: java -jar ${dist.jar} version"/>
            <java jar="${dist.jar}" fork="true" failonerror="true">
                <arg value="version"/>
                <assertions> <enable/> </assertions>
            </java>
            <echo message="TESTING JAR COMPLETED."/>
        </target>
        
        <!-- test the bin script -->
        <target name="mybintests" description="Test bin.">
            <chmod file="${test.script}" perm="ugo+rx"/>
            <echo message="Testing: sh -c ${test.script} ${dist.dir}"/>
            <exec executable="sh" failonerror="true">
                <arg value="-c"/>
                <arg value="${test.script} ${dist.dir}"/>
            </exec>
            <echo message="TESTING BIN SCRIPT COMPLETED."/>
        </target>
        
        <!-- test the system script -->
        <target name="mysystemtest" description="Test system.">
            <chmod file="${test.system.script}" perm="ugo+rx"/>
            <echo message="Testing: sh -c ${test.system.script} ${dist.dir} ."/>
            <exec executable="sh" failonerror="true">
                <arg value="-c"/>
                <arg value="${test.system.script} ${dist.dir} ."/>
            </exec>
            <echo message="TESTING SYSTEM SCRIPT COMPLETED."/>
        </target>
        
        <!-- findbugs target -->
        <target name="findbugs" depends="myfindbugs"
                description="Run findbugs in batch mode"/>
        
        <!-- my target that runs findbugs directly (in separate VM with 512M heap) -->
        <target name="myfindbugs" description="Run findbugs in batch mode">
            <echo message="findbugs -maxHeap 512 -textui -effort:max -low -exitcode ${dist.jar}"/>
            <exec executable="findbugs" failonerror="true" vmlauncher="false">
                <arg value="-maxHeap"/>
                <arg value="512"/>
                <arg value="-textui"/>
                <arg value="-effort:max"/>
                <arg value="-low"/>
                <arg value="-exitcode"/>
                <arg path="${dist.jar}"/>
            </exec>
            <echo message="FINDBUGS COMPLETED."/>
        </target>
        
        <!-- my own javadoc target because doc-files don't seem to get copied over -->
        <target name="myjavadoc" depends="init,-javadoc-build,-javadoc-browse" 
                description="Build Javadoc.">
            <mkdir dir="${doc.files}"/>
            <echo message="Populating doc-files directory: ${doc.files}"/>
            <copy todir="${doc.files}">
                <fileset dir="${doc.files.src}" casesensitive="yes">
                    <include name="*"/>
                </fileset>
            </copy>
            <echo message="JAVADOC COMPLETED."/>
        </target>
        
        <!-- all target -->
        <target name="all" depends="jar,test,mybintests,javadoc,findbugs"
                description="Do everything"/>
        
    
    

So all the above is new and updated. I could not get the echo to add a newline to the file, even following all the documentation on echo, so I just used the exec target and the echo command of the system. I also had problems with javadoc populating the doc-files, so I had to add my own doc-files copy. I'm not sure what I did to break this. The javadoc command seemed to be sensitive to relative vs. full paths, or the current directory setting.

My recommendation again is to avoid the use of full paths everywhere you can, it just makes life easier. Also, any need to do simple shell commands like:
  domainname | cut -d'.' -f2 | tr '[:upper:]' '[:lower:]'

had better be done in a shell script or in the Makefile. In my real Makefile I do some system specific shell commands to determine the value of an ant property that I pass into ant. This works well for me because when running ant directly I don't need this setting. The other way to do this would be to exec a shell script and capture it's output. Running shell scripts in ant seems to work best when you run them with sh command. Windows often will not recognize a shell script.

The findbugs task was not connected into the NetBeans build/test system, and is just used by the Makefile or directly from the ant script.

Well, it turns out that this works very nicely.

The following Makefile was updated to avoid the use of full paths and simplify how the ant targets are used.

The Makefile was trivial, and looks like:

# Makefile to simulate a NetBeans build

# This Makefile is located one directory below the ant basedir
TOPDIR  = ..

# Value of ant property that needed to be created OS specific
system_instance=testsystem

# How to run ant
ANT_OPTIONS += -Djprt.system.instance=${system_instance}
ANT = ant $(ANT_OPTIONS)

# All ant targets of interest
ANT_TARGETS = all jar test javadoc findbugs clean init compile

# Create a make target for each
$(ANT_TARGETS):
        ( cd $(TOPDIR) && $(ANT) $@ )

# Declare these phony (not filenames)
.PHONY: $(ANT_TARGETS)


Update, now the path to ant and findbugs just need to be put in your PATH.

I'm sure there are better ways to do some of this, but I did get the above working, and I am just an ant beginner. I still don't see any easy way to loop over a set of names with ant, and the 'condition' task is really confusing. (No wonder there are 600+ page books on how to write ant scripts.) I suppose people say the same thing about Makefiles. ;^)

Hope someone gets something out of this, and yes I've learned to live with ants. ;^)

-kto

kellyohair

JDK6 Build Cheat Sheet Blog

Posted by kellyohair Dec 15, 2006

JDK6 Build Cheat Sheet

Just thought I'd list a few ways that the JDK can be built. These apply to JDK6 and JDK7, JDK5 building is a little different but has some of the same settings. The gnumake used here is the GNU make 3.78.1 or newer.

  • On Solaris systems with the Companion CD installed, it can usually be found with the name gmake in/usr/sfw/bin/ or /opt/sfw/bin/.
  • On Linux just use make from/usr/bin/
  • On Windows and using CYGWIN, just use make from CYGWIN's /usr/bin/ (but make sure it's not 3.81, which seems to have taken away the ability to use paths with the ':' character, like C:/anything.)
  • On Windows with MKS, you will need a GNU make built for MKS.

See the JDK build instructions for more information.
NOTE: Any variables described below that start withALT_ can be set on the gnumake command line, or be set in your environment. None of the Makefiles should be setting these variables internally, so no internal make variable setting should change your environment variable setting of them. The command line variable settings will override any setting inside the Makefiles.

Control Builds

From the control/make/Makefile:

  • Build everything for a developer: 

    gnumake dev
    OR
    gnumake DEV_ONLY=true

  • Skip the fastdebug build (just builds the product bits): 

    gnumake SKIP_FASTDEBUG_BUILD=true DEV_ONLY=true

  • Don't build the fastdebug bits, and skip the hotspot and deploy areas: 

    gnumake BUILD_HOTSPOT=false BUILD_DEPLOY=false ALT_JDK_IMPORT=/jdk1.6.0 SKIP_FASTDEBUG_BUILD=true DEV_ONLY=true

  • Build just the j2se debug image: 

    gnumake BUILD_HOTSPOT=false BUILD_DEPLOY=false ALT_JDK_IMPORT=/jdk1.6.0 SKIP_FASTDEBUG_BUILD=true SKIP_DEBUG_BUILD=false DEV_ONLY=true debug_build

Here is a short list of the make variables that can be set on the gnumake command line:

  • BUILD_HOTSPOT 

    Set to false to avoid building the hotspot VM, but you will need to also set ALT_JDK_IMPORT_PATH to a built jdk product of the same version you are building. I recommend setting this to false all the time, unless you are changing the hotspot VM.

  • BUILD_DEPLOY 

    Set to false to avoid building javaws or the plugins. I recommend setting this to false all the time.

  • BUILD_INSTALL 

    Set to false to avoid building install bundles (you should never need to do this). I recommend setting this tofalse all the time.

  • BUILD_J2SE 

    Set to false to avoid the core j2se, which is unlikely you'd want to do this. I'd leave this one alone, unless you are only working on the hotspot VM.

  • BUILD_MOTIF 

    Set to false to avoid building the Motif libraries, but you would need to then set ALT_MOTIF_DIR to where to get the built Motif libraries. This doesn't take long to build so I have never played with this, and I'm not even sure it's necessary

  • NO_DOCS 

    Set to true to avoid any javadoc generation.

  • NO_IMAGES 

    Set to true to avoid any images generation. This means the creation of the more formal JDK install-like image with the rt.jar and tools.jar files. When the images are not built, the JDK is left in the OUTPUTDIR area as a simple set ofbin, lib, include, andclasses directories. Which can be run as-is, but do not represent the form that a JDK install image will take. The creation of the images is mostly a disk movement activity, and on some machines this is very fast, and with others it can be slow.

  • SKIP_FASTDEBUG_BUILD 

    Set to true to skip building any fastdebug bits (a set of -g -O built images with full runtime assert checking enabled).

  • SKIP_DEBUG_BUILD 

    Set to false to build the debug images (-g with full runtime assert checking).

  • DEV_ONLY 

    Set to true to get the settings:SKIP_COMPARE_IMAGES=true BUILD_INSTALL=false NO_DOCS=true.

  • SKIP_COMPARE_IMAGES 

    Set to true to avoid a comparison of previous release build bundles with the ones you have created. You normally won't want this, so always set this to true.

  • ALT_OUTPUTDIR 

    Alternative output directory rather thancontrol/build. I recommend that the sources and this output directory be on your build machines local disk, for best build time performance.

  • ALT_BOOTDIR 

    Alternative location for a bootstrap JDK, or install image of the previously released JDK (e.g. jdk1.5.0 for building jdk1.6.0). You really should set this all the time, just to make sure you get a local disk copy, for best build time performance. In theory, this doesn't have to be jak1.5.0, but could be a newer JDK, even the same version being built. But normally it's the previously release JDK.

  • ALT_JDK_IMPORT_PATH 

    Alternative location for a JDK build of the same release as what you are building. If you skip building some components, the Makefiles will try and copy over the pre-built bits from this location, e.g. if you use BUILD_HOTSPOT=false (or don't have ahotspot directory), then the hotspot built images will be copied from this location.

Also try running gnumake help for more help on building options.

J2SE Builds

If you only want to work with the j2se and not the VM, you might consider just going into the j2se/make directory and running gnumake from there. Just run gnumake help for some help.

I'd recommend setting ALT_BOOTDIR andALT_JDK_IMPORT_DIR (as described above).

Hotspot Builds

If you only want to work with the hotspot and not the VM, you might consider just going into the hotspot/makedirectory and running gnumake from there. Just rungnumake help for some help.

I'd recommend setting ALT_BOOTDIR andALT_JDK_IMPORT_DIR (as described above).

Traditionally the hotspot build procedure have differed greatly from the rest of the JDK. This is mostly due to the fact that for the most part, it's more like building a large C++ library rather than anything else. Building each of the platforms was slightly different, for example on Windows the tool NMAKE.EXEwas used instead of gnumake. It's the Makefiles and build scripts in the hotspot/build directory that contain the "real" hotspot build procedures. In JDK6 a gnumakeMakefile was added at hotspot/make/Makefile which will use the Makefiles in the hotspot/build area, providing you a more generic hotspot building experience. More experienced hotspot builders may prefer to dive into thehotspot/build area.

Summary

So as you can see, some similarities, some differences between these build rules.

Hope this was helpful, if you have more questions please post comments.

-kto

Filter Blog

By date: