So you've got hundreds of tests, but they take ages to run. You have a Continuous Integration server, but it takes an hour to tell anyone when there's a failure. What can you do?
This is where staged builds can come in handy. I basically distinguish fast unit tests from slower integration tests. TestNG test groups are very cool for this, but you can also use simple naming conventions. For example, your unit tests run everything called *Test except classes called *IntegrationTest, whereas your integration tests only run *IntegrationTest.
Unit tests should be snappy, and give feedback within minutes.For integration tests, they should do the job they are intended to do, but time is not really of the essence.
Strictly speaking, unit tests are tests written in isolation, with no interaction with other components. I tend to be a bit more pragamtic here, so some interaction is allowed. However, you do need to be careful loading Spring contexts or Hibernate configurations, as this can be slow. If it slows down your fast unit tests, put it with the integration tests.
Heavy-weight tests, such as performance, integration, or GUI tests, will take longer to do. So these should always go in the integration-test category.
Actually, I often distingush several types of integration test. Here's how I often do it:
Each task is a separate build on the build server (say, Hudson). The first kicks off whenever the source code repository is updated. The others kick off once the fast unit tests have succeeded.
I'll be giving a 4 day Java Power Tools Bootcamp in San Francisco, fromMay 12 to May 15 just after JavaOne. So, if your in SF at this time, come along!