Test Fests

Version 1

    Test Fests are hackdays organised around the world with a focus to improving the quality and coverage of tests in OpenJDK. In particular IBM, Oracle and Java User Groups are pushing these days. See some photos of a Test Fest in action!

    Running a Test Fest

    What works


    • Spend less time up front explaining OpenJDK procedure and getting people to write tests
    • Get people to build Your Own Environment ahead of time
    • Make sure people got through the JTREG_Tutorials
    • Pick specific areas to focus on for writing tests, e.g. The Base64 class and JSR 310 (Date and Time).
      • Narrow down the scope for these events as much as possible and perhaps even pick 1-3 classes to focus on that you know aren't tested.
      • Grepping source code for "TODO: more tests"

    What didn't work

    • Code Coverage tools are lacking and reports unpublished - TBA Pavel from RedHat is working on this
    • Still need manual work to marshall commits into openjdk (should be solved by betterrev)
    • Testing JSR-310 required quite a bit of explanation since the things that are least tested are Japanese and Islamic calendaring systems which have a lot of local domain knowledge. We should probably avoid this kind of stuff in future.

    Types of activities

    Rules for writing tests

    These originally came from IBM's QA team and were enhanced on by Ed

    1. Think before you code! - Have a specific goal, target specific methods.
    2. Make tests understandable - Fill in the summary tag, be clear and make your test goal explicit
    3. Make tests "small and simple" - Use setup and teardown as necessary
      1. Keep tests idempotent.
      2. Not cleaning up is not only impolite but can cause errors for other test cases!
    4. Test only one thing!
      1. 1 assert per test = easy to understand and simple
    5. Fast tests only!
      1. Quick tests required as they will be run over and over.
    6. Absolute repeatability
      1. Tests ONLY fail if there is a bug in the code under test.
      2. One pitfall is using Thread.sleep to handle concurrency issues
      3. Understand what your goal is (draw it out!)
      4. Use proper concurrency tools to enforce an execution order (e.g. countdown latches)
    7. Independent tests
      1. Tests should not rely on other tests or be affected by execution order.
    8. Provide diagnostic data on failure
      1. Use messages on assertions and throw sensible exceptions.
    9. Ensure portability
      1. Do not hardcode environment details into the test.
    10. Silence is golden
      1. Only speak up if the test is broken. Don't pollute the output.
      2. However, sometimes a little additional output is very helpful for debugging.