One of the principle rules of Continuous Integration (and Continuous Delivery) is that you should never knowingly commit code that will break the build. When you practice test-driven development this is easy: you write a failing test (or, more precisely, a failing "executable specification"), make it pass, and then refactor as required. You only commit your code once you have refactored and ran all of your unit tests, to ensure that you haven't inadvertently broken anything elsewhere in the code.
But acceptance tests take typically require a lot more code than unit tests, and take a lot longer to implement. If you start with a failing automated acceptance test, you may have a failing test for hours or even days.
The general principle of CI still applies for automated acceptance tests - you should never knowingly commit code that breaks one on the build server. When people do this, it inevitably results in a continual stream of broken builds, which people ignore because it is considered the normal state of affairs. There is no easy way to know if a build is broken because of a regression, or because of an "in-progress" acceptance test. In these circumstances, CI has very little value. The status reporting becomes flawed. If "real" regression issues occur, they are detected and fixed more slowly. And any attempt at Continuous Delivery becomes impossible, since you can never reliably know when a build is ready to be released into production.
Here are a few techniques that teams use to get around this problem:
Tagging the acceptance tests
One common approach used with tools like JBehave, Cucumber and SpecFlow is to tag the acceptance tests that are work in progress, and to configure the Continuous Integration build to only run the stories without the work-in-progress tag. For example, the following JBehave scenario uses the @wip tag to mark a scenario that is work-in-progress:
In order to prevent unauthorized use of member points
As the system admin
I want users to authenticate before they can access their account
Scenario: Successful authentication
Given Jane is a registered Frequent Flyer
When Jane authenticates with a valid email address and password
Then Jane should be given access to her account
This approach manages the living documentation well enough, but some other aspects need to be considered when it comes to actually implementing the features.
Many teams use short-lived, preferably local, branches to develop new features. This is fast, easy and common practice for teams using git. The linux code base, for example, relies extensively on feature branching to develop and integrate new features.
For teams still on centralized version control systems, it is a little more dissuasive, as branching and merging using tools like Subversion can be a painful process, and the concept of a local branch generally does not exist.. But can still be a viable option. The trick is not to let the branch live for too long (for example, more than a couple of days), because long-lived branches create a risk of integration issues down the track.
At the risk of stating the obvious, feature branches should also include the corresponding automated tests, whether they be unit, integration, acceptance, or any other automated tests that will be run on the build server. These are written and run locally, alongside the application code, and merged back into the master branch along with the application code when the feature is finished.
Another, preferable, approach is to break down the new feature into small pieces that can be built and delivered incrementally. Even if short-lived feature branches are often used for this sort of work (simply because they are convenient, and make it easier to experiment safely), the increments are completed quickly, often within a few hours, before being merged back into the master.
For bigger changes, you can use a slightly different approach. This usually involves building the new feature in isolation, maintaining the existing solution until you are ready to replace it completely. For example suppose you need to replace a payment processing module in your application. This is a large chunk of work, that you won't be able to do in one sitting. The first thing you do is to isolate the payment processing module, for example using an interface (if you are using a dependency injection framework such as Spring or Guice, this may already be done as part of your normal development work). You then build an alternative implementation of the module, according to the new or modified requirements, using TDD to drive the design and implementation. Your new acceptance tests use the new module; once these all pass, you are ready to replace the old implementation with the new.
This approach is similar to the idea of "Feature Toggles" promoted by Martin Fowler, but much simpler to implement. It makes it feasible to work directly against the master branch, though it will not reduce the risk of integration issues if the development takes too long.
In both of these cases, the aim of the game is to never commit code that breaks a build, but at the same time to keep your code up to date with the latest changes in the code base.
Thucydides is an open source library designed to make practicing Behaviour Driven Development easier. Thucydides plays nicely with BDD tools such as JBehave, or even more traditional tools like JUnit, to make writing automated acceptance tests easier, and to provide richer and more useful living documentation. In a series of two articles, we will look at the tight one and two-way integration that Thucydides offers with JIRA.
The rest of this article assumes you have some familiarily with Thucydides. For a tutorial introduction to Thucydides, check out the Thucydides Documentation or this article for a quick introduction.
Getting started with Thucydides/JIRA integration
JIRA is a popular issue tracking system that is also often used for Agile project and requirements management. Many teams using JIRA store their requirements electronically in the form of story cards and epics in JIRA
Suppose we are implementing a Frequent Flyer application for an airline. The idea is that travellers will earn points when they fly with our airline, based on the distance they fly. Travellers start out with a "Bronze" status, and can earn a better status by flying more frequently. Travellers with a higher frequent flyer status benefit from advantages such as lounge access, prioritized boarding, and so on. One of the story cards for this feature might look like the following:
This story contains a description following one of the frequently-used formats for user story descriptions ("as a..I want..so that"). It also contains a custom "Acceptance Criteria" field, where we can write down a brief outline of the "definition of done" for this story.
These stories can be grouped into epics, and placed into sprints for project planning, as illustrated in the JIRA Agile board shown here:
As illustrated in the story card, each of these stories has a set of acceptance criteria, which we can build into more detailed scenarios, based on concrete examples. We can then automate these scenarios using a BDD tool like JBehave.
The story in Figure 1 describes how many points members need to earn to be awarded each status level. A JBehave scenario for the story card illustrated earlier might look like this:
Frequent Flyer status is calculated based on points
Scenario: New members should start out as Bronze members
Given Jill Smith is not a Frequent Flyer member
When she registers on the Frequent Flyer program
Then she should have a status of Bronze
Scenario: Members should get status updates based on status points earned
Given a member has a status of <initialStatus>
And he has <initialStatusPoints> status points
When he earns <extraPoints> extra status points
Then he should have a status of <finalStatus>
| initialStatus | initialStatusPoints | extraPoints | finalStatus | notes |
| Bronze | 0 | 300 | Silver | 300 points for Silver |
| Silver | 0 | 700 | Gold | 700 points for Gold |
| Gold | 0 | 1500 | Platinum | 1500 points for Platinum |
Thucydides lets you associate JBehave stories or JUnit tests with a JIRA card using the @issue meta tag (illustrated above), or the equivalent @Issue annotation in JUnit. At the most basic level, this will generate links back to the corresponding JIRA cards in your test reports, as illustrated here:
For this to work, Thucydides needs to know where your JIRA server. The simplest way to do this is to define the following properties in a file called thucydides.properties in your project root directory:
You can also set these properties up in your Maven pom.xml file or pass them in as system properties.
Thucydides also supports two-way integration with JIRA. You can also get Thucydides to update the JIRA issue with a comment pointing to the corresponding test result.
But test results only report part of the picture. If you are using JIRA to store your stories and epics, you can use these to keep track of progress. But how do you know what automated acceptance tests have been implemented for your stories and epics, and, equally importantly, how do you know which stories or epics have no automated acceptance tests? In agile terms, a story cannot be declared "done" until the automated acceptance tests pass. Furthermore, we need to be confident not only that the tests exist, but they test the right requirements, and that they test them sufficiently well.
We call this idea of measuring the number (and quality) of the acceptance tests for each of the features we want to build "feature coverage". Thucydides can provide feature coverage reporting in addition to the more conventional test results. If you are using JIRA, you will need to add thucydides-jira-requirements-provider to the dependencies section of your pom.xml file:
Now, when you run the tests, Thucydides will query JIRA to determine the epics and stories that you have defined, and list them in the Requirements page. This page gives you an overview of how many requirements (epics and stories) have passing tests (green), how many have failing (red) or broken (orange) tests, and how many have no tests at all (blue):
If you click on an epic, you can see the stories defined for the epic, including an indicator (in the "Coverage" column) of how well each story has been tested.
From here, you may want to drill down into the details about a given story, including what acceptance tests have been defined for this story, and whether they ran successfully:
Both JIRA and the JIRA-Thucydides integration are quite flexible. We saw earlier that we had configured a custom "Acceptance Criteria" field in our JIRA stories. We have displayed this custom field in the report shown above by including it in the thucydides.properties file, like this:
Thuydides reads the narrative text appearing in this report ("As a frequent flyer…") from the Description field of the corresponding JIRA card. We can override this behavior and get Thucydides to read this value from a different custom field using the jira.custom.narrative.field property. For example, some teams use a custom field called "User Story" to store the narrative text, instead of the Description field. We could get Thucydides to use this field as follows:
Thucydides has rich and flexible one and two-way integration with JIRA. Not only can you link back to JIRA story cards from your acceptance test reports and display information about stories from JIRA in the test reports, you can also read the requirements structure from JIRA, and report on which features have been tested, and which have not.
In the next article in this series, we will learn how to insert links to the Thucydides reports into the JIRA issues, and how to actively update the state of the JIRA cards based on the outcomes of your tests.
Reduce wasted efforts due to misunderstood requirements!
Engage stakeholders more effectively to discover not just what they ask, but what will really help them!
These are just a few of the benefits of Behavior Driven Development (BDD), a powerful collaborative practice that helps teams deliver not only software that works, but to ensure that the features they deliver will provide real business value, and to reduce errors, wasted effort and rework along the way.
* The "three amigos" meeting is a BDD collaborative practice where a business analyst, a developer and a tester work together to precisely define the expected outcomes of a set of requirements. This simple practice is a surprisingly effective way to eliminate misunderstandings and incorrect assumptions.