Necessary and Sufficient — The Logic of BDD and Automation

Is BDD automation sufficient for our testing purposes?

In high school and college, I had to work really hard at math in order to do well. There are many reasons for that, but there was one exception: geometry. I was a whiz at geometry. I don’t know why, but I suspect that I grasped the logic of geometric proofs better than I did algebra or trigonometry problems.

Fast forward to the early ‘90s when I earned a degree in computer science. Part of that degree was deeper learning about mathematics and logic. Included in that logic learning were the concepts necessary and sufficient. Necessity and sufficiency are distinct concepts, but they’re also related. How are these concepts related to Behavior-Driven Development (BDD) and automation? Let’s find out.

When a condition is necessary, that condition must happen, be present, or be true to meet a specific criterion. For example, for Jane to paint black and white stripes on a wall, she must have access to white paint. If Jane does not have access to white paint, she cannot paint black and white stripes on a wall. It is necessary for Jane to have access to white paint to perform the requisite task.

When a condition is sufficient, it means that the condition will fill a specific need, but it may contain other aspects that are not related to what the original condition required. Following the previous paint example, if Jane has access to a paint palette of five colors, red, orange, black, white, and blue, Jane can paint the requisite black and white stripes on a wall. For this task, Jane does not need the palette’s other colors apart from black and white. This palette is sufficient for her to complete her task even though some of these colors are not needed, or necessary, to complete her task.

What does all this paint stuff have to do with testing, automation, or BDD? Great question! It goes to the automation created during the BDD process. Is that BDD automation sufficient for our testing purposes?

Teams often have the idea that if they are engaging in BDD and using an automation tool such as Cucumber or SpecFlow to create their BDD automation, they have created all of the automation they need. Indeed, much, if not all, of the automation that’s created via a BDD approach is providing value by showing that the application can work under the understanding of the specified acceptance criteria. This is likely valuable and, from a testing standpoint, necessary.

I propose, however, that this automation, and by extension, the testing, is not sufficient.

In general, teams don’t specify every possible testable condition of a feature or story in its acceptance criteria. Even if we could, doing so would threaten to slow down the planning process and jeopardize that feature’s delivery, thus we don’t. This necessarily means that we specify only a subset of the criteria for a feature being “done”; by extension, we’ve only specified a subset of the testable aspects of our software.

Since BDD automation is derived from a story’s or feature’s acceptance criteria, BDD automation is not created for unspecified criteria. This can leave, at the very least, a gap between what has been automated and realizing the full value of what automation can provide. I’ll also throw in that the BDD automation derived from acceptance criteria are really executable specifications as opposed to being test automation, but people like John Ferguson Smart and Andrew Knight do a much better job of explaining that than I.

Does this mean we should abandon BDD? Is BDD harmful? Nope, not at all! When applied appropriately, many teams and organizations see real, material benefits. There is nothing wrong with BDD!

The point of all this is that if you are adopting BDD (or similarly, ATDD) as described by John Ferguson Smart, Andrew Knight, or my teammate Raja Gollamudi, the automation you are creating as a convenient by-product of your initiative is likely necessary for your context, but chances are you’ll also need additional automation to help the testers do their jobs.

One parting comment is that the “non-BDD” automation that you need doesn’t need to be in the same tool or framework that you use for your BDD automation. Many teams believe that creating “traditional” test automation using a BDD tool is the right way to go. Other teams think that the goal for automation is to have one-and-only-one tool for all things automated. You may find success following either of these paths, or you may find heartache. I once worked with a test lead who advocated for using their BDD tool for all of their automation. In retrospect, she said that it initially looked like a good idea, but that over time the additional overhead of the feature files in conjunction with the code files was more a drag than a lift for her team. As in all things, your mileage may vary.

Likely, the tool or framework that you choose for your BDD automation should be different from the framework or tool that you use for your “traditional automation.” Don’t despair! Having two tools (or even three) for an organization or company developing sufficiently different applications on sufficiently different technologies is not the end of the world. Additionally, some testing frameworks support code reuse between their BDD-style interfaces and their traditional interfaces. Look for these opportunities for reuse.

As always, be appropriate in your tool selection and application.