The promise of pairwise testing is that it can help you reduce the number of test cases you need to execute without significantly increasing the risk that you’ll miss a critical defect. While this may be true in some situations, it’s not true in all situations (see “Pairwise Testing: A Best Practice That Isn’t” by James Bach and Patrick Schroeder).
I believe that learning about pairwise testing actually offers a much more important benefit to testers and developers that can pay dividends throughout your entire career in software.
There are plenty of explanations about pairwise testing on the web, so I won’t repeat them here. If you’re not familiar with the technique, I suggest you start here, and then come back to this article.
My First Experience with Pairwise Testing
I had the good fortune to learn about pairwise testing very early in my test career while working on the Windows Shell team. At first, there was lots of FUD around how effective the technique would be, how we might miss important bugs, etc. In the end, however, it proved out to be an incredibly effective technique to find lots of bugs across the huge surface area of the Shell in relatively little time. We built a custom tool that would let a tester specify variables & possible values for those variables, prioritize them, and even link between variables. For example, a certain feature would only be available if the system had a 32-bit processor, but not if it had a 64-bit processor. We used the tool to generate a pseudo-random selection of test cases to execute for each regression test pass. When a generated case lead us to a defect, we could mark it to be preserved for future test passes. Commercially available tools today offer similar features (and probably work a lot better than ours did!).
Believe it or not, accelerated bug-finding was not actually the biggest benefit of pairwise testing. More importantly, we saw a shift in how testers thought about breaking down complex systems into components, and this lead to more effective testing overall. This benefit carried forward into all types of test planning, even when pairwise testing wasn’t used.
4 Steps to Defining a Pairwise Test Plan
Before I go into more details about the derivative benefits of learning pairwise testing, let’s review the steps for creating a pairwise test plan:
- Identify all the potential variables in the system under test.
- List all the possible values or states for those variables individually.
- Determine which combinations might be invalid and indicate this in the test plan so invalid pairs aren’t generated.
- Add weights and priorities to the states based on your best understanding of how the system will be used, business priorities, etc.
Step 1 is about identifying all the “moving parts” or “things that can vary” in the system. These can be explicit variables, such as parameters passed into an API or “implicit variables” such as the position of a particular node in a linked list.
Step 2 is about identifying all the possible ways each of those variables could change.
Step 3 is about winnowing down your matrix to only the possible states. This step shouldn’t weed out possible error states, because those are extremely important to test. Rather, this is about weeding out the impossible states.
Finally, Step 4 is about prioritizing your testing to help guide defect-finding to the most likely or most impactful areas first.
With these steps completed, your pairwise testing tool will generate a reduced set of test combinations to be executed. But the value doesn’t end here.
The Derivative Value of Pairwise Testing
At this point, you’ve actually done something far more valuable than generate a set of test cases. You’ve thought deeply about the system under test, you’ve decomposed it into sub-parts, and you’ve identified all the interesting ways those parts can change. While it sounds obvious, learning to do this effectively can take lots of practice, and sadly, this skill is missing in many people who consider themselves to be “test experts” today.
Consider your last interview for a testing position. You were probably asked to how you would test a particular feature or system or to generate test cases for a function or class. As you came up with every possible test you could think of, your interviewer was likely most interested in how you thought about the problem and how you identified and prioritized risks.
Any complex system is just a composition of simpler systems. By breaking a problem down into smaller parts, you can focus on testing those parts independently, and then test them in various combinations. Building a pairwise test matrix doesn’t help you discover the sub-systems directly, but it forces you to think about what can vary in the system you’re testing and how it can vary. By thinking top-down from this “meta level” rather than about each individual case, I believe you’re more likely to find the full set of interesting tests to execute than if you just start throwing out individual test cases. Additionally, reviewing the a set of smaller systems is a much easier problem than reviewing an entire complex system all at once to look for missing cases.
Let’s walk through an example. Suppose you’re asked to test a function that searches a two-dimensional sorted array to determine if the number is in the array. If the number is found, it returns true, otherwise false.
Your first instinct might be to test a case where the number is in the array and one where the number is not in the array. But those are specific cases. The variable here is “number_is_in_array” and it can have a value of true or false. To generalize, we could change the variable to “number_of_times_value_appears_in_array”. Now it can have the values 0, 1, 2, 3, etc. but we’d probably weight the values 0 and 1 most heavily if those are indeed the most likely cases.
What else could change? Maybe the number exists in the beginning of the first row, somewhere in the middle, or at the end of the last row. There are two variables here: “position_in_row” and “position_in_column”. Each of these could have values of index 0, 1, somewhere in the middle, last position, just before last position. Of course, this variable only applies if “number_of_times_value_appears_in_array” is 1 or greater.
If we’re talking about the position within a row and column, then we probably also need to have variables for “number_of_rows” and “number_of_columns”.
What else could change? The array is supposed to be sorted, but maybe it isn’t… “array_is_sorted” could be another variable. Most of the time, it would be sorted, so we’d probably weight that so that 90% of the generated cases use a sorted array, but 10% of the time the array would be filled randomly.
Up to this point, we haven’t generated a single test case. Rather, we’ve identified several smaller systems within the larger system, and we’ve identified how they can vary individually. The pairwise testing tool can do the hard part of generating the actual test cases for us.
Pairwise testing can be a powerful technique for building a minimal set of test cases, but the most valuable aspect of pairwise testing is that it forces you to practice decomposing complex systems into smaller systems that are easier to think through.