Combinatory Test

Combinatorial tests are necessary when you suspect that multiple parameters can interact to produce a complex fault. We look at different ways of designing for combinatorial tests.

Why combinatory test

The Problem with Exhaustive Testing

Let’s say a system has:

• $n$ parameters
• Each parameter can take $k$ values

Then, to test all possible combinations, you would need $k^n$ test cases.

This is exhaustive testing, and it grows exponentially—quickly becoming impractical as the number of parameters increases.

Combinatorial Testing

To make testing manageable, we use the assumption that at most $p$ parameters interact to cause a bug. This leads to p-way interaction testing, meaning:

• Instead of testing all combinations, we test all possible combinations of $p$ parameters, and all their value combinations.

If at most $p$ parameters interact to cause a fault:

• Number of test cases needed = $\left(\genfrac{}{}{0ex}{}{n}{p}\right)\cdot k^p=\frac{n!}{p!(n-p)!}\cdot k^p$ Where:

• n = total parameters

• k = values per parameter

• p = number of interacting parameters you want to test

Examples of combinatorial testing

Two-way testing (Pairwise)

If we assume faults arise from interactions of at most two parameters, pairwise testing ensures that every pair of parameter values appears together at least once.

Example

Imagine you’re testing a login form with these 3 input parameters:

Parameter	Possible Values
Browser	Chrome, Firefox
OS	Windows, macOS, Linux
User Type	Admin, Guest

In exhaustive testing, the number of test cases would be: $2\text{ (browsers)}\times 3\text{ (OS)}\times 2\text{ (user types)}=12\text{ test cases}$

The goal of pairwise testing is we want to cover all possible value pairs of:

• Browser & OS (Each Browser appears with OS at least once)
  • Chrome – Windows
  • Chrome – macOS
  • Chrome – Linux
  • Firefox – Windows
  • Firefox – macOS
  • Firefox – Linux
• Browser & User Type (Each Browser appears with User Type at least once)
  • Chrome – Admin
  • Chrome – Guest
  • Firefox – Admin
  • Firefox – Guest
• OS & User Type (Each OS appears with User Type at least once)
  • Windows – Admin
  • Windows – Guest
  • macOS – Admin
  • macOS – Guest
  • Linux – Admin
  • Linux – Guest

We now design a minimal set of test cases that covers all these pairs at least once.

TC#	Browser	OS	User Type
1	Chrome	Windows	Admin
2	Chrome	macOS	Guest
3	Chrome	Linux	Admin
4	Firefox	Windows	Guest
5	Firefox	macOS	Admin
6	Firefox	Linux	Guest
7	Chrome	macOS	Admin
8	Firefox	Windows	Admin
9	Chrome	Linux	Guest

You can verify that every pair from above is present at least once in these 9 test cases. In exhaustive testing there would be 12 test cases, in pairwise testing, only 9 test cases needed, but you still get coverage of every parameter pair.

You can use tools, e.g., the python library allpairspy, for generating test combinations.

from allpairspy import AllPairs
#...
pairwise_test_cases = list(AllPairs([browsers, operating_systems, user_type]))
 
for i, case in enumerate(pairwise_test_cases, start=1):
    print(f"Test {i}: OS={case[0]}, Browser={case[1]}, Day={case[2]}")
 
print(f"\nTotal needed: {len(pairwise_test_cases)}")

---

Back to parent page: Software Testing

Web_and_App_Development Software_Testing Blackbox_Testing SOFT3202 Combinatory_Test 2-Way_Testing