Skip to main content
Software Testing · lesson 6 of 8

Beginner

1

Intermediate

4

Advanced

3

Also part of Software Testing

12
Open the Software Testing page →
Software Testing advanced Lesson 6 of 8

Mutation Testing: When 100% Coverage Still Isn't Enough

A test suite with full branch coverage still lets 3 injected bugs slip through undetected. Mutation testing finds exactly which ones, and why.

By SkillByExample · Published

testing mutation-testing mutmut coverage

A test suite can reach 100% branch coverage and still not notice when a developer changes >= to >, or rounds to the wrong number of decimal places. Coverage measures what ran; mutation testing measures whether your assertions would have caught it if the code had been wrong.

The function and its fully-covered test suite

def final_price(price, is_member):
    """10% member discount; $5 shipping credit for orders >= $50 (based on original price)."""
    original = price
    if is_member:
        price = price * 0.9
    if original >= 50:
        price -= 5
    return round(price, 2)
from final_price import final_price

def test_member_order_over_50():
    assert final_price(100, True) == 85.0

def test_member_order_original_over_50_but_discounted_below():
    assert final_price(55, True) == 44.5

def test_non_member_order_under_50():
    assert final_price(20, False) == 20.0

This is the same function from the coverage-metrics lesson, now with all three tests needed to reach 100% branch coverage on both if statements. Running mutmut — a real Python mutation-testing tool — against it, inside a Linux container since mutmut has no native Windows support:

$ mutmut run
    Generating mutants... done (1 files mutated, 0 ignored, 0 unmodified)
    Running stats... done
    Running clean tests... done
    Running forced fail test... done
Running mutation testing
14/14  🎉 11  🫥 0  ⏰ 0  🤔 0  🙁 3  🔇 0  🧙 0

$ mutmut results
final_price.x_final_price__mutmut_5: survived
final_price.x_final_price__mutmut_6: survived
final_price.x_final_price__mutmut_14: survived

11 mutants killed, 3 survived — despite 100% branch coverage. 🎉 is a killed mutant (a test failed, as it should); 🙁 is a survivor (every test still passed with the code changed). Three specific mutations slipped through undetected.

What the three survivors actually changed

$ mutmut show final_price.x_final_price__mutmut_5
# final_price.x_final_price__mutmut_5: survived
--- final_price.py
+++ final_price.py
@@ -2,6 +2,6 @@
     original = price
     if is_member:
         price = price * 0.9
-    if original >= 50:
+    if original > 50:
         price -= 5
     return round(price, 2)

$ mutmut show final_price.x_final_price__mutmut_6
# final_price.x_final_price__mutmut_6: survived
--- final_price.py
+++ final_price.py
@@ -2,6 +2,6 @@
     original = price
     if is_member:
         price = price * 0.9
-    if original >= 50:
+    if original >= 51:
         price -= 5
     return round(price, 2)

$ mutmut show final_price.x_final_price__mutmut_14
# final_price.x_final_price__mutmut_14: survived
--- final_price.py
+++ final_price.py
@@ -4,4 +4,4 @@
         price = price * 0.9
     if original >= 50:
         price -= 5
-    return round(price, 2)
+    return round(price, 3)

Two of the three surviving mutants (_5 and _6) change the >= 50 boundary condition itself — to > 50 and >= 51. Both are functionally the same bug: an order of exactly $50 would incorrectly lose its shipping credit. Neither survives by accident — none of the three existing tests (100, 55, 20) ever calls final_price with original set to exactly 50. This is precisely the gap boundary-value analysis targets directly: testing well inside a range, or a value that merely crosses a boundary after other logic runs, is not the same as testing the boundary value itself. The third survivor (_14) changes round(price, 2) to round(price, 3) — none of the existing assertions use a value whose second and third decimal places actually differ, so rounding to the wrong precision produces an identical result for every input this suite happens to try.

Killing all three

def test_boundary_at_exactly_50():
    assert final_price(50, False) == 45

def test_rounding_precision():
    assert final_price(33.33, True) == 30.0

final_price(50, False) sits exactly on the boundary: original >= 50 is True, so the credit applies — a mutated > 50 or >= 51 would wrongly skip it and return 50 instead of 45. final_price(33.33, True) computes 33.33 * 0.9 = 29.997 before rounding — rounding to 2 decimals gives 30.0, but rounding to 3 gives 29.997, so the round(price, 3) mutant now produces a different, wrong result.

$ mutmut run
    Generating mutants... done (1 files mutated, 0 ignored, 0 unmodified)
    Running stats... done
    Running clean tests... done
    Running forced fail test... done
Running mutation testing
14/14  🎉 14  🫥 0  ⏰ 0  🤔 0  🙁 0  🔇 0  🧙 0

All 14 mutants killed. The test suite’s coverage didn’t change at all between the two runs — it was already at 100% branches before these two tests existed. What changed is whether the assertions actually depended on the values a real bug would corrupt.

Practice

  1. Predict a survivor. Before running it, guess whether a mutant that changes price = price * 0.9 to price = price * 0.09 would survive against the original 3-test suite (before the boundary/rounding tests were added). Run mutmut run and mutmut results to check.

  2. A mutant that can’t be killed reasonably. mutmut will also generate mutants like changing the docstring or a comment — these always survive because they can’t affect behavior. Why doesn’t a 0% “kill” rate on those particular mutants indicate a real testing gap?

  3. Combine with coverage. Run coverage run --branch -m pytest with the final 5-test suite from this lesson, then compare the branch-coverage percentage to the mutation score (14/14 killed). What does agreement between “100% branch coverage” and “100% mutation score” tell you that either number alone wouldn’t?

  4. Apply it to your own function. Pick any function you’ve already written tests for (in this topic’s other lessons, or your own code), install mutmut in a Linux environment or container, and run it. Report how many mutants survive and what each one reveals about an untested assumption.

Solution 1

It survives. test_member_order_over_50 calls final_price(100, True) and asserts 85.0. With the mutation, 100 * 0.09 = 9.0, then the $50+ credit check: original is 100 (unchanged, since original is captured before the discount), so 9.0 - 5 = 4.0 — the test would expect 85.0 but get 4.0, so this mutant is actually killed, not a survivor. (If your prediction was “survives,” this is exactly the kind of assumption mutation testing exists to check — run it and let the tool settle the question rather than reasoning it out by hand.)

Solution 2

A mutant on a comment or docstring changes text that has zero effect on the compiled behavior of the function — there is no possible test, however thorough, that could ever “kill” it, because nothing observable changed. mutmut (like most mutation testers) mutates executable code paths, not comments, specifically to avoid this — a survivor is only meaningful when the mutation could have changed behavior and your tests didn’t notice.

Solution 3

Agreement between 100% branch coverage and 100% mutation score means every decision path ran and every assertion was specific enough to depend on the exact values that could be corrupted along the way — the strongest evidence available (short of formal verification) that this test suite would catch a real regression here. Branch coverage alone (as this topic’s earlier lesson showed) can be 100% while the actual assertions are too loose to notice a bug; mutation testing is the check that closes that gap.

Solution 4

(Open-ended — depends on the function chosen.) The general pattern to expect: functions with boundary conditions, off-by-one-prone arithmetic, or loosely-typed assertions (checking “no error raised” instead of an exact value) are the most likely to have survivors, for the same reason final_price’s two boundary mutants did — nothing in the existing suite happened to probe that exact edge.

Frequently Asked Questions

What does a mutation testing tool actually do?
It makes one small, mechanical change to your source code at a time — flipping >= to >, changing a constant, swapping an operator — creating a 'mutant.' It then reruns your test suite against that mutant. If a test fails, the mutant is 'killed' — your tests caught the change. If every test still passes, the mutant 'survived' — meaning your tests wouldn't have caught that exact bug if a developer had introduced it for real.
Isn't 100% branch coverage already good enough?
Coverage only tells you a line or branch executed during the tests, not that any test's assertion depends on the specific value that ran through it. A test suite can execute every branch of a function and still never assert on the boundary value, the exact rounding precision, or any other detail a bug could hide in. Mutation testing is the direct check on whether your assertions would actually notice if that detail changed.
Why do mutants that change a boundary condition (like >= to >, or a constant) survive so often?
Because most test suites test well inside a valid range and never call the function with an input at the exact boundary value. A mutant that only changes behavior right at that one specific value has nothing to disagree with if no test ever lands there — which is exactly the gap boundary-value analysis is designed to close: test the edges, not just the middle of each range.
Is mutation testing practical to run on every commit?
Usually not the whole suite — mutation testing is computationally expensive because it reruns your tests once per mutant (hundreds or thousands of mutants on a real codebase). Most teams run it periodically or scoped to recently changed files (mutmut and similar tools support this), treating a low mutation score as a signal to strengthen specific tests rather than a per-commit gate.