Fault, Error, Failure: Software Testing Fundamentals Every Tester Should Know

A test can execute a faulty line, take the wrong path, and still return the right answer.

That is one of the most useful ideas in Chapter 1 of Introduction to Software Testing by Paul Ammann and Jeff Offutt. It explains why a green test suite needs interpretation, why coverage alone is incomplete evidence, and why choosing inputs is only part of designing a test.

These are practical reading notes from the book's first edition (2008), with sketches and a small free-shipping example to make the ideas concrete. The example and checklist are applications of the chapter's concepts.

1. Fault, error, and failure describe different things

“There is a bug” is a useful starting point. A more precise description helps us investigate it.

Term	Meaning	Free-shipping example
Fault	A defect present in the software	The condition uses >= 50 where the requirement calls for > 50.
Error	An internal state that differs from the intended state because of a fault	The program takes the wrong branch, then sets found to true too early.
Failure	Externally visible behavior that disagrees with the expected behavior	The function reports that an order qualifies when none does.

The fault exists before the test runs. Whether it produces an error, and whether that error becomes a failure, depends on the execution.

This also clarifies testing and debugging. In the chapter's execution-based terminology, testing evaluates the software by observing it run. Debugging investigates the defect behind an observed failure.

Practical takeaway: a useful bug report records the input, setup, expected behavior, and actual behavior. A suspected root cause is valuable, but label it as a hypothesis until the evidence supports it.

2. A fault needs a path to become a failure

The chapter describes three conditions in the RIP model:

1. Reachability: execution gets to the faulty location.
2. Infection: executing it produces an incorrect internal state.
3. Propagation: the incorrect state affects an externally visible result.

This gives us three separate questions when reviewing a test. Does it reach the relevant logic? Do its values distinguish the faulty behavior from the intended behavior? Can that difference survive to an output?

For an automated test to report the failure, its checks must also observe the relevant output and compare it with a trustworthy expectation. An incorrect shipping decision can slip through an assertion that only checks HTTP status 200.

Practical takeaway: after confirming that a line executed, follow the possible wrong state all the way to the assertion.

3. One fault, four very different tests

Suppose the rule is: an order qualifies for free shipping only when its total is over $50. This function answers whether any order in a list qualifies. For clarity, the example uses whole-dollar integer totals and a non-null array.

static boolean anyFreeShipping(int[] totals) {
    boolean found = false;
    for (int i = 0; i < totals.length; i++) {
        if (totals[i] >= 50) { // Fault: should be > 50
            found = true;
        }
    }
    return found;
}

Input	What happens	Expected / actual
[]	The loop body never runs, so the faulty comparison is never reached. The function itself still runs.	false / false
[20, 80]	The comparison runs, but >= and > agree for both values. No infection occurs.	true / true
[50, 80]	At 50, the wrong branch is taken. Later, 80 legitimately qualifies, masking the earlier error in the final result.	true / true
[50, 20]	At 50, the wrong branch is taken, and no later value masks the mistake.	false / true

The last test exposes the fault. Changing >= to > makes it return false.

The third test is particularly instructive. It includes the exact boundary value and creates an error, yet its final assertion can pass. A boundary value can lose its diagnostic value when combined with another input.

Practical takeaway: test a boundary in a context where other data cannot conceal its effect. Then add combinations deliberately to study interactions.

The first error may be the next instruction

Program state includes more than data variables. It also includes the program counter: where execution goes next.

For [50, 20], immediately after evaluating the first if, found can still be false in both the intended and faulty executions. Their next instructions already differ: the intended execution skips the assignment; the faulty one proceeds to found = true.

At this level of tracing, the wrong control-flow position is the first error. The wrong value of found comes afterward. This is also why omitted code can cause an error even when there is no incorrect assignment to point at.

Practical takeaway: when investigating the first divergence, inspect skipped steps, branch choices, and loop progression as well as variable values.

4. A test case includes setup, observation, and cleanup

“Test a $50 order” identifies an input. It leaves several decisions open.

The chapter treats a complete test case as a combination of test values, expected results, prefix values, and postfix values. Postfix values include the actions needed to observe the result and to finish in a stable state.

For a checkout test, that becomes:

• Prefix values: establish a logged-in customer and the required cart state.
• Test case values: supply an order total of exactly $50.
• Expected result: this order does not qualify for free shipping.
• Verification values: open the order summary or request the result needed to inspect the shipping decision.
• Exit commands: remove test data or restore the session and cart to a stable state.

Verification values are the inputs or actions that make the result available. The assertion then evaluates that result.

Two related definitions help explain difficult tests:

Concept	Question to ask	Example improvement
Controllability	How easily can we supply the required inputs and establish the required state?	A test clock lets us reproduce an expiration boundary.
Observability	How easily can we inspect the relevant behavior and effects?	A stable result API exposes the shipping decision explicitly.

The clock and API examples are practical applications of the chapter's definitions. If a scenario is hard to reproduce or its result is hard to inspect, improving testability may be more valuable than adding retries.

5. Coverage needs a named criterion

“We have 90% coverage” is incomplete without knowing what was counted.

A coverage criterion supplies rules for deriving test requirements from an artifact. A test requirement is a specific item to exercise or satisfy. The coverage level is the fraction of those requirements satisfied by the test set.

coverage level = satisfied test requirements / total test requirements

The chapter uses jelly beans to make this concrete. If the criterion is “sample every flavor” and six flavors exist, sampling five gives 5/6, or about 83%, flavor coverage. Sampling more beans of an already-covered flavor does not cover the missing one.

The same reasoning applies to software. For a simple binary decision, branch coverage asks for its true and false outcomes. A statement-coverage requirement asks for a particular statement to execute.

Two distinctions matter in practice:

• Infeasible requirement: no test input can satisfy it under the model being tested. An unreachable statement is an example. Difficulty finding an input does not, by itself, prove infeasibility.
• Subsumption: criterion A subsumes criterion B when every test set satisfying A also satisfies B. In the chapter's standard example, branch coverage subsumes statement coverage. This relationship does not guarantee that an arbitrary suite satisfying A reveals every fault another suite satisfying B reveals.

Coverage helps both with designing tests and with measuring an existing suite. It does not establish that the expectations are correct, the assertions are useful, or all failures are absent.

Practical takeaway: report the criterion, numerator, denominator, and reasons for exclusions. Treat a percentage as evidence about the chosen requirements, alongside evidence about behavior and risk.

6. Test levels answer different questions

The chapter connects testing levels to the development artifacts from which tests are derived:

Level	Main focus in the chapter	Example question
Unit	An individual function or method and its implementation	Is the shipping threshold implemented correctly?
Module	Related units and their shared data within a module	Do the cart operations maintain consistent totals?
Integration	Interfaces and assumptions between modules	Does one service send cents while another expects dollars?
System	The assembled system against its architecture and specification	Does the checkout workflow behave correctly across its components?
Acceptance	User needs and requirements	Does the delivered checkout support the intended purchasing workflow?

Teams use these labels differently, so agree on scope explicitly. The historical examples in the sketch illustrate the chapter's argument about matching tests to fault types; adding a test level alone does not guarantee detection.

The V-shaped relationship also supports early test design. Tests can be designed while requirements and designs are being discussed, before there is executable code. Writing a concrete expected result often exposes an ambiguity immediately.

Three related terms belong here. Verification checks whether an artifact meets the requirements established for it by the preceding development work. Validation checks fitness for intended use. Regression testing checks for unintended loss of existing behavior after a change and can happen at multiple levels.

7. The purpose of testing shapes the work

The chapter presents Beizer's progression in testing mindset:

At the early levels, testing is treated as debugging, as demonstrating that software works, or as demonstrating that it fails. The later levels focus on reducing the risk of using the software and developing a discipline that improves quality throughout the team.

A practical application is to make a test report useful for a decision: explain which behaviors were checked, what failures were found, what remains uncertain, and why that uncertainty matters.

Automation supports this work by making execution, comparisons, cleanup, and reporting repeatable. Choosing meaningful scenarios and expected results still needs care. Repeating a weak check reliably does not make it a strong check.

The book organizes its later testing techniques around four kinds of models: graphs, logical expressions, input domains, and syntax. Chapter 1 supplies the vocabulary for using those models deliberately.

A checklist for the next test you write

1. State the behavior and expected result precisely.
2. Choose values that can distinguish a plausible fault from the intended behavior.
3. Identify how execution reaches the relevant logic.
4. Check whether later inputs or operations could mask the wrong state.
5. Assert on the output or effect that would reveal the failure.
6. Include the required setup, observation steps, and cleanup.
7. Name the test requirement or risk the test addresses.

Try this on one passing test in your own suite: deliberately change a relevant boundary condition and check whether the test detects it. If it still passes, use reachability, infection, propagation, and observation to explain why.

---

Source: Paul Ammann and Jeff Offutt, Introduction to Software Testing, first edition, Cambridge University Press, 2008, Chapter 1, “Introduction,” especially Sections 1.1–1.4. These notes paraphrase selected ideas; the free-shipping example and checklist apply them to everyday testing. The accompanying sketches were prepared for these chapter notes and are not reproduced textbook figures.