Building Faultline: A Reusable Chaos-Injection and Linearizability-Checking Harness

Faultline is an open-source Go harness that injects faults into distributed systems, records every concurrent operation clients actually observed, and checks whether that history is linearizable — modeled on the methodology behind Kyle Kingsbury's Jepsen. Its reference target is a three-node etcd cluster; a second, architecturally distinct target (NATS JetStream's key-value store) proves the harness is actually reusable, not just an etcd-specific tool with extra steps.

No consistency violation was found in either target under the tested conditions. That's reported here as legitimate evidence, not a disappointing result — a rigorous "no violations found across N campaigns, fully reproducible" outcome is real infrastructure-testing work. What makes this worth reading, though, is the six concrete bugs the harness caught in itself while being built — because a correctness tool that has never caught anything isn't credible.

Architecture

flowchart TB
    subgraph cli["cmd/faultline (CLI)"]
        campaign["campaign package\nRunN: seed → schedule + workload, checked"]
    end
    subgraph engine["harness (target-agnostic)"]
        chaos["chaos + chaos/docker\nseeded fault schedule\npartition / kill / delay / drop / reorder\nApply → Verify → Clear"]
        workload["workload package\nconcurrent clients, seeded op mix\nfull invoke/return history"]
        checker["checker package\nWing–Gong search, memoized\nexact numeric equality, deletion-minimal"]
    end
    subgraph contract["client.Client / client.SequentialSpec contract"]
        note["Connect / Invoke / Close · Init / Apply\nsame shape for every target"]
    end
    subgraph targets["target integrations"]
        etcd["targets/etcd\ngRPC, linearizable reads, txn-based CAS"]
        nats["targets/nats\nJetStream KV, revision-guarded CAS"]
        toykv["targets/toykv\nprimary-backup fixture"]
    end
    campaign --> chaos
    campaign --> workload
    workload --> checker
    workload --> contract
    contract --> etcd
    contract --> nats
    contract --> toykv

A four-method contract — Connect/Invoke/Close, Init/Apply — is the only thing a new target implements. The fault injector, workload generator, and checker are written once and never change when a target is added. Adding NATS required zero changes to any of them.

Every applied fault is independently verified, never assumed: a partition's Verify pings across the intended break and asserts the ping fails; a claimed fault only counts as coverage once confirmed against the real container.

The checker

checker.Check is a Wing & Gong style search over sequential orderings of a recorded history, memoized on (remaining-operations, sequential-state), with a bounded search budget so an unresolvable history reports inconclusive rather than hanging forever. Two things were hardened deliberately:

• Exact numeric equality via big.Rat, recursively across JSON-shaped values — because any client whose wire protocol round-trips numbers through JSON (as ToyKV's HTTP client does) silently turns a Go int into a float64, and a naive equality check would flag every such operation as a false violation.
• Collision-safe state caching — the memoization key is only a bucket hash; every cache hit is verified with full structural equality before being trusted.

Before being trusted on any real target, the checker was validated against seven hand-built known-good/known-bad histories. A checker that hasn't been validated proves nothing.

Six bugs the harness found in itself

1. ToyKV's CAS wire response used the wrong field. The server wrote the CAS outcome to wireResult.OK; the client read wireResult.Value. Every real client saw nil for every CAS regardless of outcome — undetected because unit tests exercised the store directly, never through the HTTP path.
2. containerIP only read Docker's legacy default-bridge field. .NetworkSettings.IPAddress is empty for any container on a user-defined network — which every deployment here uses. The partition injector silently resolved an empty peer IP until the first live test.
3. The fault injectors bailed on the first cleanup failure. Clear() for partition/kill/netem faults returned immediately on the first error, abandoning cleanup for every other node. This isn't hypothetical — it stranded a live three-node etcd cluster in a partitioned, unrecoverable state for several days before diagnosis. Fixed to be best-effort: attempt every node regardless of earlier failures.
4. Ping-based verification always "passed." The container images didn't ship iputils-ping, so ping failed with "command not found" — indistinguishable, in code, from "partition is genuinely blocking traffic." Verify reported every partition as confirmed even after it had been cleared.
5. NATS JetStream KV defaults to non-linearizable reads. CreateKeyValue unconditionally sets AllowDirect: true, letting any replica (not just the Raft leader) answer a read — a documented, deliberate latency/consistency tradeoff. Left on, an early batch produced 3 false violations with single-operation counterexamples impossible from an empty initial state: a stale replica read, not a real bug. Fixed by disabling it once, forcing every read through the leader.
6. Fixing #5 the obvious way created a new bug. Doing that disable from every workload client's own Connect call created a thundering herd of concurrent reconfiguration attempts on a just-booted cluster — slow enough to time out Connect itself. Fixed by moving it into a one-time Bootstrap step.

Findings: etcd, 100 runs

Metric	Value
Valid, distinct successful seeds	100
Linearizability violations	0
Inconclusive checks	2
Invalid attempts	2
Total recorded operations	35,470
Successful operations	35,378
Ambiguous operations	92
Confirmed operation failures	0
Successful recovery operations	1,500

Fault	Overlapping ops	Successful
delay	443	443
drop	707	705
kill	405	380
partition	463	446
reorder	486	486

Both invalid attempts are kept in the dataset, not discarded — one (seed 1006) was later retried independently and succeeded; both artifacts remain.

Result: zero violations across 35,378 successful operations and 100 valid runs covering all five standard fault types.

Findings: NATS JetStream, a second real target

Seed	Validity	Ops	Ambiguous	Recovery	Verified faults
300	valid	713	1	20/20	reorder
301	valid	832	1	20/20	partition, reorder
302	invalid	834	3	19/20	delay, kill
303	invalid	849	2	19/20	drop, kill
304	valid	1093	2	20/20	partition, reorder
305	valid	913	4	20/20	partition
306	valid	495	2	20/20	kill
307	valid	1012	2	20/20	delay, drop
308	valid	1148	6	20/20	delay
309	valid	1258	2	20/20	partition

8/10 valid, 0 violations, all 5 fault types verified. The two invalid runs have an understood cause: with direct-get disabled, a client routed through a partition-isolated node waits on an internal leader-forward that can outlast the fault's own duration — expected CP-system behavior, not a bug.

Clock skew: the case for not taking the obvious shortcut

Docker containers share their host's wall clock. date -s inside a container — the naive approach — changes the clock for every container on that daemon, not just the target; on Docker Desktop that's the whole VM. So ordinary Docker mode simply rejects clock_skew by design.

The real fix instruments one Go process directly: an optional Linux/amd64 ptrace-based launcher clears the vDSO discovery flag in the traced child, then adjusts only successful CLOCK_REALTIME/gettimeofday reads — monotonic time, and therefore Go's own timers, stay untouched. Verify requires a fresh intercepted sample matching the request; a stored config value can't pass. Both a +2s and a −2s step against a real etcd process produced linearizable histories with successful recovery.

Methodology

• Verify, don't assume — every fault is independently confirmed in effect.
• Invalid attempts are retained, never silently retried away.
• Exhausted search is inconclusive, not a pass.
• The checker is validated before being trusted on any real target.
• Every artifact is self-describing and replayable — seed, config, actual fault timing, image identity, source/binary SHA-256 hashes.

Scope

This project does not claim Byzantine fault tolerance testing, simultaneous fault combinations (faults are currently serialized), leader-directed fault selection, or portability beyond the tested Docker Desktop/Linux environment. No violation found is reported as a bounded, honest negative result — not proof of absence of bugs under untested conditions.

---

Source, every recorded artifact, and CI configuration: github.com/patelpratyush/faultline (MIT licensed).

Reproduce any run offline:

go run ./cmd/faultline check -artifact results/final-b/seed-1101.json