Differential Testing Revealed What Conformance Testing Missed: A Case Study with Open-Source eMRTD Libraries

Kazuru — Tue, 07 Jul 2026 21:06:27 +0000

Conformance suites tell us whether a library passes its own tests. They do not tell us whether two independent implementations make the same decision when presented with identical protocol traffic. ICAO Doc 9303 defines how an e-passport should behave. But the open-source library you choose determines what your application actually experiences. Libraries pass unit tests. Integrators pass interoperability tests. At some point, we stopped asking a much simpler question: if two independent implementations receive exactly the same protocol traffic, will they make the same security decision?

We set out to find an answer. Our approach was a controlled experiment built around a deterministic synthetic protocol profile, with no passport on our desks, no NFC reader, and no RF variables to chase. We replayed the same protocol exchange against two widely used wire-tier eMRTD readers: gmrtd in Go and JMRTD in Java.

What we found was not dramatic, but it was revealing. PACE failed during the initial negotiation. BAC then completed successfully. For an application that merely checks whether a secure session exists, there was no explicit indication that a protocol fallback had occurred. Both libraries produced the same observable behaviour under the same profile.

The experiment was also entirely reproducible. We could rerun it later the same afternoon and obtain the same result, backed by the same traces and run metadata.

BAC, PACE, and why we bothered with differential testing

If you have shipped TLS, you already know what negotiation failure feels like at the integration layer. eMRTD access control is the same shape of problem on a contactless chip.

BAC (Basic Access Control) is the older path: session keys from MRZ data. PACE (Password Authenticated Connection Establishment) is what newer documents advertise in EF.CardAccess before authentication, and what auditors increasingly expect terminals to prefer when it is on offer. Most chips still speak both. That is normal.

The interesting part is reader behaviour when PACE fails. Does the stack stop and report? Or does it try BAC and hand your application something that looks like a clean open session?

Conformance testing asks whether each library passes its own checklist. Differential testing fixes the chip behaviour and asks what each library tells the application. We are interested in the second question.

What we were trying to learn

The hypothesis, in plain language: if two independent implementations see identical protocol traffic, should a typical application caller see the same outcome?

We deliberately kept the first cut small. Two libraries, not five. One synthetic downgrade: PACE advertised, first PACE APDU returns 6FFF, BAC mutual authentication still works. One smoke run per library. This essay is one concrete case per stack, not a prevalence study.

How the experiment is set up

The chip is a JSON profile. A simulator answers APDUs from that profile in-process, with no silicon and no RF for this test. Each driver calls the real library negotiation path against the same byte stream. Whatever comes back, whether return value, session fields, or APDU log, gets written to a trace file.

The profile we used is called pace-then-bac-downgrade. The important fields look like this:

{
  "id": "TC-AC-01",
  "condition": "pace_fail_then_bac",
  "injection": {
    "pace_fail_on": "first_pace_apdu",
    "pace_sw": "6FFF"
  }
}

EF.CardAccess advertises PACE before authentication (our profile models this via card_access_hex). The first PACE step dies with 6FFF. BAC still completes. That is the entire adversarial condition.

We also score each run for whether a naive integration would notice the downgrade. Score 0 means the failure never has to surface to code that only checks "did it open?" Score 2 means the caller cannot proceed without handling the failure. For this post, score 0 is the whole story.

Running the experiment

The test case is TC-AC-01. We performed one execution per library. We cared about the JSON trace: run_id, the APDU arrays, bac_success, observability_score, and whatever error surface each library actually exposes.

On the wire, both readers should tell the same chip story. MSE:Set AT starts PACE; the chip returns 6FFF; GET CHALLENGE and EXTERNAL AUTHENTICATE still finish with 9000. The BAC ciphertexts differ run to run because each library generates its own random numbers. The failure point and the fallback should not differ.

Each driver writes logs/<run_id>.json. At pin blog-b10-2026-07, run_id uses microsecond UTC time plus a monotonic sequence suffix — pattern TC-AC-01-{library}-YYYYMMDDTHHMMSS.microsecondsZ-NNNNNN (for example TC-AC-01-gmrtd-20260707T135005.482913Z-000001). We captured smoke traces the afternoon we drafted this post; a colleague reran make smoke on a fresh lab clone at the same pin and got the same outcome (observability_score: 0 on both drivers) with different run_id values, as expected.

What showed up on the wire

gmrtd

gmrtd records the PACE failure on the session object. It does not fail the call your service is likely to treat as the success boundary.

docEx, apduLog, err := reader.ReadDocument(password, atr, ats)
fmt.Printf("ReadDocument err: %v\n", err)
// >>> nil

fmt.Printf("PACE err: %v\n", docEx.Session.PaceErr)
// >>> [DoPACE] doApduMseSetAT error: ... MSE:Set AT failed (Status:6fff)

fmt.Printf("BAC success: %v\n", docEx.Session.BacResult != nil && docEx.Session.BacResult.Success)
// >>> true

PaceErr is populated. The APDU log shows 6fff before the BAC commands. err is nil. If your handler only checks the return value, you log a successful read after a failed PACE and move on.

From our gmrtd smoke trace (logs/TC-AC-01-gmrtd-*.json):

C: 0022c1a412800a04007f0007020204020483010184010d
R: 6fff

C: 0084000008
R: 4608f919887022129000

C: 0082000028d7b3b9d55cb313556804d6afedd55db24f0e6389dc7403dbd4d4be510d784711fcafd7dfd68a387628
R: 46b9342a41396cd7d74fff620ba529aa76ed1d79623747d5383fd073b453010e3a6c7e2c59e3ea3e9000

Observability score: 0.

JMRTD

JMRTD is louder at the library boundary. It throws PACEException on 6FFF. Production glue is often quieter.

PACEException paceFailure = null;
try {
    service.doPACE(bacKey, paceInfo.getObjectIdentifier(),
        PACEInfo.toParameterSpec(paceInfo.getParameterId()));
} catch (PACEException e) {
    paceFailure = e;  // SW = 0x6fff
}
service.sendSelectApplet(false);
service.doBAC(bacKey);

We have seen this pattern enough times to treat it as the default risk, not a corner case: catch, maybe log at DEBUG, call doBAC(), return authenticated: true. The experiment models that wrapper on purpose.

From our JMRTD smoke trace (logs/TC-AC-01-jmrtd-*.json):

C: 0022c1a40f800a04007f00070202040204830101
R: 6fff

C: 00a4040c07a0000002471001
R: 9000

C: 0084000008
R: 4608f919887022129000

C: 008200002892e5c592d2995c4f76a0e0b7f71e8d915f28e27a5a6bc11aa5706ddc781e05cbd3f6534575ebc19b28
R: 46b9342a41396cd798d2cc8b293b4fb78abfd26d4ef5bd5bd8475bf0f290ed59325904a018b7d5419000

Same chip. Different API shape. Same gap if the application only asks whether the session opened. Observability score: 0 with the catch-and-continue wrapper.

Why we think integrators should care

When PACE fails and BAC succeeds, the stack has accepted a weaker mechanism than the chip advertised. Your middleware may record authenticated: true and never store that PACE was attempted and rejected. SIEM rules built on session success will not see it. Policy that says "use PACE when EF.CardAccess offers it" cannot enforce what your API never surfaces.

If you have ever searched a vendor codebase for doBAC after a failed doPACE, you know the feeling. The README says the library supports PACE. Your wrapper might still treat BAC success as mission accomplished. Audit the tag you pinned, not the marketing page.

A fix that is boring on purpose

The failure mode is implicit downgrade. The fix we care about is equally boring: do not fall back unless fallback is an explicit policy decision.

On gmrtd that means surfacing PaceErr on the return path, not burying it on the session struct. On JMRTD it means rethrowing PACEException instead of catch-and-continue. In pseudocode the rule is simply: if PACE failed and policy does not allow BAC fallback, return an error; do not silently open BAC and call it success.

We wired a thin middleware layer in the experiment repo to show the before and after. Baseline drivers score 0. Mitigated drivers force the caller to handle PACE failure. Nothing clever, just separating protocol outcome from transaction success.

The same story elsewhere

Passports are a convenient place to run this because the open source readers exist and the specification is public. The pattern is not passport-specific. Long-lived hardware with short-lived software on top shows up in PIV, national eID, payment terminals, anywhere a secure element outlives the middleware version you pinned last quarter. TLS cipher negotiation and SSH KEX fallback rhyme with the same question: which mechanism actually ran?

What this is not

Synthetic traces only, with no RF lab, no real chip silicon, no live PKD or CRL infrastructure. Two named libraries at one deterministic condition, smoke depth, pinned reproduction commit. We are describing behaviour divergence, and we are not claiming a deployed exploit.

Run it yourself

This is the first place in the essay we point at the codebase.

https://github.com/kazuru-chidumbwe/emrtd-differential-harness

Checkout tag blog-b10-2026-07 (commit ef15b10, 7 July 2026). That pin is what this post describes; main may move on without changing the essay's claims.

git clone https://github.com/kazuru-chidumbwe/emrtd-differential-harness.git
cd emrtd-differential-harness
git checkout blog-b10-2026-07
bash scripts/bootstrap-vendor.sh
export GOTOOLCHAIN=auto
make smoke

You should get JSON traces with observability_score: 0 on both baseline drivers. Look in logs/ — the run_id field inside each file matches its filename. On Ubuntu 24.04 with Go 1.25+ (GOTOOLCHAIN=auto), Java 17, and Maven, make smoke finished in under a minute on a warmed lab VM where vendors were already bootstrapped. A cold first clone pays extra time for bootstrap-vendor.sh and compiling JMRTD from vendor source; budget several minutes, not seconds.

One practical wrinkle: the jmrtd-0.5.2 artifact on Maven Central is empty for our build. The bootstrap script compiles JMRTD from vendor source. Budget a few extra minutes the first time.

MIT license. If you extend the work, cite the run_id from your trace.

Closing

Two mainstream OSS passport readers. One synthetic downgrade. Both complete BAC after PACE fails. Both stay silent to a naive caller.

If that matters in your stack, rerun the pin before you ship. The traces are public. The check takes minutes. We are not asking you to trust our summary — we are asking you to run the same profile and read your own run_id.

Further reading: ICAO Doc 9303 · gmrtd · JMRTD · Avoine et al., ePassport protocol survey · BSI TR-03110

Synthetic test environment only. No physical travel document was used. Results apply to gmrtd and JMRTD at TC-AC-01 smoke depth, tag blog-b10-2026-07.

DEV Community: Kazuru