I maintain two open-source Magento modules that must agree:
- a generator (module-ucp) that publishes a store's UCP profile at
/.well-known/ucp - a validator (module-aeo-audit) that checks that profile
For six weeks they disagreed, and the validator was the one that was wrong. It told store owners that a correct profile had no signing keys.
What changed
UCP (Universal Commerce Protocol) ships dated releases. The 2026-08-25 release has this breaking change: signing_keys[] is removed from the profile, and keys[] (a JWK Set) is the only canonical field.
{
- "ucp": { "version": "2026-04-08", ... },
- "signing_keys": [ { "kty": "EC", "crv": "P-256", "kid": "k1" } ]
+ "ucp": { "version": "2026-08-25", ... },
+ "keys": [ { "kty": "EC", "crv": "P-256", "kid": "k1" } ]
}
The generator moved on 1 September. The validator shipped two releases after that, on 11 September and 15 October, and still read only signing_keys.
The two messages
UCP protocol version "2026-08-25" is not one this audit knows
(2026-04-08, 2026-01-11, 2026-01-23) — it may be newer than this release
No signing_keys declared — RFC 9421 HTTP Message Signatures will not validate
The first one is fine. "I don't know this version" is an honest answer from a validator that is behind.
The second one is the bug. A missing field does not look unknown. It looks absent. So the validator reported a false problem with full confidence.
Why CI was green
// Every fixture in the test suite looked like this:
$profile = [
'ucp' => ['version' => '2026-04-08', /* ... */],
'signing_keys' => [[ 'kty' => 'EC', 'crv' => 'P-256', 'alg' => 'ES256', 'kid' => 'k1' ]],
];
The tests proved the code matched my reading of the spec. Nothing in them could notice that the spec had moved.
The fix
Read the new field first, fall back to the old one, and say so when a profile mixes them:
$keysField = null;
$signingKeys = [];
foreach (['keys', 'signing_keys'] as $field) {
$candidate = $profile[$field] ?? ($ucp[$field] ?? null);
if (is_array($candidate) && isset($candidate['keys'])) {
$candidate = $candidate['keys']; // a JWK Set object {"keys": [...]}
}
if (is_array($candidate) && $candidate !== []) {
$keysField = $field;
$signingKeys = array_values($candidate);
break;
}
}
if ($keysField === 'signing_keys' && is_string($version)
&& strcmp($version, '2026-08-25') >= 0) {
$warnings[] = 'Profile says 2026-08-25 but publishes signing_keys[]; the field is keys[] since 2026-08-25';
}
Dated versions like 2026-08-25 sort correctly as strings, so strcmp is enough here.
Plus two new tests: an old profile on 2026-04-08 with signing_keys (warns "superseded", still counts the key), and a 2026-08-25 profile that still uses signing_keys (warns "use keys[]").
What I'd do earlier next time
- Treat a missing field as "maybe renamed". Before you report "X is missing", check the names X had in other spec releases.
- Add a contract test between generator and validator. Generate a profile with one module, validate it with the other, in CI. Then a change on one side breaks the build on the other. I haven't built this yet; it's next.
- Keep "unknown version" a warning. That one choice kept a stale validator from failing every current store.
How do you keep a validator in step with a spec that ships dated releases? Pinned fixtures, a scheduled job against the latest tag, something else?
Originally published at angeo.dev. Both modules are MIT: module-ucp, module-aeo-audit.
Top comments (1)
"The tests proved the code matched my reading of the spec" — that's the sentence. Green CI is a claim about your fixtures, not about the world. I hit the same shape with two systems that had to agree: a generator and a validator drifting apart silently, and the validator reporting a false problem with full confidence. The fix I'd add to yours: version-pin the spec in the validator and fail loudly on unknown versions instead of validating against a stale reading. Your first message ("I don't know this version") was honest; the second was confident and wrong. Do you now treat unknown fields as warnings rather than absences?