I maintain two open-source Magento modules that must agree:
- a generator (module-ucp) that publishes a store's UCP profile at
/.well-known/ucp - a validator (module-aeo-audit) that checks that profile
For six weeks they disagreed, and the validator was the one that was wrong. It told store owners that a correct profile had no signing keys.
What changed
UCP (Universal Commerce Protocol) ships dated releases. The 2026-08-25 release has this breaking change: signing_keys[] is removed from the profile, and keys[] (a JWK Set) is the only canonical field.
{
- "ucp": { "version": "2026-04-08", ... },
- "signing_keys": [ { "kty": "EC", "crv": "P-256", "kid": "k1" } ]
+ "ucp": { "version": "2026-08-25", ... },
+ "keys": [ { "kty": "EC", "crv": "P-256", "kid": "k1" } ]
}
The generator moved on 1 September. The validator shipped two releases after that, on 11 September and 15 October, and still read only signing_keys.
The two messages
UCP protocol version "2026-08-25" is not one this audit knows
(2026-04-08, 2026-01-11, 2026-01-23) — it may be newer than this release
No signing_keys declared — RFC 9421 HTTP Message Signatures will not validate
The first one is fine. "I don't know this version" is an honest answer from a validator that is behind.
The second one is the bug. A missing field does not look unknown. It looks absent. So the validator reported a false problem with full confidence.
Why CI was green
// Every fixture in the test suite looked like this:
$profile = [
'ucp' => ['version' => '2026-04-08', /* ... */],
'signing_keys' => [[ 'kty' => 'EC', 'crv' => 'P-256', 'alg' => 'ES256', 'kid' => 'k1' ]],
];
The tests proved the code matched my reading of the spec. Nothing in them could notice that the spec had moved.
The fix
Read the new field first, fall back to the old one, and say so when a profile mixes them:
$keysField = null;
$signingKeys = [];
foreach (['keys', 'signing_keys'] as $field) {
$candidate = $profile[$field] ?? ($ucp[$field] ?? null);
if (is_array($candidate) && isset($candidate['keys'])) {
$candidate = $candidate['keys']; // a JWK Set object {"keys": [...]}
}
if (is_array($candidate) && $candidate !== []) {
$keysField = $field;
$signingKeys = array_values($candidate);
break;
}
}
if ($keysField === 'signing_keys' && is_string($version)
&& strcmp($version, '2026-08-25') >= 0) {
$warnings[] = 'Profile says 2026-08-25 but publishes signing_keys[]; the field is keys[] since 2026-08-25';
}
Dated versions like 2026-08-25 sort correctly as strings, so strcmp is enough here.
Plus two new tests: an old profile on 2026-04-08 with signing_keys (warns "superseded", still counts the key), and a 2026-08-25 profile that still uses signing_keys (warns "use keys[]").
What I'd do earlier next time
- Treat a missing field as "maybe renamed". Before you report "X is missing", check the names X had in other spec releases.
- Add a contract test between generator and validator. Generate a profile with one module, validate it with the other, in CI. Then a change on one side breaks the build on the other. I haven't built this yet; it's next.
- Keep "unknown version" a warning. That one choice kept a stale validator from failing every current store.
How do you keep a validator in step with a spec that ships dated releases? Pinned fixtures, a scheduled job against the latest tag, something else?
Originally published at angeo.dev. Both modules are MIT: module-ucp, module-aeo-audit.
Top comments (3)
"The tests proved the code matched my reading of the spec" — that's the sentence. Green CI is a claim about your fixtures, not about the world. I hit the same shape with two systems that had to agree: a generator and a validator drifting apart silently, and the validator reporting a false problem with full confidence. The fix I'd add to yours: version-pin the spec in the validator and fail loudly on unknown versions instead of validating against a stale reading. Your first message ("I don't know this version") was honest; the second was confident and wrong. Do you now treat unknown fields as warnings rather than absences?
Thank you. “Green CI is a claim about your fixtures” says it better than I did.
To your question: not in a general way yet. The fix is narrow. The validator reads keys[] first, falls back to signing_keys[], and warns when a profile mixes them. The general rule I took from it: before reporting “X is missing”, check the names X had in other spec releases.
On failing loudly I see it a bit differently. Unknown version stays a warning on purpose. This validator runs on live stores. If it failed hard on an unknown version, one stale release would fail every current store. That is the same false confidence, only louder.
But your point still lands. When the version is unknown, the validator should not trust its own “missing” checks. So the idea I’m now looking at: unknown version → “missing” findings drop to warnings and say “may be renamed in a newer release”. The honest first message should make the second one less sure of itself.
The other fix is the contract test between generator and validator in CI. Not built yet, it’s next.
How did you catch the drift in your two systems?
The boring way: a user reported it. My "validator" was the SES API's 200 OK, my "generator" was the actual inbox — and they drifted for a week while I trusted the first one. The tell wasn't in the code I was looking at, it was in the system I wasn't: delivery events. Once I pulled bounce/complaint/delivery event streams next to the send logs, the lie was obvious — accepted is not delivered, same shape as your confident second message. The general rule I took from it, matching yours: before you trust a component's self-report, find the one signal it can't fake. Are you running that contract test between generator and validator in CI yet, or still staging it?