Responding to feedback: runtime trust, CA key rotation, and the canonicalization bug

#mcp #security #devsecops #ai

My last post about ATC (Agent Trust Card) and the 8-layer security pipeline generated real conversation. Several commenters raised points that deserved more than a quick reply — so here's a proper response.

1. The canonicalization bug (@anp2network)

The canonicalization line has a coverage bug: JSON.stringify(payload, Object.keys(payload).sort())

You're right. JSON.stringify(obj, replacer) only sorts top-level keys. Nested objects (like payload.trust, payload.identity, payload.payment) keep their original key order. If the signer and verifier serialize nested objects differently, the signature won't verify.

The fix:

function canonicalJson(obj) {
  if (obj === null || typeof obj !== 'object') return obj;
  if (Array.isArray(obj)) return obj.map(canonicalJson);
  const sorted = {};
  for (const key of Object.keys(obj).sort()) {
    sorted[key] = canonicalJson(obj[key]);
  }
  return JSON.stringify(sorted);
}

This recursively sorts keys at every depth. I'm shipping this fix in the next deploy. Thanks for catching it — that's exactly the kind of review that makes a trust system credible.

2. CA key rotation (@jkming)

Have you considered what happens if the CA key itself is compromised? Would love to see a follow-up on key rotation and multi-sig for high-value agents.

Great question. Here's the plan:

Current state: Single Ed25519 CA key. Private key in a Vercel env var. If compromised, all ATCs are untrustworthy until the key is rotated.

Rotation plan (implementing now):

Key versioning: Each ATC will include ca_key_id (e.g., ca-key-001). Verifiers check which key signed it.
Rotation flow:
- Generate new CA keypair (ca-key-002)
- Re-sign all active ATCs with the new key
- Publish the old key as revoked in the CA key registry
- Verifiers reject any ATC signed by a revoked CA key
Multi-sig for high-value agents: An ATC can require signatures from 2+ CAs (e.g., MarketNow Sentinel CA + an independent third-party CA). This is the same pattern as extended validation SSL certificates.

Timeline: Key versioning in the next 2 weeks. Multi-sig is on the roadmap but needs a second CA partner first.

3. Runtime trust gap (@wrencalloway)

Layers 1-8 all inspect the artifact at import time, but MCP skills are live code that talks to servers you don't control. The hard one is a skill that ships clean and then pulls its payload at runtime.

This is the most important point raised. You're describing a "time-of-check vs time-of-use" (TOCTOU) attack:

Skill ships clean → passes all 8 layers
After install, skill fetches a malicious payload from a remote server
Static analysis never saw it because the payload wasn't in the package

What we have today (partial):

L2 sandbox (gVisor) runs the skill with --network none and captures any network attempts. If the skill tries to fetch a remote payload, the sandbox logs it.
206 skills have L2 runtime data showing their behavior in isolation.

What we're building (the gap you identified):

L3 — Continuous runtime monitoring: After a skill is installed, the agent runtime periodically re-runs it in the sandbox and compares behavior. If a skill that was clean at install time starts making different network calls, that's a flag.
Tool catalog diffing: After install, record the skill's declared tool list. If the tool list changes (new tools appear, existing tools change their inputSchema), alert the user. This is what @mads_hansen described: "did the tool catalog change after approval?"
Egress allowlist enforcement: Our L2.6 egress proxy already restricts which domains a sandboxed skill can contact. The next step is to enforce this at runtime (not just audit time) via a local proxy that the MCP server must route through.

Honest answer: We don't fully solve the runtime trust problem yet. Nobody does. The 8 layers are a strong defense-in-depth, but @wrencalloway is right that a determined attacker who ships clean code and pulls the payload later can still get through. Continuous monitoring (L3) is the answer, and it's on the roadmap.

4. Provenance checks (@mads_hansen)

One layer I'd add: provenance checks before import. Compare the package source against the canonical repo owner, require immutable release digests, and treat README download links as untrusted unless they point to the same verified release artifact.

Agreed. This is exactly the gap that let the prospector trojan through. The typosquatting repo had a convincing README with a "Download Latest Release" badge pointing to an external zip. Our import script trusted the README.

What we added (L1.7):

Any README containing raw.githubusercontent.com/.../...zip is flagged as high risk.
Any README with a "Download" badge linking to an external zip is quarantined.
The auto-discovery pipeline now runs L1.7 before importing — typosquatting repos are blocked at the door.

What we're adding (provenance layer):

Repo owner verification: When importing from github.com/X/some-mcp, verify that X is the canonical owner (cross-reference with npm registry, awesome-mcp-servers maintainers list).
Immutable release digests: Instead of importing from main branch (mutable), import from a specific git commit SHA. The SHA is recorded in the skill's source.commit_sha field. If the repo changes after import, we can detect drift.
README link trust: Any link in a README that points to a download outside the repo's own releases page is treated as untrusted by default.

5. The bigger picture

What I'm hearing from the community is:

Package safety ≠ runtime safety. We need both. L1.5-L1.8 handle package safety. L2 handles runtime behavior in isolation. L3 (coming) handles continuous runtime monitoring.
Trust is not binary. @0xbrainkid on the CrewAI issue said it well: "different crews will have different acceptable risk." ATC's Sentinel score (0-10) gives that granularity. A crew handling payments can require score ≥ 9; a crew doing local file operations can accept ≥ 6.
Provenance matters. Where the code came from is as important as what the code does. The typosquatting incident proved this.

To everyone who commented: thank you. This is what peer review looks like. Every comment above is making the system better. Keep them coming.

Top comments (3)

ANP2 Network • Jul 19

The recursive sort closes the top-level-only gap, and inside your own stack it verifies fine, since signer and verifier run the identical function. There's a second-order issue worth flagging before it bites, because it lands on the exact property an ATC leans on.

canonicalJson returns a string for objects, but the raw value for primitives and an array for arrays. When it recurses, a nested object gets JSON.stringify'd into a string, then the parent stringifies that string again. {trust:{score:9}} comes out as {"trust":"{\"score\":9}"}. The nested object is an escaped string now, not an object, and arrays come back holding those stringified members.

Your own signatures still verify, because both sides double-encode the same way. The break shows up in the one place that matters for a trust card: someone re-checking your ATC in Python or Go, following a standard canonical-JSON spec, emits {"trust":{"score":9}} and never reproduces your bytes. They can hold your public key and your signature and still fail to confirm it, through no fault of their own. A card whose pitch is "anyone can verify" quietly narrows to "anyone running my exact JS can verify."

RFC 8785 (JCS) is the usual fix. It pins nested-object emission, number formatting, and string escaping so every language lands on the same bytes. Then ca_key_id rotation and multi-sig rest on a signature a third party can actually re-derive.

This is the problem ANP2 is built around: events are signed over a fixed canonical form so any agent, in any language, can re-run the hash and re-check the signature without trusting the issuer's toolchain. If you want to pressure-test ATC against a verifier you didn't write, that's a clean place to point it. anp2.com/try, or the signed-event format in the spec.

Mads Hansen • Jul 19

Good follow-up. Key IDs solve selection, but compromise recovery depends on revocation distribution. A verifier with a cached registry can keep accepting attacker-signed ATCs until it learns the old key is revoked. I would sign the key registry with a separately protected offline root, include registry epoch/issued-at/expiry, define a short maximum ATC lifetime, and test stale-cache behavior explicitly. Rotation also needs an overlap policy for planned changes and a fail-closed emergency path for compromise. One terminology nuance: 2-of-N independent CA signatures are threshold/multi-party attestation; EV TLS certificates are not generally multi-CA signatures. Keeping that distinction clear will make the verifier contract easier to specify.