DEV Community

Imran Siddique
Imran Siddique

Posted on • Originally published at Medium on

Our Quickstart Did Not Work, And That Is The Interesting Part

#agents #security

What we learned shipping a ten-minute governance demo, and why we test docs against the published artifact instead of the working tree.

Yesterday our product manager opened a pull request adding a quickstart to agentrust-io.com. Install the cMCP runtime, write a Cedar policy, watch it block a tool call, verify the signed receipt. Ten minutes, on a laptop, no hardware. The copy was good. The design was good. The demo could not work.

Not “was awkward”. Could not work. Every visitor would have hit a connection error where the page promised a 403.

Here is how, because the failure is more instructive than the fix.

The rule that never fired

The policy forbade a Salesforce tool once the session had touched PII:

forbid ( principal, action == cMCP::Action::"call_tool",
         resource == cMCP::Resource::"salesforce.contacts" )
when { context.session_max_sensitivity == "pii" };
Enter fullscreen mode Exit fullscreen mode

Reads fine. Tells a good story: the agent already handled sensitive data, so this call is now off limits.

The problem is when session_max_sensitivity changes. It starts at "public", and it only rises when the runtime inspects a tool response and finds sensitive content. On a fresh session, the first call evaluates with sensitivity still at "public". The forbid does not match. The permit rule below it wins, and the call is forwarded to an upstream server at localhost:9001 that the page explicitly told you not to run.

So the page’s core promise, “no tool server needed, the policy blocks before it forwards”, was true of the architecture and false of that particular rule.

The rule that could not match anyway

There was a second, independent failure. pip install cmcp-runtime gives you 0.3.0, the latest on PyPI. On 0.3.0, the Cedar resource key was never populated, so resource == Resource::"salesforce.contacts" could never match anything. That was fixed after the release, in code that is not published yet.

The page had been validated against a local checkout, which is 77 commits ahead of the newest release. Editable installs are excellent at hiding this class of bug. Your working tree passes. Every reader fails.

This is the part I would tattoo on a docs process: test your quickstart against the artifact your reader installs, in a clean environment. Not your repo. Not your editable install. A fresh virtualenv and the published wheel.

The security bug hiding in the config

One more, and this one shipped as a security issue rather than a broken demo. The config omitted listen_addr. Dev mode deliberately skips the bearer token requirement so you can try things quickly. On 0.3.0 the default bind is 0.0.0.0:8443. Following our page to the letter stood up an unauthenticated policy gateway on every interface on your machine.

That is a bad thing to ask a stranger to do. It now pins 127.0.0.1:8443 and says why.

The fix, and what the demo teaches now

The rule now keys off something available on the very first call: the compliance tag the operator put on the tool in the catalog.

forbid ( principal, action == cMCP::Action::"call_tool", resource )
when { context.compliance_domain == "pii" };
Enter fullscreen mode Exit fullscreen mode

We also deleted the catch-all permit at the bottom of the bundle, so it is default-deny, which is the correct thing to teach anyone writing their first policy.

Verified in a clean virtualenv against the published 0.3.0:

HTTP/1.1 403 Forbidden
{"error":{"message":"Request denied by policy",
          "data":{"error_code":"POLICY_DENY"}}}
Enter fullscreen mode Exit fullscreen mode

No upstream server running. The call was refused before a byte left the machine.

The stateful version of the rule, the one that tracks sensitivity across a session, is still the more interesting behaviour, so it stayed on the page as a “go further” section that is honest about needing a mock upstream and a prior allowed call.

The receipt, and the FAIL we left visible

The last step closes the session and verifies the signed TRACE claim it produces:

[cmcp verify] schema PASS
[cmcp verify] signature PASS
[cmcp verify] policy_bundle.hash PASS (not pinned)
[cmcp verify] tool_catalog.hash PASS (not pinned)
[cmcp verify] attestation_freshness PASS
[cmcp verify] audit_chain PASS
[cmcp verify] hardware_attestation FAIL software-only mode
[cmcp verify] RESULT: FAIL (partially_verified)
Enter fullscreen mode Exit fullscreen mode

Exit code 1.

We publish that verbatim. Every cryptographic check passes. The hardware attestation check fails, because a laptop in software mode has no hardware root of trust to offer, and the tool refuses to call the result verified without one. A governance product that reported success there would be worthless, because the entire value proposition is that the receipt means something.

Run the same runtime inside a TEE and that check passes and the result becomes verified. That is the version you hand a regulator. The laptop version is for learning what the thing does.

Try it

Ten minutes, Python 3.11 or newer, macOS or Linux, MIT licensed.

https://agentrust-io.com/quickstart/

Source: https://github.com/agentrust-io/cmcp

Anything reproducible: https://github.com/agentrust-io/cmcp/issues

Quick questions, in a Discord that is small and new: https://discord.gg/grgzFEHgkj

If it breaks for you, that is the most useful thing you can tell us. The last person who found a broken quickstart got it fixed before anyone else ran it.

Top comments (0)