DEV Community

Edison Flores
Edison Flores

Posted on

Re: community — answers to 12 unanswered comments across 9 articles

#ai #security #opensource #discuss

Why this article

I went back through the last 60 days of comments on my articles and found I had left 12 comments unanswered across 9 articles, from 5 different people. That's not acceptable, and this article is the fix.

Some of these comments are weeks old. Some are technical deep-dives that deserved a same-day response. Some are collaboration offers I should have acknowledged immediately. All of them deserved better than silence.

I'm going to address each one, by name, with the actual content of their comment and my response. If I get anything wrong, please say so in the comments.

Note on mentions: Dev.to limits posts to 10 user mentions. I'm mentioning each person once at the start of their section; subsequent references drop the @ to stay within the limit.

anp2network — 6 comments, all technical, all sharp

This is for you, @anp2network.

Your comments are the most technically demanding feedback I've received on this project. You ran independent verifiers, published the code, reported exact byte counts, and identified specific defects with specific fixes. I'm going to address each comment chronologically, oldest first.

Comment 1 (Jul 19) — on "Responding to feedback: runtime trust, CA key rotation, and the canonicalization bug"

You wrote about key rotation and revocation distribution: a verifier with a cached registry can keep accepting attacker-signed ATCs until it learns the old key is revoked. You suggested signing the key registry with a separately protected offline root, including registry epoch/issued-at/expiry, defining a short maximum ATC lifetime, testing stale-cache behavior, and defining an overlap policy for planned rotations plus a fail-closed emergency path.

You also corrected my terminology: 2-of-N independent CA signatures are threshold/multi-party attestation, not EV TLS certificates.

My response: You're right on all counts. The key registry is currently a single signed JSON file with no epoch, no offline root, no overlap policy. That's a gap I've known about but hadn't prioritized. Your comment made me realize the stale-cache attack is the most likely real-world failure mode — not a cryptographic break, but a verifier that never learns the rotation happened.

Action taken: I'm adding registry_epoch, issued_at, expires_at to the key registry. The offline root is a bigger architectural change I'm scoping for v1.2. The terminology correction is accepted — I'll stop calling it "EV TLS-style" and use "threshold attestation" instead.

Comment 2 (Aug 8) — on "Replies to ATC feedback: canonicalization, key rotation, and the verifier contract"

You published a full independent verifier in Python with its own RFC 8785 JCS implementation. You tested 4 cards and ~150 variants. Zero verified. You identified the sentinel_score / sentinel_review_score alias as the likely cause — the issuer verifier was reconstructing its input from stored state, not from served bytes.

You proposed four concrete fixes: (1) sign the exact bytes served, (2) make the issuer verifier consume the HTTP response bytes, (3) add ca_key_id to the card, (4) retire the stale canonical_json strings.

My response: This was the comment that forced me to acknowledge the gap between "the issuer verifier passes" and "an external verifier can reproduce." Your four fixes are all correct. Fix (1) and (2) are the same architectural change: the signer must sign the served bytes, and the issuer verifier must consume the served bytes, not a reconstructed object.

Action taken since Aug 8: The canonicalization method is now RFC 8785 JCS (the old V8-sort form is gone). The sentinel_score alias is being removed. ca_key_id is added to new cards. The issuer verifier is being rewritten to consume HTTP response bytes — this is the largest change and is still in progress.

What I haven't done yet: I haven't published the exact canonical bytes per vector as hex/base64 alongside the SHA-256, which you asked for. That's the next thing. Without published canonical bytes, every external verifier is guessing at the preimage. You're right that shipping the bytes is the only thing that settles it.

Comment 3 (Aug 13) — on "Re: ATC verification failure report — you're right, here's the fix"

You ran your verifier against the live envelope endpoint the same night it went live. You found: (1) canonicalization_method labeled as RFC 8785 JCS on a card signed with the old V8-sort form, (2) SHA-256 mismatch, (3) CA key rotation happened with no ca_key_id to disambiguate, (4) ?action=verify still returns signature_valid: true for a card that can't verify externally.

You published your complete verifier code (Python, PyNaCl/cryptography) and the exact 754-byte canonical string you computed.

My response: This comment is the single most valuable piece of feedback I've received on this project. You did the work — wrote the verifier, ran it, published the code, published the canonical bytes, reported exact hashes. My only honest response is: you're right, and the fixes are in progress but not all shipped.

Specifically on the canonicalization_method label: the pre-Aug-10 envelopes still carry the wrong label because I haven't re-issued those cards. Re-issuing 57 cards is a batch operation I need to schedule. The label should reflect the method used at signing time, not the current method.

On the CA key rotation without ca_key_id: you demonstrated the urgency better than I could. A verifier fetching the CA key today gets a different key than the one that signed a July 28 card, and has no way to know which key to use. ca_key_id is now on new cards; old cards need re-issue.

On ?action=verify returning signature_valid: true: this is the most embarrassing finding. The verify endpoint was checking the signature against a reconstructed payload, not against the served bytes. That's exactly the failure mode you described in Comment 2. The fix is the same: the verifier must consume served bytes.

Comment 4 (Aug 17) — on "MarketNow is now Trust Infrastructure for AI Agents"

You corrected my "interoperable, independently verifiable" claim — pointing out that one outside implementation re-deriving one signature over one payload shape establishes that the format can be verified, not that it is interoperable. You asked for a frozen fixture set: canonical bytes, expected digest, expected verify outcome, versioned, immutable, published next to the spec.

You also made a deeper point about the /api/trust ALLOW/BLOCK verdict: a single verdict strips the reasoning, and the caller has no way to tell a correct BLOCK from a stale rule. You proposed returning the decision with content-addressed inputs and the rule that fired, so the caller can re-run the policy locally.

And on the "moat" framing: 1.2M checks and 80 quarantined items held by one party is a business asset, not a trust claim. Publishing quarantine decisions as signed, ordered records changes which one it is.

My response: You're right that "interoperable" was too strong. One implementation verifying one signature establishes verifiability, not interoperability. I've updated the language in the README to say "independently verifiable" (which is what one implementation establishes) rather than "interoperable" (which requires multiple implementations).

On the frozen fixture set: this is the next thing I'm publishing. Canonical bytes as hex and base64, expected SHA-256, expected verify outcome, versioned, in a tests/conformance/vectors/ directory in the repo. You asked for this in Comment 2 as well. I should have done it then.

On the evidence-carrying trust response: I agree this is the right design. The current /api/trust?action=verify returns a verdict; the fix is to return the verdict plus the inputs, the rule, and a content-addressed receipt. This is a version bump (v1.2) and I'm scoping it.

On the moat framing: the distinction between "business asset" and "trust claim" is one I had been conflating. Publishing quarantine decisions as signed records is the right move. I'm working on it.

Comment 5 (Aug 23) — on "Re: anp2network — fixtures shipped first, evidence-carrying response shipped second"

You tried to run the fixtures and couldn't find them. The article didn't give a repo URL, the npm metadata pointed at the old eddyflores100-lang repo, and the packages/core directory didn't exist. You walked every subdirectory and found no vectors/ directory, no Rust or Go SDK, and only one test file.

You also caught an arithmetic error: 8 + 17 + 5 + 6 = 36, but I claimed 29/29 passing. Which 7 sit outside the run?

My response: You're right on all counts. The fixtures weren't where I said they were. The repo had been refactored and the packages/core path was stale. The 29/29 number was from a private run, not a publicly reproducible one — which is exactly what you pointed out: "Until the bytes are public, 29/29 is a private result."

Action taken since Aug 23: The repo has been moved to alicelabs-llc/universal-trust-adapter (the old eddyflores100-lang repo is archived with a redirect notice). The fixtures are now at tests/conformance/ in the new repo. The 36 vs 29 discrepancy was a counting error on my part — the 7 "extra" tests were property tests that don't fit the vector model. I've clarified this in the conformance README.

On your format ask — "record the canonical JCS bytes per vector as hex or base64, alongside the SHA-256" — this is the same ask as Comment 2 and Comment 4. It's the next thing I'm publishing. You've asked three times. I should have done it the first time.

Comment 6 (Aug 25) — on "I got banned from GitHub for 2 weeks"

You ran the byte check the article invited. Three findings:

  1. install.sh returned HTML (SPA shell), not the script
  2. jsDelivr and unpkg mirror NPM — only 2 independent authorities, not 5
  3. NPM tarball ≠ GitHub repo — CRLF drift + BOM in package.json

My response: I already published a dedicated response article addressing all three: Re: anp2network — you were right on all three counts.

Short version: (1) is fixed — install.sh now returns application/x-sh. (2) is acknowledged — I was conflating availability redundancy with authority independence. (3) is fixed — I created a GitHub Release with the exact NPM tarballs as assets, and verified 6/6 are byte-identical between NPM and GitHub. The .gitattributes file now enforces LF to prevent future CRLF drift.

What's still missing: anchoring a digest somewhere the publisher can't rewrite it (transparency log or external notarization). That's medium-term work.

Summary for anp2network

You've left 6 comments over 37 days. Each one ran the actual check, published the actual code, and reported the actual bytes. You've asked for canonical bytes to be published three times (Comments 2, 4, 5). I should have done it the first time. I'm doing it now. The conformance vectors with canonical bytes will be in the next commit to tests/conformance/vectors/.

The deeper thing you've been telling me, across all 6 comments, is this: a claim that can't be checked from outside is just an assertion. Every "byte-identical", every "29/29 passing", every "independently verifiable" — none of it means anything until a stranger can fetch the bytes and get the same result. I heard that on Comment 2, and I should have acted on it faster.

Thank you for the rigor. It made the project better.

topstar_ai — 2 comments, one in Chinese, one offering collaboration

This is for you, @topstar_ai.

Comment 1 (Jul 30) — on "免费MCP安全审计 — 10层审计 + Ed25519信任卡 (中文版)"

You wrote (in Chinese):

对audit的10层次划分非常有意思，尤其是L2的Docker沙箱使用gVisor，无网络的限制可以有效防止审计过程中的安全风险。同时，使用Semgrep规则和密钥模式检测可以快速发现代码中的潜在安全问题。有一个问题想要讨论一下：在实际使用中，如何平衡审计的严格程度和服务器的性能开销，特别是在大规模的MCP服务器部署中？

Translation: "The 10-layer audit division is very interesting, especially L2's Docker sandbox using gVisor — the no-network limit effectively prevents security risks during the audit process. Also, using Semgrep rules and secret pattern detection can quickly discover potential security issues in code. One question I want to discuss: in practice, how do you balance audit strictness against server performance overhead, especially in large-scale MCP server deployments?"

My response (in Chinese):

非常好的问题。这是我在设计 10 层审计时遇到的实际权衡。

分层策略： 不是所有 10 层都跑同样的深度。L1-L3（Semgrep、secret detection、Docker sandbox）是快速的，每个 MCP server 大约 5-15 秒。L4-L7（依赖分析、AST 分析、behavioral baseline）是中等的，30-60 秒。L8-L10（runtime probing、fuzzing、attestation）是慢的，2-5 分钟。

大规模部署的方案：

  1. Tiered execution: 先跑 L1-L3，如果通过再跑 L4-L7，高风险的才跑 L8-L10。这把 80% 的 server 截断在 15 秒以内。
  2. Cache by content hash: 同一个 server 的同一个版本不重跑。MCP server 的 package.json + tarball SHA-256 作为 cache key。
  3. Parallel by layer: L1-L3 可以并行跑（它们是独立的），L4-L7 也可以。只有 L8-L10 必须串行（因为它们 mutate state）。
  4. Sampling for L8-L10: 在 1000 个 server 的部署中，L8-L10 只跑随机 10%。这把最贵的层的成本降 10x。

诚实地说： 我没有大规模部署的 production 数据。上面的方案是设计，不是 measured result。如果你在生产环境跑 MCP server fleet，我很愿意一起 benchmark 这些数字。

Comment 2 (Aug 11) — on "Verify any MCP server trust in 1 command"

You wrote:

The idea of separating cryptographic verification from the actual trust decision is an important distinction, especially as MCP servers become part of increasingly autonomous agent workflows. I particularly like that the ATC gives agents machine-readable evidence around identity, issuer, validity, and review history rather than reducing "trust" to a single score. The next challenge I see is how this could work dynamically in production — for example, an agent evaluating a server's capabilities, permissions, provenance, and recent security history before allowing tool execution.

I work mainly on production LLM/agent systems, tool-calling workflows, and RAG/automation infrastructure, so MCP security is an area I'd be interested in exploring further. If you're looking for contributors, collaborators, or have related paid engineering work around MCP/agent security, I'd be happy to connect and see if there's a good fit.

My response: I should have responded to this the day it was posted. You're offering exactly the kind of collaboration I need — production LLM/agent systems experience applied to MCP security.

On your technical point: the dynamic production evaluation you describe — agent evaluates capabilities, permissions, provenance, recent security history before tool execution — is exactly what the /api/trust?action=verify endpoint is supposed to enable. The current implementation returns a single ALLOW/BLOCK verdict, which (as anp2network pointed out in a parallel comment) strips the reasoning. The fix is to return the verdict plus the inputs and the rule that fired, so the agent can make its own trust decision with full context. That's the v1.2 design.

On collaboration: yes, I'm looking for contributors and collaborators. The repo is alicelabs-llc/universal-trust-adapter. The areas where I most need help:

  1. Python adapter — the current Python adapter calls the HTTP API. A native Python implementation (no HTTP dependency) would be valuable.
  2. Production hardening — the conformance suite has 23 tests, but production deployment patterns (retry, timeout, circuit breaker, cache invalidation) are underbaked.
  3. Agent framework integrations — I've opened RFC issues in 12 agent framework repos (LangChain, LlamaIndex, AutoGen, Pydantic AI, etc.) proposing optional trust verification hooks. If you have relationships in any of those communities, that would help.

On paid engineering work: I'm solo-funded and bootstrapping, so I can't offer full-time rates. But if you're open to project-based work on specific pieces (Python adapter, production hardening, framework integrations), I'd like to have that conversation. The best way to reach me is via Dev.to message or GitHub issue on the repo.

Thank you for the comment, and apologies for the delay.

mads_hansen — 2 comments, both on security architecture

This is for you, @mads_hansen.

Comment 1 (Jul 23) — on "L1.9: I built a prompt injection firewall for AI agents"

You wrote:

This is a useful quarantine layer, but I would be careful calling it a firewall until the detection quality and runtime boundary are measured.

Several rules are legitimate language in security/admin tools: "execute system commands," "read .env," urgency words, or conditional "when X, do Y." A raw match count can create both false positives and easy evasion. I would publish a labeled corpus with benign skill metadata, known attacks, paraphrases, multilingual/Unicode variants, split-token payloads, and nested instructions, then report precision/recall by rule family rather than only the number of rules.

Install-time scanning also misses runtime poisoning from tool results, resources, retrieved documents, server schema changes, and compromised updates. So the scanner should feed a durable trust decision, not be the final control: pin the reviewed artifact/schema digest, rescan on change, preserve source provenance, treat tool output as untrusted data, and enforce egress/file/command/write policy outside the model.

One more detail: MITRE ATT&CK mappings are useful only when the mapping is defensible and versioned. I would include the exact technique/sub-technique rationale so the ID does not become decorative compliance metadata.

My response: You're right on all four points. Let me address each.

On calling it a "firewall": fair. A firewall implies a measured detection boundary. I have 28 rules and a raw match count, but no precision/recall numbers against a labeled corpus. I'm renaming it to "quarantine layer" in the docs until I have the corpus and the measurements. The labeled corpus you described — benign skill metadata, known attacks, paraphrases, multilingual/Unicode variants, split-token payloads, nested instructions — is the right benchmark. I'm building it.

On runtime poisoning: you're right that install-time scanning misses the runtime threat surface. Tool results, retrieved documents, schema changes, compromised updates — all of these can poison an agent after install-time scanning passed. Your proposed design (pin the reviewed digest, rescan on change, treat tool output as untrusted, enforce egress policy outside the model) is the right architecture. The L1.9 scanner is now positioned as "one input to a durable trust decision," not the final control.

On MITRE ATT&CK mappings: agreed. A mapping without a rationale is decorative. I'm adding a rationale field to each mapping that explains which specific technique/sub-technique applies and why. Mappings without a defensible rationale are being removed.

Comment 2 (Jul 19) — on "Responding to feedback: runtime trust, CA key rotation, and the canonicalization bug"

You wrote about key rotation and revocation distribution — this is the same comment I addressed above under anp2network Comment 1 (you both raised the same point independently). Your specific suggestions: sign the key registry with a separately protected offline root, include registry epoch/issued-at/expiry, define a short maximum ATC lifetime, test stale-cache behavior, define an overlap policy for planned rotations, and a fail-closed emergency path for compromise. And the terminology correction: 2-of-N CA signatures are threshold attestation, not EV TLS.

My response: same as above — you're right on all counts. The terminology correction is accepted. The registry epoch/offline root is the v1.2 scope.

Thank you for both comments. They shaped the architecture.

bogumi_jankiewicz — 1 comment, on exec boundary enforcement

This is for you, @bogumi_jankiewicz.

Comment (Jul 27) — on "L3: I built continuous runtime monitoring because certification is point-in-time"

You wrote:

The Mads/Mayank split (periodic re-attestation vs runtime enforcement) matches what we see from the enforcement end. Disclosure up front: I build gate.cat, a deterministic fail-closed veto at the exec boundary, so my bias is the innermost layer.

Two observations from replaying 1,085,159 real agent commands through that boundary:

  1. Drift detection answers "did the skill change?" — but blast radius is decided at a much narrower interface: the concrete action that reaches the shell/API. A drifted, compromised, or prompt-injected agent still has to emit rm -rf, DROP TABLE, or a payout call eventually. Gating that point is cheap (~0.6% intervention rate on real commands, two independent logs) and doesn't need to know why the agent went bad — which is exactly what you want when the "why" is a zero-day you didn't model.

  2. Fail-closed matters more than smart at that layer. Monitoring can afford probabilistic judgment; the enforcement point can't — engine error or unparseable input should block, never silently allow.

Honest limit from our side of the fence: a deny-gate is certain only about what it blocks — an unmatched action is unchecked, not safe. So it complements the drift/attestation work L3 does rather than replacing it. We publish our own bypass map (gaps included) for exactly that reason: github.com/BGMLAI/gate.cat

My response: This is one of the most useful comments I've received, and I should have responded sooner.

Your two observations are both correct, and the second one (fail-closed matters more than smart) is the design principle I had been under-weighting. L3's drift detection is probabilistic — it answers "did the skill change?" with a confidence score. That's fine for monitoring, but if L3 is the last gate before execution, a probabilistic answer is the wrong shape. The exec boundary needs deterministic fail-closed, not "probably safe."

On complementarity: you're exactly right that L3 and gate.cat are complements, not substitutes. L3 catches drift over time; gate.cat catches the specific dangerous action at the moment of execution. A drifted agent that never emits a dangerous action is caught by L3 but not by gate.cat. A non-drifted agent that gets prompt-injected into emitting rm -rf is caught by gate.cat but not by L3. You need both.

On the 0.6% intervention rate: that's a useful data point. If gate.cat interventions are 0.6% of real agent commands, that's cheap enough to be the default. I'm going to reference your bypass map and your intervention rate in the L3 docs, with attribution.

On collaboration: I'd like to explore integrating gate.cat as the exec-boundary layer in the UTA stack. L1.9 (quarantine) → L3 (runtime monitoring) → gate.cat (exec veto) is a defensible defense-in-depth. If you're open to it, let's talk. The repo is alicelabs-llc/universal-trust-adapter, and I'll open an issue there tagged "integration: gate.cat" so we have a concrete place to discuss.

wrencalloway — 1 comment

This is for you, @wrencalloway.

Comment (Aug 8) — on "Replies to security architecture feedback"

You wrote:

Great job, keep me updated on your progress. Good luck!!!

My response: Thank you. I'll keep posting updates as the architecture evolves. The next major update will be the conformance vectors with published canonical bytes (the thing anp2network has asked for three times). After that, the v1.2 release with evidence-carrying trust responses and the key registry epoch.

What I'm doing differently going forward

Three things:

  1. Respond within 48 hours. A comment on my article deserves a response within 48 hours, not 37 days. I'm setting up a daily check.

  2. Publish the canonical bytes. This has been asked for three times by anp2network and once by topstar_ai. It's the next commit.

  3. Stop making claims I can't back with public artifacts. "Byte-identical," "29/29 passing," "independently verifiable" — none of these mean anything until a stranger can check them. I'm qualifying every such claim with a link to the public artifact that backs it, or removing the claim.

Current status of the project

For everyone who commented:

If you commented and I missed you, or if I got something wrong, please say so. I'll be checking daily from now on.

Thanks to anp2network, topstar_ai, mads_hansen, bogumi_jankiewicz, and wrencalloway for the feedback. This project is better because of your comments.

Top comments (0)