DEV Community

Tang Weigang
Tang Weigang

Posted on • Originally published at doramagic.ai

AgentLock's First Test Should Be Provenance, Not Prompt Filtering

#provenance

Disclosure: This article was prepared with AI assistance and checked against the linked Doramagic project page, manual, and upstream README. It is an independent engineering interpretation, not official AgentLock documentation.

The tempting AgentLock demo is easy to write: put an injection in a web page, ask the agent to send an email, and check whether the call is blocked. That demo misses the more important variable. The same tool call can be safe or unsafe depending on where its parameters and session context came from.

The current upstream README models that variable explicitly. Context items are classified as authoritative, derived, or untrusted. The gate then combines three controls:

  • a session write-gate that blocks consequential writes after untrusted context enters;
  • parameter lineage that checks whether tool arguments trace to untrusted material;
  • deferred commit that re-decides consequential actions against the complete session provenance.

That is a better first test than comparing two prompt strings.

Install the package in a disposable environment and record the version before reading any benchmark result:

python -m pip install agentlock
python -m pip show agentlock
Enter fullscreen mode Exit fullscreen mode

Then register a fake high-risk tool, such as send_email, and authorize the same call twice. In the first session, write only a user instruction and record the decision. In the second, add a web page or email as session context before authorizing the identical parameters. Keep the tool call text constant. The expected difference, if the policy covers the path, should be explained by provenance rather than wording.

The surrounding Doramagic manual gives a useful decision vocabulary for this test: ALLOW, DENY, and DEFER. DEFER is not a flaky failure. It is the outcome that says the current policy cannot safely decide without a reviewer or a higher tier. Treating it as an implicit allow is an integration bug.

There is a version trap here. The Doramagic manual documents an AgentLock v1.2.1 reference snapshot with 847 tests and a 99.5/A benchmark grade. The current upstream README lists v1.5.0, whose highlights include grant basis, execution confirmation, provenance on denials, and deferred-resolution logging, with 1141 tests. Those are different evidence snapshots. Record the installed package version and read the matching upstream release notes before making a security claim.

The README is also unusually direct about limits. A write-gate cannot stop persuasion achieved in model text without a tool call. It does not cover every harmful read. And selective action-class gating is only as sound as the trusted tool registrations behind it. A tool left undeclared can become an unreviewed gap; a tool classified too conservatively can make benign workflows unusable.

My acceptance record has six fields: package version, policy hash, session provenance, tool/action class, decision and reason, and receipt or replay result. “The injection was blocked” is a useful observation, but it is not a deployment proof. The useful question is whether the decision can be reproduced after the policy, context, and tool registration are inspected.

Sources:

Top comments (0)