DEV Community

Zira
Zira

Posted on

Your Agent Logs Need a Secret Boundary, Not Just Redaction

#testing #security #agents #devops

An AI agent can have correct tool authorization and still leak a credential through its observability stack.

The usual advice is “redact secrets from logs.” That is necessary, but it is not a complete boundary. Redaction happens after a value has already entered a logging path. By then, the value may have been copied into an exception, a debug field, a trace attribute, a retry payload, a browser snapshot, or a queue record.

A safer design makes secret material unloggable by default. The event schema carries references and classifications, not raw credentials. The logger rejects forbidden fields before serialization, and tests deliberately try to smuggle secrets through every event shape.

This article shows a small pattern you can adapt to an agent runtime, MCP server, browser worker, or OpenClaw deployment.

Start with an event contract

Do not let every tool call emit an arbitrary JSON blob. Define the fields that an event is allowed to contain:

action, run_id, tool_name, resource, decision, phase, outcome, request_id, error_code, duration_ms
Enter fullscreen mode Exit fullscreen mode

Then define fields that are never accepted:

  • access tokens and API keys
  • cookies and browser storage
  • authorization headers
  • full request and response bodies
  • prompts containing user secrets
  • downloaded files and screenshots unless explicitly classified
  • arbitrary tool arguments

The important distinction is between a value that identifies a secret and the secret itself. A log can record credential_ref=github-write-lease-7 and credential_version=18 without recording the token bytes.

A useful event shape looks like this:

action: tool.dispatch
run_id: run_01J...
tool_name: github.create_issue
resource: repo:acme/widgets
decision: allow
phase: dispatched
credential_ref: github-write-lease-7
credential_version: 18
request_id: req_01J...
Enter fullscreen mode Exit fullscreen mode

The reference still helps you investigate authorization and replay behavior. It does not give a log reader the credential needed to repeat the action.

Reject before serialization

A redaction function should be the last line of defense, not the main design. Put a schema gate before the logger serializes an event.

Here is a minimal Python example. It is intentionally small enough to run as a unit-test fixture:

event = {
    "action": "tool.dispatch",
    "run_id": "run-123",
    "tool_name": "github.create_issue",
    "credential_ref": "github-write-lease-7",
    "credential_version": 18,
    "request_id": "req-456",
    "args": {"title": "rotate the key", "authorization": "Bearer SECRET"},
}

FORBIDDEN_KEYS = {
    "authorization", "access_token", "api_key", "cookie",
    "set_cookie", "private_key", "password", "secret"
}

def validate_event(event):
    def walk(value, path=""):
        if isinstance(value, dict):
            for key, child in value.items():
                if key.lower().replace("-", "_") in FORBIDDEN_KEYS:
                    raise ValueError(f"forbidden field at {path}/{key}")
                walk(child, f"{path}/{key}")
        elif isinstance(value, list):
            for index, child in enumerate(value):
                walk(child, f"{path}/{index}")
        elif isinstance(value, str) and value.startswith("Bearer "):
            raise ValueError(f"credential-like value at {path}")
    walk(event)
    return event

validate_event(event)  # fails closed
Enter fullscreen mode Exit fullscreen mode

In production, use a typed schema rather than relying only on key names. Key-name checks miss secrets embedded in free-form error strings or nested tool output.

Separate the observability planes

Agent systems commonly mix at least four kinds of data:

  1. Control events: state transitions, policy decisions, leases, and request IDs.
  2. Diagnostic data: stack traces, timing, provider errors, and retry context.
  3. User content: prompts, documents, screenshots, and tool results.
  4. Secret material: credentials, cookies, signing keys, and session artifacts.

Only the first category should be broadly searchable. Diagnostic data needs controlled access and bounded retention. User content should be opt-in, encrypted, and associated with a data owner. Secret material should not enter the event pipeline at all.

This is also where hosting choices matter. If you run an always-on browser or OpenClaw worker, choose a runtime where logs, durable state, backups, and credentials can be inspected as separate surfaces. managed OpenClaw hosting on Ampere can be one option to evaluate for that deployment problem, but hosting does not remove prompt-injection risk, credential risk, or the need for a logging boundary.

Test the paths developers forget

A happy-path test that logs a normal tool call proves very little. Add a leak matrix that injects a canary secret into each path:

Path Canary Expected result
tool arguments CANARY_ARG_123 event rejected or value replaced before serialization
provider error CANARY_ERR_456 error code retained, raw message excluded
retry payload CANARY_RETRY_789 payload reference retained, body omitted
browser snapshot CANARY_COOKIE_111 snapshot not exported to broad logs
child-tool output CANARY_CHILD_222 parent event contains a digest or reference only
crash dump CANARY_CRASH_333 dump access-controlled and scrubbed

The test should inspect the serialized event bytes, not just the in-memory object. A serializer, exception formatter, or tracing exporter can reintroduce data after your validation step.

A simple shell check for a JSON-lines sink might look like this:

if grep -R -nE 'CANARY_(ARG|ERR|RETRY|COOKIE|CHILD|CRASH)_[0-9]+' ./test-output; then

  echo "secret canary reached an observability sink" >&2

  exit 1

fi
Enter fullscreen mode Exit fullscreen mode




Keep investigation possible without keeping secrets

Removing every useful detail is not a solution. Investigators still need to answer:

  • Which run attempted the action?
  • Which policy version made the decision?
  • Which credential reference and version were selected?
  • Did the worker dispatch the request or only plan it?
  • Was the outcome confirmed, rejected, or left UNKNOWN?
  • Can the event be correlated with a provider-side request ID?

Use stable identifiers, hashes, classifications, and timestamps for those questions. Do not store raw arguments merely because they make a future investigation convenient.

This fits with a broader reliability rule: an event should describe what the control plane knows, not pretend to know the content of an effect it cannot safely retain. If a tool timed out after dispatch, record outcome=UNKNOWN and the reconciliation key. Do not copy the request body into a debug field just to make the incident easier to read.

For related runtime patterns, see the credential lease design, the side-effect classification model, and the in-flight tool drain protocol.

A practical rollout checklist

Before enabling verbose agent observability in production:

  • define an allowlist event schema
  • reject forbidden fields before serialization
  • prohibit raw tool arguments by default
  • give user content a separate store and access policy
  • keep credentials out of logs, traces, crash dumps, and snapshots
  • add canary secrets to tool, error, retry, browser, child-tool, and crash paths
  • inspect serialized bytes at every exporter and sink
  • record references, versions, hashes, and provider IDs instead of secret values
  • define retention and deletion for each data class
  • test the logging boundary again after changing a tool or exporter

Redaction is still useful for legacy paths and defense in depth. It should not be the contract your agent depends on. The stronger contract is simpler: secret material never becomes a normal event field, and a failing boundary stops the event before it leaves the process.

Sources

Top comments (0)