Audit Trails for DevOps AI Agents: Attribute Every kubectl, AWS Call, and Commit to a Run and a Human

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

The question RBAC and tracing can't answer

An audit trail for a DevOps AI agent is a record that, for any change the agent made, answers four questions from the target system's own logs: which agent, which run, which human authorized it, and what evidence the model acted on. You get it by propagating a single run ID into three places the agent already touches — the Kubernetes User-Agent header, the AWS STS SourceIdentity, and git commit trailers — and writing one append-only ledger record per run that ties them together.

Here is why that matters. You've done the right things: the agent runs under a least-privilege ServiceAccount, every tool call is traced with OpenTelemetry, and writes go through an approval gate. Then an auditor, or your own postmortem, asks: "On the 14th at 02:13, a Deployment in payments was scaled to zero. Who authorized that?" The Kubernetes audit log says system:serviceaccount:agents:ops-agent-write. Your OTel traces say the agent proposed a scale-down and someone clicked approve. Nothing connects the two. The audit log has no run ID; the trace has no proof it was this API call. You end up correlating on timestamps, which is exactly the kind of answer that fails a SOC 2 CC7 control review.

The fix is not more logging. It is putting the same identifier in every system on the write path, so the correlation is a string match instead of a guess.

Step 1: mint one run ID and make it the trace ID's twin

Every agent invocation gets a ULID at the moment it starts — before the model sees a prompt. Everything downstream carries it:

# run_context.py — one ID to bind them all
import hashlib
import os
from ulid import ULID
from opentelemetry import trace

class RunContext:
    def __init__(self, agent: str, trigger: str, approver: str | None = None):
        self.run_id = str(ULID())            # e.g. 01J9C4V0X3K9Y2R7QH8P6TN5MB
        self.agent = agent                   # "ops-agent"
        self.version = os.environ["AGENT_VERSION"]
        self.trigger = trigger               # "alert:PaymentsHighErrorRate"
        self.approver = approver             # filled in by the approval gate
        self.prompt_sha = None

    def record_prompt(self, system_prompt: str) -> None:
        self.prompt_sha = hashlib.sha256(system_prompt.encode()).hexdigest()[:16]

    def user_agent(self) -> str:
        # Kubernetes stores this verbatim in audit events.
        return f"{self.agent}/{self.version} run={self.run_id} approver={self.approver or 'none'}"

    def tag_span(self) -> None:
        span = trace.get_current_span()
        span.set_attribute("agent.run_id", self.run_id)
        span.set_attribute("agent.approver", self.approver or "")

Why a separate ID instead of reusing the OTel trace ID? Because the trace ID is 32 hex characters that nobody will type into an Athena query at 03:00, and because trace retention is usually 7 to 30 days while audit retention is measured in years. The run ID goes onto the span as an attribute, so a trace search still finds it; the ledger below is what outlives the trace.

Step 2: Kubernetes — the User-Agent channel

The Kubernetes audit log records the User-Agent request header in every event, and setting it requires no extra RBAC. That makes it the cheapest attribution channel you have. With the Python client:

from kubernetes import client, config

def k8s_client(ctx: RunContext) -> client.ApiClient:
    config.load_incluster_config()
    api = client.ApiClient()
    api.user_agent = ctx.user_agent()   # sets the default User-Agent header
    return api

apps = client.AppsV1Api(k8s_client(ctx))
apps.patch_namespaced_deployment_scale(
    name="checkout", namespace="payments",
    body={"spec": {"replicas": 0}},
)

The resulting audit event carries everything you need:

{
  "kind": "Event",
  "level": "RequestResponse",
  "verb": "patch",
  "user": {"username": "system:serviceaccount:agents:ops-agent-write"},
  "userAgent": "ops-agent/1.4.2 run=01J9C4V0X3K9Y2R7QH8P6TN5MB approver=[e-mail]",
  "objectRef": {"resource": "deployments", "subresource": "scale",
                "namespace": "payments", "name": "checkout"},
  "responseStatus": {"code": 200},
  "requestReceivedTimestamp": "2026-09-14T02:13:41.220Z"
}

Now the auditor's question is one line:

# Every API-server write made by run 01J9C4V0X3K9Y2R7QH8P6TN5MB
jq -c 'select(.userAgent | test("run=01J9C4V0X3K9Y2R7QH8P6TN5MB"))
       | select(.verb | IN("create","update","patch","delete"))
       | {t: .requestReceivedTimestamp, verb, ns: .objectRef.namespace,
          res: .objectRef.resource, name: .objectRef.name, code: .responseStatus.code}' \
  /var/log/kubernetes/audit.log

Make sure your audit policy captures the agent SA at RequestResponse level for write verbs. The policy stanza from the RBAC post already does this; the only addition is that userAgent is present at every level, including Metadata, so even cheap policies keep the attribution.

Two caveats worth being honest about. First, User-Agent is set by the client, so it is attribution, not authentication. Authentication is the ServiceAccount token; the header just tells you which run used it. A compromised agent process could lie in the header, but it could not become a different SA, which is why the two layers are complementary. Second, kubectl has no flag for a custom user agent. If your agent shells out to kubectl, you lose this channel; that's one more reason to make the agent call the API through a library or an MCP server that sets the header, and for the gateway to stamp it on the agent's behalf.

If you also need the API server to record the human as a first-class identity, the harness (never the agent) can impersonate: send Impersonate-User: ops-agent-write plus Impersonate-Extra-approver: [e-mail], and the audit event gains an impersonatedUser block with that extra field. That requires the impersonate verb on the harness's credential, which is a real privilege — scope it with resourceNames to the one SA it may become and keep it out of the agent's own Role.

Step 3: AWS — SourceIdentity and the session name

CloudTrail has two fields built for this, and both survive role chaining. When the agent assumes its role, set RoleSessionName to the run ID and SourceIdentity to the approver:

import boto3

def aws_session(ctx: RunContext) -> boto3.Session:
    sts = boto3.client("sts")
    creds = sts.assume_role(
        RoleArn="arn:aws:iam::123456789012:role/ops-agent-write",
        RoleSessionName=ctx.run_id,                      # max 64 chars; ULID is 26
        SourceIdentity=ctx.approver or "unapproved",    # immutable for the session
        DurationSeconds=900,
    )["Credentials"]
    return boto3.Session(
        aws_access_key_id=creds["AccessKeyId"],
        aws_secret_access_key=creds["SecretAccessKey"],
        aws_session_token=creds["SessionToken"],
    )

SourceIdentity is the important one. Once set on a session it cannot be changed, it is copied into every session that chains from it, and CloudTrail writes it into userIdentity.sessionContext.sourceIdentity on every event the session makes. The role's trust policy has to permit it, and you can use the same policy to refuse sessions that don't carry an approver:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::123456789012:role/agent-harness"},
    "Action": ["sts:AssumeRole", "sts:SetSourceIdentity"],
    "Condition": {
      "StringLike": {"sts:SourceIdentity": "*@example.com"}
    }
  }]
}

That condition turns attribution into enforcement: the write role literally cannot be assumed without a human identity attached. Pair it with the approval gate so the approver's email is only populated after a real approval, and an "unapproved" session gets AccessDenied at STS before it can touch anything.

Querying it back is an Athena one-liner against your CloudTrail table:

SELECT eventtime, eventname, awsregion,
       useridentity.arn AS session_arn,
       useridentity.sessioncontext.sourceidentity AS approver,
       json_extract_scalar(requestparameters, '$.instanceId') AS instance
FROM cloudtrail_logs
WHERE useridentity.arn LIKE '%/ops-agent-write/01J9C4V0X3K9Y2R7QH8P6TN5MB'
  AND readonly = 'false'
ORDER BY eventtime;

The session ARN embeds the run ID because that is what you put in RoleSessionName, so a LIKE on the ARN suffix finds every mutating call from that run. The same pattern covers the AWS waste reclamation agent: each EBS delete is now traceable to a run and an approver without any application-side logging at all.

Step 4: git — trailers, enforced in CI

If your agent changes infrastructure the way it should — by opening pull requests, not running kubectl — then git is a fourth audit system, and it already has a convention for structured metadata: commit trailers.

git -c trailer.ifexists=addIfDifferent commit -q -F - <<EOF
chore(payments): scale checkout to 0 during incident INC-2291

Agent-Run: 01J9C4V0X3K9Y2R7QH8P6TN5MB
Agent: ops-agent/1.4.2
Approved-By: [e-mail]
Prompt-SHA: 3f9a1c2e8b7d4a05
Evidence: https://grafana.example.com/explore?run=01J9C4V0X3K9Y2R7QH8P6TN5MB
EOF

Trailers are parsed by git interpret-trailers --parse and searchable with git log --grep='Agent-Run: 01J9C4'. What makes them an audit control rather than a convention is a required check on the agent's PRs that fails when a trailer is missing:

# .github/workflows/agent-commit-policy.yml
name: agent-commit-policy
on: { pull_request: { branches: [main] } }
jobs:
  trailers:
    if: github.event.pull_request.user.login == 'ops-agent[bot]'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - name: Require attribution trailers on every agent commit
        run: |
          set -e
          for sha in $(git rev-list origin/main..HEAD); do
            body=$(git log -1 --format=%B "$sha")
            for key in Agent-Run Approved-By Prompt-SHA; do
              echo "$body" | grep -Eq "^$key: \S+" \
                || { echo "::error::$sha missing $key trailer"; exit 1; }
            done
            echo "$body" | grep -Eq '^Approved-By: [^ ]+@example\.com$' \
              || { echo "::error::$sha Approved-By is not a human identity"; exit 1; }
          done

Because the check runs only on the bot's PRs and main is protected, an agent commit without a human approver cannot merge. Argo CD then deploys a commit whose metadata already names the run and the approver; the Kubernetes audit event for that sync is attributed to Argo, but the git trailer closes the loop.

Step 5: the ledger that outlives everything else

Traces expire, audit logs rotate, and the person who approved a change leaves the company. The ledger is one JSON record per run, written when the run ends, to storage that cannot be modified:

{
  "run_id": "01J9C4V0X3K9Y2R7QH8P6TN5MB",
  "agent": "ops-agent", "version": "1.4.2", "model": "claude-sonnet-5",
  "trigger": "alert:PaymentsHighErrorRate",
  "started": "2026-09-14T02:11:03Z", "ended": "2026-09-14T02:14:20Z",
  "prompt_sha": "3f9a1c2e8b7d4a05",
  "approver": "[e-mail]",
  "approval_ref": "slack:C0AGENTS/p1757815999123456",
  "tool_calls": [
    {"tool": "prom_range_query", "args_sha": "9c0e…", "ok": true},
    {"tool": "k8s_scale_deployment", "args": {"ns": "payments", "name": "checkout", "replicas": 0}, "ok": true}
  ],
  "writes": {"kubernetes": 1, "aws": 0, "git": 1},
  "transcript_ref": "s3://agent-transcripts/2026/09/14/01J9C4V0X3K9Y2R7QH8P6TN5MB.jsonl",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}

Store it in an S3 bucket with Object Lock in compliance mode and a retention period matching your audit policy; nobody, including root, can shorten it. Read-only tool calls are recorded by argument hash rather than raw arguments, both to keep the record small and because query text can contain the very things the secrets post says must never be persisted. Write calls keep their full arguments, since those are the ones an auditor will ask about.

If your agents sit behind an MCP gateway, the gateway is the natural writer of this record: it already sees every tool call from every agent, and it can refuse any call whose request lacks a run ID header. That turns "every run is in the ledger" from a convention into an invariant.

Answering the auditor in three queries

With all four channels populated, the 02:13 scale-to-zero question resolves like this:

1. Kubernetes audit log — the jq filter on userAgent returns one patch on deployments/scale in payments, run 01J9C4V0X3K9Y2R7QH8P6TN5MB, approver [e-mail].
2. Ledger — the record for that run shows the trigger alert, the approval Slack permalink, the prompt hash, and the transcript pointer that holds the model's stated reasoning.
3. Git and CloudTrail — searched by the same ID, they show either the matching PR and no AWS writes, or they show something the ledger doesn't, which is the finding you actually wanted to catch.

That third case is the real payoff. Once every system carries the run ID, a nightly job can diff the ledger's declared writes against what the audit logs actually recorded for that run. A Kubernetes write with no matching ledger entry, or a CloudTrail event under a session name that no ledger record knows about, is an agent acting outside its harness. That is the alarm you build the whole thing for.

Limits to state plainly

• Attribution is not authorization. User-Agent and trailers are self-reported. RBAC, the STS trust condition, and branch protection are what actually stop an unattributed action; the identifiers make the stopped and the allowed actions explainable.
• The ledger records what the harness saw. If a tool runs a subprocess that talks to another API on its own credentials, that path is invisible unless you thread the run ID through it too. Audit the tool implementations, not just the agent.
• Model reasoning is not evidence. The transcript shows what the model said; the audit logs show what happened. Keep both, but trust the second.
• Retention costs money. Full transcripts at RequestResponse audit levels add up. Hash the reads, keep the writes, and set the Object Lock period to your compliance requirement rather than "forever."

None of this is exotic. It is the same discipline you would apply to a human operator with a shared break-glass role, applied to a process that generates its own commands. The agent gets to act only when a human's name rides along with every call, and the target systems, not the agent, are what remember it.

---

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.