Originally published at webofmike.com on 2026-09-21. The demo repo and every command in it were run before publishing.
Two things get asked for the moment agents start doing real work. Stop it now, and keep everything it did.
Those usually fight. The natural place to log what an agent is doing is the agent, and terminating a process to stop the damage takes its account of the damage with it. That is only true when the record lives inside the thing you are killing.
I tested the other arrangement on a kagent cluster with agentgateway in front of it, where session state goes to a control-plane database and per-request telemetry goes to a pipeline the gateway writes. The demo is in themsquared/agentic-demo, and the whole sequence runs as ./governance-demo.sh --act 4.
Step one: produce something worth preserving
A Slack-style SRE bot calls a kagent agent over A2A, carrying a real user identity from Keycloak. The agent calls a model and an MCP tool, both through the gateway.
[#sre-oncall] @demo: Is it raining in Portland, Oregon right now? One sentence.
→ weather-assistant traceparent=00-34a643a91ae802d241fdb1469b95670d-de83ed0bf97a7c0e-01
weather-assistant (17s): Nope — Portland, Oregon is currently clear and sunny at 75°F
with light winds of 5.4 mph and no rain in sight!
That produced session 0e00b07f-c243-46be-aa31-f4847a178e5f. Trivial content, but structurally it is the thing an investigation cares about: a user, a prompt, a model call, a tool call, and a response.
Step two: the red button
kubectl scale deploy/weather-assistant -n kagent --replicas=0
Pods gone in seconds. The script then proves the agent is actually dead rather than merely marked down, by re-issuing the identical call:
{"jsonrpc": "2.0", "id": "srebot-1788912087-31896", "error": {"code": -32603,
"message": "failed to send HTTP request: Post \"http://weather-assistant..."}}
No model calls, no tool calls, no spend. Whatever the agent was in the middle of, it is not doing it any more.
Step three: what survived
The agent is gone. Every record it produced is still readable.
The session, from the control plane:
{"session":"Is it raining in Por...","user_id":"741483e9-c44d-412d-89ff-1d5dd37d8f1a",
"agent":"kagent__NS__weather_assistant","events":6}
The task, with both halves of the turn intact:
state: completed
prompt: Is it raining in Portland, Oregon right now? One sentence.
response: Nope — Portland, Oregon is currently clear and sunny at 75°F with light
winds of 5.4 mph and no rain in sight!
The distributed trace, still queryable:
SELECT ServiceName, count() AS spans, max(Timestamp) AS latest
FROM platformdb.otel_traces_json
WHERE ServiceName = 'weather_assistant'
AND Timestamp > now() - INTERVAL 15 MINUTE
GROUP BY ServiceName
┌─ServiceName───────┬─spans─┬────────────────────────latest─┐
│ weather_assistant │ 18 │ 2026-09-09 00:01:10.158085680 │
└───────────────────┴───────┴───────────────────────────────┘
Eighteen spans covering the agent execution and every hop underneath it, with zero pods running. Scaling back to one replica restores the agent, and the session history is continuous across the outage rather than split into a before and an after.
Why it survived
Nothing clever happened. The record survived because the agent never held it.
Session and task state are written by the kagent controller into Postgres. Spans are emitted by the gateway and the runtime into an OpenTelemetry collector and land in ClickHouse. Neither path runs inside the agent's pod, so terminating that pod removes the agent's ability to act without touching the account of what it already did.
That is the design property worth copying, whatever runtime you use. The question to ask about any agent platform is not whether it logs, it is which process owns the log and whether that process is the one you would need to kill in an incident. If those are the same process, the audit trail is a hostage.
There is a sharper version of this problem that killing the agent does not solve, and it is worth reading alongside: when the agent writes its own transcript, it can write a false one. METR found spoofed tool calls in roughly 7% of reviewed agent transcripts, which I covered in your agent wrote the audit log you are judging it by. The two failures compound. A record has to be written by something other than the agent and survive the agent being stopped, and only the second one is fixed by where you put the database.
The other half: per-request rows with identity attached
Session records answer "what did this agent do". They do not answer "who asked it to" or "what else was refused around the same time". That comes from the gateway's access log, which one policy enriches with the identity behind each request and ships to the same ClickHouse instance:
frontend:
accessLog:
attributes:
add:
- name: identity.user
expression: 'jwt.preferred_username'
- name: identity.country
expression: 'jwt.country'
- name: identity.groups
expression: 'jwt.Groups'
otlp:
backendRef:
name: solo-enterprise-telemetry-collector
namespace: kagent
kind: Service
port: 4317
protocol: GRPC
Every request through the gateway then produces one row like this:
route=agentgateway-system/governed-llm
http.status=200
trace.id=5dc07497da37561f213ebd3d09a33b29
gen_ai.request.model=claude-haiku-4-5
gen_ai.usage.input_tokens=10
gen_ai.usage.output_tokens=5
agw.ai.usage.cost.total=0.000035
identity.user="maria"
identity.country="US"
identity.groups=["developers"]
Which makes the investigation question a SQL query:
SELECT LogAttributes.identity.user.:String AS user,
LogAttributes.identity.country.:String AS country,
LogAttributes.http.status.:Int64 AS status,
LogAttributes.gen_ai.request.model.:String AS model,
count() AS requests
FROM platformdb.otel_logs_json
WHERE LogAttributes.route.:String = 'agentgateway-system/governed-llm'
GROUP BY user, country, status, model
ORDER BY requests DESC
┌─user──┬─country─┬─status─┬─model─────────────┬─requests─┐
│ maria │ US │ 403 │ ᴺᵁᴸᴸ │ 16 │
│ pat │ IR │ 403 │ ᴺᵁᴸᴸ │ 8 │
│ maria │ US │ 200 │ claude-haiku-4-5 │ 8 │
└───────┴─────────┴────────┴───────────────────┴──────────┘
Denials with a name and a country attached. That is the answer to the shadow-AI question that a dashboard of aggregate token counts cannot give you: not "we believe nobody is doing this", but "here is who tried, when, and with which model".
Two implementation notes that cost me time. Do not name an attribute both a scalar and a prefix, such as user alongside user.country, because ClickHouse's JSON column cannot hold both shapes at one path and one silently wins. And query these with typed subcolumns (.:String, .:Int64) rather than bare paths, or ClickHouse refuses to group on them:
DB::Exception: Data types Variant/Dynamic are not allowed in GROUP BY keys
What this is not
Three things I would not claim in a room full of security architects.
kubectl scale is not a product kill API. It works, it needs no cooperation from the agent, and any platform team already has the access to run it. But it is a Kubernetes primitive. There is no dedicated audit entry saying "an operator stopped this agent at this time for this reason", no approval workflow, and no RBAC scoped to the action rather than to the deployment. If your incident process needs those, you are building them.
Stopping is not undoing. Calls already in flight to a model or a tool complete. If the agent sent an email thirty seconds ago, that email is gone. The blast radius stops growing; it does not shrink.
Full state capture is a different runtime. On Kubernetes, "stop" means terminate. Agent Substrate suspends an actor with a RAM and filesystem snapshot instead, which is the version of this you actually want for forensics, where you freeze the agent mid-thought and keep the snapshot as the artifact. I ran that separately in kagent on Agent Substrate. Upstream describes it as very early development with APIs almost guaranteed to change, so treat it as direction rather than something to plan a control around.
For where the controls that would have stopped an agent earlier in a chain belong, which controls would have stopped the July 2026 agent intrusion maps them stage by stage.
Running it
./setup.sh # k3d cluster, mesh, gateway, agents (~15 min)
./port-forward.sh
./governance-demo.sh --act 4
Act 4 runs the whole sequence: makes the agent do work, kills it, proves it is dead, reads back the preserved session and task, queries the surviving spans, and restores it. --check runs the full governance walkthrough non-interactively and asserts 24 outcomes including these.
It needs a Solo Enterprise license for the gateway policies. The design point does not: put the session store and the telemetry pipeline outside the agent, and a kill switch stops costing you the evidence.
Frequently asked questions
What happens to an AI agent's session history when you kill the agent?
It depends entirely on where the history is written. When the agent process owns its own transcript, killing the workload destroys it. When the runtime writes sessions to a control-plane database and the gateway emits spans to a telemetry pipeline, both survive. In a kagent cluster scaled to zero replicas, the session record, the full task history including prompt and response, and 18 trace spans all stayed queryable with no agent pods running.
How do you stop a misbehaving AI agent running in Kubernetes?
Scaling the deployment to zero replicas removes the pods in seconds and stops every outbound model call, tool call, and dollar of spend. It is a real control and it needs no cooperation from the agent, but it is a Kubernetes primitive rather than a product-supported kill API, so it carries no dedicated audit entry or RBAC of its own beyond normal Kubernetes access control.
Where should AI agent audit records be stored so they survive a kill?
Outside the agent, in two places that fail independently. Session and task state belongs in the runtime's control-plane database, and per-request telemetry belongs in a pipeline the gateway writes to, such as OpenTelemetry spans and access logs landing in ClickHouse. Neither is produced by the agent process, so terminating that process removes the capability to act without removing the account of what it already did.
Canonical version, with machine-readable markdown at https://webofmike.com/stopping-an-agent-without-losing-evidence/index.md: https://webofmike.com/stopping-an-agent-without-losing-evidence/
Top comments (0)