observability sinks are becoming the audit contract for AI agents

You built an agent that can execute trades.

Now prove what it did.

Not in the demo. Not in the Slack thread where everyone was excited. Not in the application log that the same agent runtime can write, redact, rotate, or accidentally forget.

Prove it to the regulator.

Prove which model was used, which prompt was active, which tools were called, which data was read, which policy allowed the action, which human approved the exception, which account paid for the run, and why the agent chose one path instead of another.

That is the part most agent demos skip.

And it is the part that will matter first in fintech, infrastructure, security, healthcare, and any company brave enough to let agents touch production.

My take is simple: observability sinks are becoming the audit contract boundary for AI agents.

Not dashboards.
Not "we log everything."
Not a vendor trace view inside the same hosted agent product.

An operator-owned place where agent behavior is streamed, stored, verified, and governed independently from the agent runtime itself.

application logs were built for a different world

Traditional observability assumes a fairly boring relationship between code and behavior.

Humans write code. The code runs. The system emits logs, metrics, and traces. When something breaks, engineers inspect the signals and reconstruct what happened.

That model is still useful. It is not enough for agents.

Agents are different in four annoying ways.

First, the execution path is dynamic. A normal service may branch, but the shape of the branch is encoded in the program. An agent can choose a tool, inspect the result, call another tool, rewrite its own plan, ask another agent, retry with a broader query, and only then mutate state.

Second, the important thing is often semantic, not mechanical. It is not enough to know that POST /orders happened at 10:42. You need to know the agent believed the user had approved the order, that the risk policy returned green, that the position size was under the limit, and that no override was used.

Third, hosted agents create custody problems. If the agent runs inside a vendor platform, and the only audit trail also lives inside that vendor platform, the operator has a weak story. "Trust us, our supplier has the logs" is not the same as an independent record.

Fourth, agents can spend money invisibly. A bad loop can burn model tokens, cloud APIs, search calls, database queries, and human review time. Cost is no longer just a monthly dashboard problem. It is part of the runtime behavior that needs attribution.

So when people say "just add tracing," I get the instinct, but it is incomplete.

Tracing tells you how work flowed.

An audit contract tells you what must be recorded, where it must land, who controls it, how it is verified, how long it is retained, and what happens when the agent cannot emit it.

That is a different bar.

the sink matters more than the screenshot

The word "sink" sounds boring because it is boring. Good.

A sink is where telemetry lands: an S3 bucket, an append-only database, a data lake, an observability backend, a ledger-like event store, or a streaming system that fans out to several of those.

For agent auditability, the important part is not that the sink exists. It is who owns it and what contract it enforces.

If the agent runtime owns the only durable record, the agent runtime is marking its own homework.

That may be fine for local experiments. It is not fine for regulated workflows.

The serious version looks more like this:

• The operator owns the sink.
• The agent cannot delete or rewrite events after emission.
• Events are forwarded in near real time, not uploaded later as a convenience.
• Each event carries enough structure to reconstruct the decision path.
• Events are signed or chained so tampering is detectable.
• Missing telemetry is treated as a policy failure, not a debugging inconvenience.

This is the mental shift: observability is not just a debugging aid anymore.

For autonomous systems, observability becomes part of the permission model.

An agent should not be allowed to execute a high-risk action unless it can also produce the audit events required for that action.

No audit trail, no mutation.

That rule sounds harsh until the first incident.

event sourcing is the better metaphor

Most logging strategies are optimized for diagnosis after a system misbehaves.

Agent audit trails need to be optimized for reconstruction.

That is why event sourcing is a better metaphor than logging.

In an event-sourced system, the important facts are represented as a sequence of events. The current state is something you derive from the history, not something you trust by itself.

That maps very naturally to agents.

Instead of "the agent completed task 123," the audit stream should contain the decision history:

• task accepted
• identity selected
• policy context loaded
• data source queried
• model invoked
• tool call proposed
• tool call approved or rejected
• external system mutated
• cost attributed
• result verified

Some of those events belong to the agent. Some belong to the control plane. Some belong to the tool gateway. Some belong to the human approval system.

That is exactly why the sink should be outside the agent.

The agent is one witness. It is not the court.

what belongs in the audit stream

I would not try to record every token forever. That gets expensive, creepy, and often useless.

But I would be strict about the events that define accountability.

At minimum, for production agents, I want:

Run identity. Agent id, run id, conversation id, tenant, environment, model, model provider, prompt or policy version, and the human or system that initiated the work.

Authority. The identity used for each tool call, scopes granted, policy decision, approval id if applicable, and whether the action was read-only or mutating.

Decision points. The agent's plan changes, tool selection, rejected alternatives when available, confidence signals if used, and the reason a risky action was allowed.

Tool I/O boundaries. Not necessarily every payload in full, because privacy still matters, but enough hashes, references, schemas, and redaction markers to prove what category of data crossed the boundary.

Cost. Tokens, provider calls, cloud resources, search calls, database queries, and anything else that can become a bill. Cost attribution should not be a separate archaeology project.

Outcome. What changed, where it changed, how it was verified, and whether a human reviewed the final result.

This is not exotic. It is the same boring engineering discipline we already apply to payments, ledgers, deployments, and security-sensitive admin actions.

Agents just force us to apply it earlier.

OpenTelemetry helps, but it is not the contract

OpenTelemetry is probably the right default vocabulary for a lot of this work.

The ecosystem is already moving toward generative AI and agent semantic conventions: model calls, tools, agent spans, conversation identifiers, token usage, and the kind of trace structure you need to see multi-step behavior.

That is good.

Standard naming matters because nobody wants each framework inventing a private schema for "tool call with LLM context and cost metadata."

But I would be careful not to confuse a telemetry standard with an audit contract.

OpenTelemetry can describe events and traces. It does not, by itself, decide:

• which events are mandatory before a production action
• which sink is authoritative
• whether the event stream is tamper-evident
• how long records must be retained
• which fields must be redacted or tokenized
• which missing span blocks the workflow
• who is allowed to query the evidence later

Those are platform and governance decisions.

The best architecture will probably use OpenTelemetry-style instrumentation for portability, then enforce stronger organization-specific rules at the sink.

Common format. Operator-owned contract.

That is the useful combination.

cost is part of the audit trail

One underrated reason this matters: agents can spend money in ways humans do not notice until the invoice arrives.

A human engineer usually has friction. They get bored. They stop. They ask someone. They go to lunch.

An agent with a retry loop has none of those virtues.

It can call a model hundreds of times, fan out across tools, run cloud queries, create test environments, scrape documents, invoke evaluators, and ask another agent to inspect the result.

Each individual step may look small.

The chain may be expensive.

So the sink should not only answer "what did the agent do?"

It should answer:

• what did this run cost?
• which tool or model created the cost?
• which user, team, product, or customer should own it?
• which policy allowed the spend?
• did the spend produce a verified outcome?

If the observability sink can attribute cost at the same boundary where it records action, it becomes a budget enforcement point.

That is much better than discovering a token bonfire in a monthly report.

fintech will not accept vibes as evidence

This is where fintech gets interesting.

Financial systems already know that logs are not enough. Payments, ledgers, approvals, reconciliations, fraud reviews, access decisions, and trade workflows all need durable evidence.

The moment an agent enters that world, it inherits the same expectations.

If an agent drafts a customer support answer, maybe a normal trace is enough.

If an agent changes a fraud rule, approves a chargeback, liquidates collateral, moves money, tunes a trading strategy, or sends data to a regulator, the story changes.

The organization needs to explain the decision chain.

Not because regulators are anti-AI. Because regulated businesses have always had to prove control over material decisions. AI does not remove that requirement. It makes the evidence harder to collect.

The SEC has been publicly wrestling with AI and predictive analytics in investment contexts. The CFTC has reminded registered entities that AI usage does not erase existing obligations. The PRA's model risk principles already push banks toward governance, validation, controls, and accountability for models.

You do not need to overclaim that every regulator has issued agent-specific audit-sink rules.

They have not.

The safer and more useful claim is this: regulated firms already need control, evidence, explainability, and accountability around automated decision-making. Autonomous agents increase the pressure on those existing obligations.

That pressure has to land somewhere.

I think it lands in the observability sink.

what I would build first

I would not start with a grand "agent ledger" project.

That will become a platform initiative with a steering committee and six months of schema arguments.

Start with one high-risk agent workflow and make the audit boundary explicit.

For example:

• an agent that opens infrastructure pull requests
• an agent that investigates production incidents
• an agent that reviews fraud cases
• an agent that performs cost optimization recommendations
• an agent that touches customer data

Then define the contract.

What events must exist before the agent can act?
Which events are emitted by the runtime, the gateway, the policy engine, and the human approval tool?
Where is the authoritative sink?
Who can write to it?
Who can read it?
What gets signed?
What gets redacted?
What happens when emission fails?

The last question is the one that separates serious systems from demos.

If the sink is down, does the agent keep working?

For low-risk read-only actions, maybe yes.

For production mutations, financial actions, customer data export, or security-sensitive changes, I think the default should be no.

Observability failure becomes control-plane failure.

That is the whole point.

the skeptical part

Observability sinks are not magic.

They do not make a bad policy good.
They do not prove the model reasoned correctly.
They do not prevent prompt injection by themselves.
They do not replace least privilege, human approvals, sandboxing, evals, rollback design, or boring access control.

They also create their own risks.

An audit sink can become a sensitive data swamp if teams dump full prompts, retrieved documents, customer identifiers, secrets, and model outputs into it without discipline.

So the sink needs privacy engineering too: redaction, tokenization, field-level access, retention classes, and a clear distinction between evidence and hoarding.

Still, I would rather have that problem than the alternative.

Because the alternative is an agent that can act, spend, mutate, and coordinate with other agents while the operator has only whatever logs the runtime felt like leaving behind.

That is not governance.

That is optimism with timestamps.

the punchline

Agents are pushing observability out of the debugging corner and into the governance layer.

For normal software, logs help engineers understand what the code did.

For autonomous agents, observability sinks help organizations prove what work was delegated, what authority was used, what decisions were made, what costs were incurred, and what changed as a result.

That proof has to live somewhere the operator controls.

It has to be structured enough to reconstruct the run.

It has to be durable enough to survive an incident.

And for serious workflows, it has to be mandatory enough that an agent cannot simply act first and explain later.

That is why I think the next boring piece of agent infrastructure will matter a lot:

your bucket, your event stream, your audit database, your retention policy, your cryptographic proof.

The agent can do the work.

The sink keeps it honest.

sources

• OpenTelemetry semantic conventions
• OpenTelemetry: AI agent observability - evolving standards and best practices
• OpenTelemetry GenAI agent span conventions
• Martin Fowler: Event Sourcing
• Honeycomb: Agentic AI observability for production agents
• Claude Help Center: Access audit logs
• Google Cloud: Agent Registry audit logging
• SEC: Artificial Intelligence and the Future of Investment Management
• CFTC staff advisory on the use of AI by registered entities and registrants
• Bank of England PRA: SS1/23 model risk management principles for banks

To test my projects, I use Railway. If you want $20 USD to get started, use this link.