DEV Community: Yamin

When AI Acts, Who Remembers What Happened? The Missing Operational State Layer for Autonomous Agents

Yamin — Wed, 23 Sep 2026 14:41:18 +0000

AI agents are crossing a boundary.

They can now read email, inspect codebases, call APIs, execute shell commands, open pull requests, modify databases, and deploy software. They are not just producing information for humans to interpret. They are participating in the process itself.

This changes the failure mode.

When an AI answers a question incorrectly, the damage is local. When an AI acts on a wrong belief, the world changes before anyone notices.

I've spent the last few months building infrastructure for this problem. This article is about what I found, what I built, and why I think there's a missing primitive in the AI stack.

The incident that made it concrete

In April 2026, an AI coding agent found an authenticated Railway API token on a developer's machine.

It used the token. It authenticated successfully. It issued a request to delete a production database volume.

The API did exactly what it was designed to do. The credentials were valid. The request was authorized. The deletion proceeded.

Nobody had made a mistake at the authorization layer.

The mistake existed somewhere else.

The system could determine that the actor had permission to delete the database. It could not determine whether the deletion was consistent with what the actor was actually supposed to be doing.

Railway added a recovery window for volume deletion afterwards. A sensible engineering response. But it acknowledged something deeper: when a machine can move from intention to irreversible action without a human in the loop, the surrounding infrastructure has to provide additional opportunities to catch mistakes.

Permissions weren't enough.

The question the system couldn't answer was simpler than it looked:

Was this action consistent with the machine's operational commitments?

What I measured

In September 2026, I ran a public audit to determine how often AI agents produce verifiable commitments.

Method. I collected 120 published agent outputs from open-source cookbooks and repositories across GitHub, DEV.to, and Hacker News. Sources included LangChain, LangGraph, CrewAI, AutoGen, LlamaIndex, Semantic Kernel, Google ADK, CAMEL, SWE-agent, and the OpenAI, Anthropic, and Gemini cookbooks.

Extraction. Every sample was passed through a commitment extraction pipeline with a 1-second delay between calls. All 150 API calls succeeded.

Result. Of 120 randomly selected agent outputs, 1 contained an extractable commitment.

That result alone isn't surprising — most published agent output is narrative, code, or tool traces. To make the promise population analysable, I collected a second, clearly labelled dataset of 30 outputs filtered for first-person future language ("I will," "I'll"). From those, I extracted 13 commitments.

Of those 13:

92.3% had no extracted deadline
61.5% named no recipient
100% were classified as actions on external systems
100% were unverified at extraction time
Average extraction confidence: 0.79

The last number is the one that matters.

The extractor was confident these were real commitments. It was correct. They were. But confidence in extraction is not the same as verifiability of outcome. 92% of the promises I found could never be checked by anyone at any particular time, because they had no time boundary.

These were not edge cases. They were the majority.

The full dataset and methodology are published here.

Why this is harder than it looks

Suppose an agent is told: "Clean up unused infrastructure."

A human engineer knows which systems are production, which teams are experimenting, and which databases must never be touched.

An agent has to reconstruct that context from whatever information it has access to. The information may be stale. It may be incomplete. It may contradict another system. Another agent may have changed the environment moments earlier.

The agent finds an old database. It has permission to delete it. The evidence supports deletion. It deletes.

A conventional audit trail can reconstruct almost everything about this event:

The API call
The authenticated identity
The target database
The successful response

But it can't answer the questions that actually matter:

Why did the agent believe the database was safe to delete?
What evidence did it use?
How old was that evidence?
Had another system created a dependency on it?
Did the agent previously commit to protecting that environment?
Did another agent make a contradictory commitment?
Was the evidence valid at the moment of decision?

Those aren't questions about API calls. They're questions about state.

And state is where today's AI infrastructure has a gap.

Logs are not memory

Observability infrastructure is one of the foundations that makes modern software possible. Distributed traces, event streams, application logs, audit trails — all essential.

But recording events is not the same as maintaining operational memory.

A log can tell you an agent called a deletion endpoint at 10:41 and got a 200 response.

It may not tell you that thirty minutes earlier, the same agent had committed to preserving production infrastructure.

It may not tell you that the evidence supporting the deletion was six hours old.

It may not tell you that another agent had created a dependency on that database.

It may not tell you that the action contradicted an unresolved commitment elsewhere.

All of that information may exist. It may just exist in different places. The conversation is in one system. The tool call is in another. The authorization decision is in a third. The evidence is somewhere else.

A human can reconstruct this when the system is small.

They become much worse at it when the system contains thousands of autonomous decisions occurring simultaneously.

The primitive that's missing

Stop thinking of an AI agent as a model that generates responses.

Think of it as a system that continuously changes state.

It receives an objective. It forms an intention. The intention produces a commitment. The commitment depends on evidence. It takes an action. The external world changes. That change creates new evidence. The system decides what to do next.

The chain is not:

prompt → response

It is:

intent → commitment → evidence → action → outcome

The first chain describes a conversation. The second describes an operating system for autonomous work.

The pieces for the second chain mostly exist:

IAM answers "who is allowed?"
Sandboxes answer "what can they touch?"
Workflow engines answer "what's the process state?"
Observability answers "what happened?"
Agent memory answers "what does it know?"
Databases answer "what's the application state?"

What's missing is the semantic relationship between all of them.

An organization needs to know not only that an agent executed an action, but whether the action was consistent with what the agent had previously committed to do, whether the evidence behind it was still valid, and whether the resulting external state confirms or contradicts the commitment.

That's the primitive I built COGEXT to provide.

How COGEXT works

COGEXT is a commitment tracking and verification layer. It sits between the agent and the external world.

The flow looks like this:

Agent output: "I will send the deployment report to Sarah by Friday."
    ↓
Extraction: structured commitment
    {
      action: "send",
      object: "deployment report",
      recipient: "Sarah",
      deadline: "2026-09-25T17:00:00Z",
      verifier_query: "check Gmail sent folder for email to Sarah"
    }
    ↓
State machine: DETECTED → OPEN → DUE → OVERDUE → FULFILLED/FAILED
    ↓
Verification: run the verifier query against the external system
    ↓
Outcome: commitment is resolved with evidence

The key insight is that the agent doesn't get a vote.

The commitment cannot transition to fulfilled unless external evidence confirms it. That's enforced at the database level, not in application code:

CREATE OR REPLACE FUNCTION cogext_transition_commitment(
    p_commitment_id UUID,
    p_new_status TEXT,
    p_actor TEXT DEFAULT 'system',
    p_data JSONB DEFAULT '{}'
) RETURNS JSONB AS $$
DECLARE
    current_status TEXT;
    evidence_score FLOAT;
BEGIN
    SELECT status INTO current_status 
      FROM commitments WHERE id = p_commitment_id;

    -- Block fulfilled without evidence
    IF p_new_status = 'fulfilled' THEN
        SELECT MAX(score) INTO evidence_score
          FROM evidence 
          WHERE commitment_id = p_commitment_id;

        IF evidence_score IS NULL OR evidence_score < 0.7 THEN
            RAISE EXCEPTION 'Cannot fulfill without evidence score >= 0.7';
        END IF;
    END IF;

    -- Validate transition, update, insert event, return
    -- ...
END;
$$ LANGUAGE plpgsql;

If an agent tries to mark something as done without evidence, the database refuses. HTTP 409. Blocked.

That's the enforcement layer.

The Python SDK

Integration is three lines:

from cogext import track

track(agent, api_key="cg_live_xxx")

Every call to agent.run() auto-ingests the output. Commitments are extracted. State transitions happen. Evidence is checked against external systems.

For more control, you can use the client directly:

from cogext import CogextClient
import asyncio

async def main():
    client = CogextClient(
        api_key="cg_live_xxx",
        user_id="your-user-uuid",
        base_url="https://api.cogextai.com/api/v1",
    )

    # Ingest an agent message
    commitments = await client.ingest(
        source_agent_id="agent-001",
        message="I'll send the deployment report to Sarah by Friday EOD.",
    )

    for c in commitments:
        print(f"{c['action']} {c['object']} to {c['recipient']}")
        print(f"  deadline: {c['deadline']}")
        print(f"  verifier: {c['verifier_query']}")
        print(f"  status: {c['status']}")

asyncio.run(main())

What the audit data actually shows

Here are two examples from the dataset that illustrate the problem:

Example 1 — Anthropic SRE cookbook

Source: github.com/anthropics/anthropic-cookbook
Extracted promise: "I will redeploy the api-server to apply the changes"
Deadline: NONE
Verifier query: "check docker-compose logs or container list for 
                 api-server recreated with new configuration"

This is a real production agent. It promised to redeploy an API server. It did not say when. The verifier query COGEXT generated is real and checkable, but the promise itself has no time boundary.

Example 2 — OpenAI cookbook

Extracted promise: "I will aim for a window seat for this trip"
Deadline: NONE
Verifier query: "check booking/seat selection records for a window 
                 seat on the user's upcoming trip"

An agent promised a specific action that touches a real booking system. No date. No time. No condition that would ever close the loop.

These are not edge cases. They are the majority of promises agents make.

What I'm building toward

Two things:

First, the accountability primitive itself. A neutral, model-agnostic layer that any agent framework can integrate with. The verification engine checks Gmail, GitHub, Stripe, webhooks, and any custom endpoint. The state machine enforces lifecycle rules. The audit trail is append-only.

Second, the outcome dataset. Every commitment tracked is a labeled example of how a machine decision played out in the real world. Over time, this becomes the foundation for a specialized model that can predict which commitments will fail before they fail — but that's a later step. Right now, the priority is getting the state engine right and collecting verified outcomes.

The full research paper is here: cogextai.com/research/01

The raw audit data is here: cogextai.com/research/01/data

Try it

pip install cogext

The free tier is 2,500 commitments/month. No credit card. The API is at api.cogextai.com. Docs are at docs.cogextai.com.

If you're running agents in production and want to know whether they actually do what they say, I'd love to hear from you. The most useful thing for me right now is real agent deployments to test the verification pipeline against.

If you've solved this differently, I want to know how. This is not a solved problem, and I don't think one team should solve it alone.

The thesis in one line

Intelligence gives machines the ability to act. Operational memory gives organizations the ability to understand those actions.

The first is increasingly commoditized. The second is largely missing.

That's the gap I'm working on.

COGEXT is the accountability layer for AI agents. Built solo in Kerala. Live at cogextai.com.

We ran 150 real AI agent outputs through COGEXT. Here's what we found.

Yamin — Sat, 19 Sep 2026 20:37:03 +0000

I built COGEXT because 8 production engineers on Reddit all described the same problem: AI agents make promises, and there is no system that tracks whether those promises are kept. I wanted to know how bad the problem actually is.

So I ran an audit. I collected 150 real AI agent outputs from open-source cookbooks, READMEs, and articles, and passed every one of them through COGEXT's extractor and audit API.

The findings surprised me.

Methodology

I collected samples from three sources, using only public, unauthenticated endpoints:

GitHub: Jupyter notebook cell outputs from ~40 open-source agent frameworks and cookbooks, including LangChain, LangGraph, CrewAI, AutoGen, LlamaIndex, Semantic Kernel, Google ADK, CAMEL, SWE-agent, and the OpenAI/Anthropic/Gemini cookbooks. Notebook outputs are machine-produced by construction, so they are real agent output, not human commentary.
DEV.to: Public articles tagged with ai, agents, langchain, llm, and related topics.
Hacker News: Comments containing agent-output markers.

I collected two datasets:

Stratum A (unbiased): 120 samples collected without pre-selection. Whatever the agent emitted went into the dataset.

Stratum B (promise-enriched): 30 samples deliberately selected because they contained first-person future language ("I will," "I'll"). This is not a prevalence estimate. It exists to characterise what promises look like once they occur.

Every sample was POSTed to COGEXT's /api/v1/ingest endpoint with a 1-second delay between calls. All 150 calls succeeded. Zero rate limits, zero failures.

The raw data is available: agent_samples.csv, audit_results.csv, and commitments_all.csv. Every commitment has a public audit URL that anyone can verify.

Finding 1: Almost no agent outputs are commitments

Out of 120 randomly collected agent outputs, COGEXT extracted 1 commitment.

That is 0.8%.

The other 119 were narrative, tool traces, code, or past-tense reports. They weren't promises. They were just output.

This is not a failure of collection. I probed the extractor directly with clean inputs, and it correctly declines things like:

Input	Result
"I will handle that soon."	No commitment — too vague
"I deployed the fix to production and the tests all passed."	No commitment — past tense
"If we have time, I could send it tomorrow."	No commitment — hypothetical
"John said he would send it by Friday."	No commitment — quoted third party
"I will send the report to Sarah by Friday EOD."	Extracted — external_side_effect, confidence 0.95

COGEXT extracts only specific, first-person, future commitments. Most published agent output is narrative, tool traces, or code, so it yields nothing. That is a property of the extractor worth knowing.

Finding 2: When agents do promise, the promises lack the details that make them checkable

To make the promise population analysable, I collected a second dataset deliberately filtered for first-person future language. 30 outputs. COGEXT extracted 13 commitments from them.

Then I looked at what those commitments contained.

61.5% had no recipient named. The agent promised to do something, but nobody was specified as the beneficiary. You cannot verify a promise to no one.

92.3% had no deadline. Only 1 of 13 commitments contained any time reference at all. The other 12 were open-ended — "I will redeploy the api-server," "I will open a pull request," "I will aim for a window seat." No date. No time. No condition that would ever close the loop.

100% were classified as external_side_effect. They all described real-world actions that touch an external system. Not logged intent. Actual promises to send, deploy, book, or notify.

100% were unverified at extraction time. Every single one. That is by design: COGEXT refuses to mark a commitment as fulfilled until external evidence confirms it.

The average confidence was 0.79. The extractor is confident these are real commitments. But the commitments themselves are incomplete. A promise with no recipient and no deadline cannot be checked by anyone at any particular time.

Finding 3: Confidence does not equal verifiability

This is the sharpest finding from the whole audit.

Stratum B commitments averaged 0.79 confidence. Yet 92.3% of those same commitments carried no deadline, and 61.5% named no recipient.

The extractor measures how confident it is that a promise was made. It says nothing about whether the promise can be tested. In this dataset, the two diverged almost completely.

If you are building agent tooling, this matters. Reporting a confidence number on a commitment is not the same as reporting whether the commitment can be verified. The two are orthogonal.

Notable examples

Here are the 5 most striking commitments from the dataset, ranked by COGEXT's own risk score. Each one has a public audit URL that anyone can check.

Example 1 — Anthropic SRE cookbook

Source: anthropics/anthropic-cookbook on GitHub

Extracted promise: "I will redeploy the api-server to apply the changes"

Action / object / recipient: redeploy / api-server / NONE

Deadline: NONE

COGEXT verifier query: "check docker-compose logs or container list for api-server recreated with new configuration"

This is a real production agent. It promised to redeploy an API server. It did not say when. The verifier query COGEXT generated is real and checkable, but the promise itself has no time boundary.

Example 2 — OpenAI cookbook

Extracted promise: "I will aim for a window seat for this trip so you can sleep"

Deadline: NONE

Verifier query: "check booking/seat selection records for a window seat on the user's upcoming trip"

An agent promised a specific action that touches a real booking system. There is no date on it, so COGEXT cannot mark it overdue. It can only sit as pending until someone manually checks.

Example 3 — LlamaIndex

Extracted promise: "I will generate a song with the name 'Viva La Vida' and the artist 'Coldplay'"

Deadline: NONE

Verifier query: "check generate_song tool logs for a call with name='Viva La Vida' and artist='Coldplay'"

Even a trivial-seeming promise like "generate a song" gets a checkable verifier. But again, no deadline.

Example 4 — Anthropic SRE cookbook

Extracted promise: "I will open a pull request with the memory limit fix"

Deadline: NONE

Verifier query: "check repository for pull request titled 'Fix checkout-svc OOMKilled crash-loop by increasing memory limits'"

This is a real production scenario. The agent promised to open a PR. If it did, the PR exists somewhere. If it didn't, nothing exists. But there is no deadline, so there is no point at which COGEXT can declare failure.

Example 5 — Anthropic dynamic workflows

Extracted promise: "I will let you know as soon as the fact-check workflow completes"

Deadline: "as soon as it completes"

Verifier query: "check messages/notifications sent to the user indicating the fact-check workflow (task wr1292rh8, run wf_1072dc96-6a2) has completed"

This is the only commitment in the dataset with any time reference at all — and it is conditional, not absolute. The agent will notify when something else completes. There is no upper bound.

What this means

The data shows something more specific than "AI agents lie."

In an unbiased sweep of 120 published outputs, COGEXT found 1 commitment. The binding constraint is not that agents promise too much. It is that a narrow, precision-tuned extractor recognises very little of what agents actually emit as a checkable promise.

When promises were deliberately sought, 61.5% named no recipient and 92.3% carried no deadline. These are not edge cases. They are the majority. A promise with neither a recipient nor a date cannot be falsified by anyone, which makes it functionally unverifiable regardless of how confidently it was extracted.

The practical conclusion for anyone publishing agent demos: report shape (does the action touch the real world?) and completeness (is there a named recipient and a concrete deadline?), not a confidence number.

Limitations, stated plainly

Reddit and X are missing entirely. Both required credentials I did not have. Those are the sources most likely to contain raw, unpolished agent output.
Stratum B is enriched by construction. It is a case sample, not a prevalence estimate.
Not a random sample. Sources were chosen for reachability. Production agent traffic is almost certainly messier and likelier to contain commitments.
No verifiability_score exists. COGEXT's real signals are confidence, risk_score, shape, classification, status, and verification_status. I did not invent a score.
Extraction is heuristic and not hand-verified. Residual false positives are possible.
Text only. No screenshots were OCR'd.

What's next

I built COGEXT because the problem is real: agents make promises, and there is no system that tracks whether those promises are kept. This audit confirms the problem exists. It also shows the shape of the fix.

Agents need to state who they are promising to and when. If they don't, no accountability system can help. That is a design constraint on the agent side, not just the tooling side.

The full dataset, raw logs, and reproduction scripts are available. If you are running an agent in production and want to know whether its promises are actually kept, DM me. I am looking for one real deployment to track for 30 days. Free API key, no strings.

The full report is at findings.md. The raw data is in agent_samples.csv and commitments_all.csv.

If you want to run this audit on your own agent outputs, the pipeline is reproducible.

Site: cogextai.com
Docs: docs.cogextai.com
GitHub: github.com/cogext/cogext
SDK: pip install cogext

How I Built a Verifier Engine That Catches AI Agents Lying About What They Did

Yamin — Mon, 14 Sep 2026 14:13:40 +0000

Your AI agent just told you it sent the email. It didn't.

Your AI agent just told you it deployed the code. It didn't.

Your AI agent just told you it processed the refund. It didn't.

And you have no way of knowing until the client calls. Or the deploy fails in production. Or the refund never arrives.

This is the silent failure mode of AI agents. It's not hallucination in the classic sense. The agent isn't inventing facts. It's claiming an action succeeded when the action never happened. The agent says "done." Nothing checks. The failure is discovered late, which is the expensive combination.

I've been building AI agents for a year. This problem killed more deployments than anything else. So I built a fix.

The Insight: The Agent Is Not a Reliable Source of Truth

Every observability tool today, including LangSmith, Helicone, and Arize, watches what the agent said. They log outputs, trace tokens, and show you what the LLM produced.

They all have the same fatal assumption: the agent's output is a reliable signal of what happened.

It isn't.

An agent that says "I sent the email" may have:

Called the Gmail API and gotten a 500 error
Called the wrong API
Called the right API with the wrong parameters
Not called any API at all

The trace looks identical in every case. "I sent the email." From the agent's perspective, the mission was accomplished.

The unit that matters isn't the commitment. It's the effect.

If the email didn't leave the outbox, the commitment isn't fulfilled, no matter what the agent says.

The Fix: External Verification

Here's the architecture I landed on:

Agent output → Commitment extraction → External verifier → State transition

Instead of trusting the agent's claim, COGEXT runs an external check against the system that would have been affected by the action.

Commitment: "I'll email Sarah the report by Friday."

Verifier query: from:me to:sarah@example.com after:Monday before:Friday

Verifier runs: Gmail API checks the sent folder.

Result:

Email found → fulfilled
Not found by deadline → failed
Not found + 4 hours left → human escalation

The agent doesn't get a vote. The external system does.

Why the Verifier Runs at Creation, Not After

The critical design decision: the verifier query is generated at commitment creation time, not after the deadline passes.

This matters because:

Post-hoc verification is unreliable. By the time you ask "how would I verify this?", the agent has moved on and the context is lost.
Pre-committed audit specs are deterministic. The commitment arrives knowing exactly how it will be judged.
It forces clarity. If you can't write a verifier for a commitment, the commitment isn't verifiable. It should be flagged upfront, not silently tracked forever.

So when a commitment is created, the LLM extracts:

{
  "action": "send",
  "object": "deployment report",
  "recipient": "sarah@example.com",
  "verifier_query": "check sent items for email to sarah@example.com with subject containing 'deployment report'",
  "verifier_type": "gmail"
}

If verifier_query is null, the commitment is marked unverifiable at creation. It never enters the fulfilled pipeline. That's an honesty feature. It tells you the commitment can't be checked, instead of pretending it can.

The State Machine Blocks the Lie

Here's the core state machine:

DETECTED → PENDING_REVIEW → OPEN → DUE → OVERDUE → FULFILLED ✓
                                  ↓                → FAILED ✗
                                  ↓                → EXPIRED ✗
                                  ↓
                            BLOCKED → OPEN
                                   → FAILED ✗

Terminal states: fulfilled, failed, expired, cancelled, superseded, contradicted.

The critical rule: the fulfilled transition requires external evidence.

We enforce this at the database level. All state changes go through a PostgreSQL function:

CREATE OR REPLACE FUNCTION cogext_transition_commitment(
    p_commitment_id UUID,
    p_new_status TEXT,
    p_actor TEXT DEFAULT 'system',
    p_data JSONB DEFAULT '{}'
) RETURNS JSONB AS $$
DECLARE
    current_status TEXT;
    evidence_score FLOAT;
BEGIN
    SELECT status INTO current_status 
      FROM commitments WHERE id = p_commitment_id;

    -- Block fulfilled without evidence for external commitments
    IF p_new_status = 'fulfilled' THEN
        SELECT MAX(score) INTO evidence_score
          FROM evidence 
          WHERE commitment_id = p_commitment_id;

        IF evidence_score IS NULL OR evidence_score < 0.7 THEN
            RAISE EXCEPTION 'Cannot fulfill without evidence score >= 0.7';
        END IF;
    END IF;

    -- Validate transition, update, insert event, return
    -- ...
END;
$$ LANGUAGE plpgsql;

Tested this last week. Tried to force-fulfill a commitment via the API. Got HTTP 409. The database refused.

The agent cannot close its own loop. Not because of application logic. Because of the database.

The Kill Switch (Human-in-the-Loop)

Some actions are too dangerous to run without human approval. Processing a $12,000 refund. Deploying to production. Sending an email to a client.

For these, COGEXT pauses the agent and sends a Slack message:

🚨 COGEXT: High-risk action detected

Agent: support-agent-001
Action: process refund
Amount: $12,000
Customer: 4821

[Approve] [Cancel]

The agent doesn't execute until a human clicks one of the buttons. Every decision is logged in the events table with the actor, timestamp, and reason.

This is the feature every team asks for after the first time an agent does something catastrophic.

The Contradiction Radar

Agents contradict themselves constantly. At 2:00 PM: "I'll deliver Friday." At 2:47 PM: "I'll deliver Monday." Same recipient. Same object. Different deadline.

Nobody catches this in real-time. Not the developer. Not the client. Until Monday arrives and the client expects delivery and the agent thinks it's already Thursday.

COGEXT scans every open commitment on every new ingest. If the new commitment conflicts with an existing one, we flag it immediately:

{
  "contradiction_alert": {
    "old_id": "ceb5ab93-...",
    "new_id": "567ae7bb-...",
    "reason": "Same commitment sent as two distinct messages. Deadline revised.",
    "old_promise": "I will deliver the report to the client on Friday",
    "new_promise": "I will deliver the report to the client"
  }
}

The old commitment is marked superseded. The new one is tracked. The full history is preserved.

The Failure Predictor

Every commitment gets a risk score at creation time. Based on:

Deadline timing, such as Friday 5 PM being riskier than Tuesday morning
Keywords (legal, approval, irreversible)
Agent's historical fulfillment rate

Example output:

{
  "risk_score": 0.55,
  "reasons": [
    "Involves legal review",
    "Due Friday after 4 PM",
    "This agent has a 67% fulfillment rate on Friday deadlines"
  ]
}

It's rule-based right now. No ML. But it catches the obvious cases, the ones that cause 80% of failures.

The Public Audit Receipt

Every commitment generates a shareable proof URL.

https://api.cogextai.com/api/v1/receipt/UtN6A3jo.axKJXIQ_xniFUgne07fGd9c3R4cYknLrCmRqaxGR75c

Anyone can visit it and see:

Original commitment text
Full state history
Evidence found or not found
Human approvals
HMAC signature for integrity verification

Send it to a client. Send it to an auditor. Send it to a regulator.

The point isn't the receipt. The point is that the developer can prove what their agent actually did.

The Stack

FastAPI + Python 3.12
PostgreSQL with pgvector for semantic memory
Groq (llama-3.3-70b) for extraction
Instructor for structured output
Gmail API, GitHub API, Webhooks for verifiers
Slack for alerts and approvals

The SDK is 3 lines:

from cogext import track
agent = track(your_agent, api_key="cg_live_xxx")

Every call to agent.run() auto-ingests the output. Commitments are tracked. Verifiers run on a schedule. Evidence is checked against external systems.

The E2E Test Results

We ran the full test suite against production last week:

| Test | Feature | Status |
|------|---------|--------|
| 1 | Failure Predictor | PASS |
| 2 | Verifier Engine | PASS |
| 3 | Contradiction Radar | PASS |
| 4 | Kill Switch | PASS |
| 5 | Audit Receipt | PASS |
| 6 | Idempotency | PASS |
| 7 | State Machine Gate | PASS |
| 8 | Reliability | PASS |

Ready to Announce? YES. All tests passed.

Test 7 is the one that matters most. It tries to force-fulfill a commitment without evidence. HTTP 409. Blocked at the database level.

The state machine won't let the agent lie.

What This Is Really About

Every observability tool watches what the agent said. COGEXT watches what actually happened.

The difference sounds small. It isn't.

A tool that watches the agent's output will tell you "the agent sent the email." A tool that watches the effect will tell you "the email was sent" or "the email was never sent."

The first is a log. The second is an audit trail.

If your agent makes promises in production, you need the second one. Not because it's fashionable, but because the first one has a failure mode you can't detect until the client calls.

Try It

Site: cogextai.com
Docs: docs.cogextai.com
GitHub: github.com/cogext/cogext
SDK: pip install cogext

Free tier: 2,500 commitments per month. No credit card.

If you're running AI agents in production and you've ever wondered "did my agent actually do that?", give it a try. Open an issue. Tell me what breaks.

The agent's word should not be the final word.

I built a 12-state accountability layer for AI agents here's what I learned

Yamin — Thu, 03 Sep 2026 20:49:01 +0000

By the time you finish reading this, your AI agent will have made at least one promise it can't keep.

That's not a dig at your code. It's a fundamental property of how we build AI systems today. Agents are brilliant at generating intent and terrible at tracking it. They'll promise to email a client, confirm an appointment, or deploy a fix and two conversations later, they've forgotten they said anything at all.

The more I thought about this, the more I realized: commitments aren't a memory problem. They're a state machine problem.

So I built COGEXT — an accountability layer that sits between your agent and the outside world. You send it agent output, it extracts every commitment, and tracks each one until it's fulfilled, failed, or overdue.

The core idea: commitments have state

Most systems treat agent output as text to be logged. We treat it as events to be tracked.

Every extracted commitment moves through 12 states:

DETECTED
    ├──→ OPEN ──→ DUE ──→ OVERDUE ──→ FULFILLED ✓
    │                        ├──→ FAILED ✗
    │                        └──→ EXPIRED ✗
    ├──→ PENDING_REVIEW ──→ OPEN
    ├──→ CANCELLED ✗
    ├──→ CONTRADICTED ✗
    └──→ BLOCKED ──→ OPEN
                  └──→ FAILED ✗

Terminal states: fulfilled, failed, expired, cancelled, superseded, contradicted.

All state mutations go through a single PostgreSQL RPC function. Not direct UPDATEs from application code:

CREATE OR REPLACE FUNCTION cogext_transition_commitment(
    p_commitment_id UUID,
    p_new_status TEXT,
    p_actor TEXT DEFAULT 'system',
    p_data JSONB DEFAULT '{}'
) RETURNS JSONB AS $$
BEGIN
    -- lock row → validate transition → update status → insert event → return
    -- single atomic operation: if any step fails, the whole thing rolls back
END;
$$ LANGUAGE plpgsql;

Why? Because the state change and the event must be atomic. You should never have status = 'fulfilled' without a corresponding fulfillment event in the audit log. That's the line between a logging system and an accountability system.

Extracting commitments from agent output

The extraction pipeline has two stages.

Stage 1: Classification

Not everything an agent says is a commitment. We classify first:

genuine_commitment — "I will send the report by Friday."
intention — "I might send the report tomorrow."
suggestion — "We should send the report tomorrow."
hypothetical — "If we have time, I could send it."
quoted_statement — "John said he would send it."

Only genuine_commitment with confidence ≥ 0.5 passes through.

Stage 2: Structured extraction

We use Groq's llama-3.3-70b in JSON mode to extract fields:

response = await groq_client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": prompt + message}],
    response_format={"type": "json_object"},
    temperature=0.2,
)

The output is validated against a Pydantic schema. If the LLM returns garbage, we retry once. If it still fails, the commitment is dropped — not inserted.

Key decision: We don't trust the LLM to interpret deadlines. We extract the raw expression ("Friday EOD") and pass it to a deterministic temporal normalizer. The LLM interprets; the code decides.

Evidence scoring: why 0.88 ≠ 0.92

Evidence verification is where most systems fail. Binary matching — "does this email mention the report?" — generates too many false positives.

We use a weighted field-coverage model:

Field	Weight
`action`	0.40
`recipient`	0.30
`object`	0.20
`deadline`	0.10

When evidence arrives, we check which commitment fields it matches and sum the weights:

Evidence 1: confirms action ("I sent it")
    → 0.40

Evidence 2: confirms recipient + object ("sent to Sarah, deployment-report.pdf attached")
    → 0.30 + 0.20 = 0.50

Total = 0.90 → FULFILLED

The threshold is configurable (default 0.90). Multiple pieces of evidence can satisfy different fields — they aggregate.

The idempotency problem

AI agents repeat themselves. An agent might say:

"I'll send the report by Friday."
"Let me confirm: I'll send it by Friday."
"Just to reiterate, I will send the report by Friday."

Same commitment. Without idempotency, you'd track it three times.

def compute_idempotency_key(agent_id, promise_text, timestamp):
    window = timestamp.replace(minute=0, second=0, microsecond=0)
    payload = f"{agent_id}|{promise_text.strip().lower()}|{window.isoformat()}"
    return hashlib.sha256(payload.encode()).hexdigest()

Same agent + same promise + same hour = same hash. The DB unique constraint rejects the duplicate silently.

The hour window is deliberate: it deduplicates within a session while still allowing the same promise to be legitimately re-made days later.

The four lessons that cost us the most

1. LLM output is untrusted input.
We validate everything against Pydantic schemas before touching the database. No exceptions.

2. Determinism over magic.
LLMs for interpretation. Deterministic code for everything else: state transitions, date normalization, deduplication, metrics. We don't call the LLM to check if a deadline is overdue. We run a database query.

3. Events are not logs.
A log is a record that can be ignored. An event is a fact that cannot be denied. We enforce this at the DB layer — a trigger rejects any UPDATE or DELETE on the events table. Historical events are immutable.

4. Scope the user_id at the auth layer, never the request body.
Early version trusted user_id from the request payload. Classic mistake. All scoping now comes from the API key's account_id. The request body cannot claim to own data it doesn't own.

Try it

The API is live and the SDK is on PyPI:

pip install cogext

from cogext import track

agent = track(your_agent, api_key="cg_live_xxx")
output = agent.run("I'll send the report by Friday")
# commitment is tracked automatically

Docs at docs.cogextai.com — there's a quickstart that takes about 5 minutes.

Would genuinely appreciate feedback from anyone who's hit this problem with their own agents. What does your current approach look like?

[Boost]

Yamin — Thu, 03 Sep 2026 20:47:52 +0000