I built a 12-state accountability layer for AI agents here's what I learned

Yamin — Thu, 03 Sep 2026 20:49:01 +0000

By the time you finish reading this, your AI agent will have made at least one promise it can't keep.

That's not a dig at your code. It's a fundamental property of how we build AI systems today. Agents are brilliant at generating intent and terrible at tracking it. They'll promise to email a client, confirm an appointment, or deploy a fix and two conversations later, they've forgotten they said anything at all.

The more I thought about this, the more I realized: commitments aren't a memory problem. They're a state machine problem.

So I built COGEXT — an accountability layer that sits between your agent and the outside world. You send it agent output, it extracts every commitment, and tracks each one until it's fulfilled, failed, or overdue.

The core idea: commitments have state

Most systems treat agent output as text to be logged. We treat it as events to be tracked.

Every extracted commitment moves through 12 states:

DETECTED
    ├──→ OPEN ──→ DUE ──→ OVERDUE ──→ FULFILLED ✓
    │                        ├──→ FAILED ✗
    │                        └──→ EXPIRED ✗
    ├──→ PENDING_REVIEW ──→ OPEN
    ├──→ CANCELLED ✗
    ├──→ CONTRADICTED ✗
    └──→ BLOCKED ──→ OPEN
                  └──→ FAILED ✗

Terminal states: fulfilled, failed, expired, cancelled, superseded, contradicted.

All state mutations go through a single PostgreSQL RPC function. Not direct UPDATEs from application code:

CREATE OR REPLACE FUNCTION cogext_transition_commitment(
    p_commitment_id UUID,
    p_new_status TEXT,
    p_actor TEXT DEFAULT 'system',
    p_data JSONB DEFAULT '{}'
) RETURNS JSONB AS $$
BEGIN
    -- lock row → validate transition → update status → insert event → return
    -- single atomic operation: if any step fails, the whole thing rolls back
END;
$$ LANGUAGE plpgsql;

Why? Because the state change and the event must be atomic. You should never have status = 'fulfilled' without a corresponding fulfillment event in the audit log. That's the line between a logging system and an accountability system.

Extracting commitments from agent output

The extraction pipeline has two stages.

Stage 1: Classification

Not everything an agent says is a commitment. We classify first:

genuine_commitment — "I will send the report by Friday."
intention — "I might send the report tomorrow."
suggestion — "We should send the report tomorrow."
hypothetical — "If we have time, I could send it."
quoted_statement — "John said he would send it."

Only genuine_commitment with confidence ≥ 0.5 passes through.

Stage 2: Structured extraction

We use Groq's llama-3.3-70b in JSON mode to extract fields:

response = await groq_client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": prompt + message}],
    response_format={"type": "json_object"},
    temperature=0.2,
)

The output is validated against a Pydantic schema. If the LLM returns garbage, we retry once. If it still fails, the commitment is dropped — not inserted.

Key decision: We don't trust the LLM to interpret deadlines. We extract the raw expression ("Friday EOD") and pass it to a deterministic temporal normalizer. The LLM interprets; the code decides.

Evidence scoring: why 0.88 ≠ 0.92

Evidence verification is where most systems fail. Binary matching — "does this email mention the report?" — generates too many false positives.

We use a weighted field-coverage model:

Field	Weight
`action`	0.40
`recipient`	0.30
`object`	0.20
`deadline`	0.10

When evidence arrives, we check which commitment fields it matches and sum the weights:

Evidence 1: confirms action ("I sent it")
    → 0.40

Evidence 2: confirms recipient + object ("sent to Sarah, deployment-report.pdf attached")
    → 0.30 + 0.20 = 0.50

Total = 0.90 → FULFILLED

The threshold is configurable (default 0.90). Multiple pieces of evidence can satisfy different fields — they aggregate.

The idempotency problem

AI agents repeat themselves. An agent might say:

"I'll send the report by Friday."
"Let me confirm: I'll send it by Friday."
"Just to reiterate, I will send the report by Friday."

Same commitment. Without idempotency, you'd track it three times.

def compute_idempotency_key(agent_id, promise_text, timestamp):
    window = timestamp.replace(minute=0, second=0, microsecond=0)
    payload = f"{agent_id}|{promise_text.strip().lower()}|{window.isoformat()}"
    return hashlib.sha256(payload.encode()).hexdigest()

Same agent + same promise + same hour = same hash. The DB unique constraint rejects the duplicate silently.

The hour window is deliberate: it deduplicates within a session while still allowing the same promise to be legitimately re-made days later.

The four lessons that cost us the most

1. LLM output is untrusted input.
We validate everything against Pydantic schemas before touching the database. No exceptions.

2. Determinism over magic.
LLMs for interpretation. Deterministic code for everything else: state transitions, date normalization, deduplication, metrics. We don't call the LLM to check if a deadline is overdue. We run a database query.

3. Events are not logs.
A log is a record that can be ignored. An event is a fact that cannot be denied. We enforce this at the DB layer — a trigger rejects any UPDATE or DELETE on the events table. Historical events are immutable.

4. Scope the user_id at the auth layer, never the request body.
Early version trusted user_id from the request payload. Classic mistake. All scoping now comes from the API key's account_id. The request body cannot claim to own data it doesn't own.

Try it

The API is live and the SDK is on PyPI:

pip install cogext

from cogext import track

agent = track(your_agent, api_key="cg_live_xxx")
output = agent.run("I'll send the report by Friday")
# commitment is tracked automatically

Docs at docs.cogextai.com — there's a quickstart that takes about 5 minutes.

Would genuinely appreciate feedback from anyone who's hit this problem with their own agents. What does your current approach look like?

[Boost]

Yamin — Thu, 03 Sep 2026 20:47:52 +0000

DEV Community: Yamin