DEV Community: Aman

I Built an AI-Agent Guardrail, Then Discovered My Observer Was Watching Itself

Aman — Sat, 25 Jul 2026 19:57:37 +0000

Reactive AI cost monitoring has an awkward limitation: by the time a chart shows a runaway agent, the tokens are already gone. I wanted to test a different idea for the Agents of SigNoz hackathon: make a small execution contract before an AI coding agent starts work, then measure whether its actual behavior stays inside that contract.

That became Preflight. Given a coding task, it predicts a model tier, effort level, and expected file footprint. Those predictions become observable expectations: when the agent writes files, Preflight measures scope drift, emits OpenTelemetry, and surfaces the result in SigNoz. It is deliberately a prototype, not a universal security sandbox. The useful part is that the prediction, the observed actions, and the misses are all inspectable.

The most valuable lesson did not come from the happy path. It came from discovering that my external observer was counting its own tooling metadata as agent work.

The contract comes before the agent

A task such as 'rename one local variable' should not automatically receive the same model budget or repository-wide scope as 'introduce tenant-aware authorization across billing, reporting, migrations, tests, and rollout documentation.' Preflight classifies the task first and creates a compact contract:

Contract field	Example
Tier	small / medium / large
Effort	low / medium / high
Blast radius	isolated / moderate / wide
File ceiling	3 / 10 / 25 files

I use a cheap gpt-5.6-luna structured classification call for that small JSON decision, capped at 250 completion tokens. This is intentional: the guard should practice the same 'predict cheap when appropriate' principle it recommends to other agents.

Preflight then has two honest runtime modes:

In-process interception evaluates actions from Preflight's built-in agent while it runs. When drift gets high, it can pause and propose a human-confirmed remediation.
External watch predicts before launching another coding CLI, recursively observes writes and deletes in a selected workspace, then calculates drift after the process exits. This mode is post-hoc; it cannot preempt the child process.

That distinction matters. A filesystem watcher cannot prove semantic safety, see short-lived files, or protect paths outside the watched workspace. Calling those modes equivalent would make the demo look stronger than it is.

What SigNoz sees

I exported stable OpenTelemetry span names instead of putting task text or model names into span names:

preflight.predict
  -> preflight.execute
    -> preflight.action.evaluate
    -> preflight.drift.check
    -> preflight.signal      (only when something is wrong)
    -> preflight.complete    (successful session end)

Attributes carry the changing details: session ID, predicted tier, runtime mode, action target, verdict, and drift score. The metrics cover classifier cost, modeled savings versus an always-large baseline, tier distribution, action count, and drift. 'Modeled savings' is explicitly a comparison assumption, not a billing claim.

I used those signals across several SigNoz features:

a Preflight dashboard for classification cost, drift, routing tiers, policy verdicts, session cost, and modeled savings;
three warning alert rules for scope drift, cost spikes, and the hard ceiling;
a trace funnel with preflight.predict -> preflight.execute -> preflight.drift.check -> preflight.complete;
Query Builder aggregation to find the tiers actually showing high drift.

The dashboard query was not just a screenshot. In the saved v5 query, I calculated avg(preflight.drift_score), grouped by preflight.predicted_tier, and applied HAVING avg_drift_score > 0.5. In the current data, only the medium tier surfaced, at 0.542 average drift. That is a real query result, not a synthetic metric.

A three-session live batch reached every stable funnel stage: predict, execute, drift check, and complete. The classification cost for that batch was $0.001852. I also hit two practical SigNoz integration details while building it: dashboard group-by needed groupBy.key rather than an older groupBy.name shape, and v5 aggregation aliases must be supplied through the alias field instead of inline SQL-style aliasing. Those are the kinds of details that only show up when a dashboard is actually wired to data.

The bug: my observer was watching itself

For the external-watch proof, I wrapped the installed Codex CLI and asked it to create exactly one Markdown file. The first raw result looked like a success: the watcher saw changed files and drift rose.

But the requested file had not been created.

The culprit was a globally installed Codex plugin named Foresight. It wrote its own .foresight contract and history files inside the temporary workspace. My watcher dutifully counted those internal files as if the external coding agent had edited user code.

That was invalid evidence. I kept the failed attempts in the repository, but I did not call them successful runs.

The fix is intentionally narrow:

_INTERNAL_WATCH_DIRECTORIES = frozenset({".foresight"})

relative_path = path.relative_to(self.workspace)
if any(part in _INTERNAL_WATCH_DIRECTORIES for part in relative_path.parts):
    continue

I also made the proof runner fail closed. A run is accepted only when the external CLI returns zero, the exact requested output file exists, and its contents exactly match the expected text. Every rerun receives a fresh workspace. This is not a broad hidden-file exclusion; it filters one verified plugin metadata directory and leaves ordinary project files observable.

After the correction, the committed result contains two real Codex CLI runs. Each produced exactly one requested user file, each returned exit code 0, and each had one observed file write with 0.33 drift against a three-file contract. The two live Luna predictions used 203 input tokens each, 50 and 54 output tokens, and cost $0.001030 total.

Evidence instead of a perfect-looking score

I wanted the evaluation story to be more useful than a single flattering percentage. The initial 18-task live benchmark was 61.11% accurate on both tier and blast-radius labels. That exposed a real weakness: short requests underweighted authentication, validation, regression coverage, and durable operational work.

I applied one scope-rubric calibration and reran the same corpus. The result was 18/18, or 100%, but I label it clearly as a same-corpus calibrated result rather than independent validation. Then I froze a separate eight-task held-out set before running it. That held-out run was 8/8 for tier and blast radius. It is encouraging, not a universal guarantee.

The repository keeps v1, v2, held-out data, raw token counts, costs, failed external attempts, and successful external results. The interesting engineering outcome is not 'the model never misses.' It is that a miss, a drift event, or a broken observer leaves a trail another person can inspect.

What I would tell my past self

Predict before spending. A cheap preflight decision can bound a much more expensive agent run.
State runtime guarantees precisely. In-process interception and post-hoc external watch are useful for different reasons.
Treat failed evidence as evidence. The Foresight metadata bug made the project stronger only because I kept the bad run, explained it, and tightened the test.
Make observability falsifiable. Dashboards are more valuable when they connect to committed raw data, stable spans, and explicit boundaries.

Preflight is my attempt at making agent scope and cost visible before they become cleanup work. The full repository, Foundry deployment files, SigNoz dashboard evidence, and benchmark artifacts are available at AmanM006/Preflight.

Building Sovereign: A Dual-Database Creator Liquidity Exchange on AWS Aurora DSQL + DynamoDB

Aman — Fri, 26 Jun 2026 23:43:55 +0000

Digital creators hold billions of dollars in illiquid assets - from newsletters and YouTube channels to SaaS repositories and Notion templates. How do you fractionalize ownership with instant, atomic trading settlements and absolute data integrity under heavy concurrent load?

Enter Sovereign: a real-time Creator Liquidity Exchange built on a dual-database architecture - AWS Aurora DSQL for transactional correctness and Amazon DynamoDB for high-throughput telemetry.

The Problem: Distributed Contention on the Order Book

When multiple buyers attempt to match against the same resting sell orders concurrently, they generate a high-contention race condition:

The Over-Allocation Threat: If two buy transactions read the same sell order's remaining balance simultaneously, both believe they can buy it. This creates phantom shares - double spending where total ledger allocation exceeds asset supply.
The Latency Trap: Solving this with distributed locks slows execution significantly across regions, raising latency to hundreds of milliseconds.

The Solution: Optimistic Concurrency Control with DSQL

Instead of locking rows, Sovereign uses SERIALIZABLE isolation on Aurora DSQL:

Every matching event executes in a single database transaction. The engine reads counter-orders, computes fill quantities, and updates buyer and seller ledger rows atomically.
If another transaction modified the order concurrently, DSQL throws a 40001 serialization conflict. The engine catches it, applies jittered exponential backoff, and retries:

const isOCCCollision = (err: unknown): boolean => {
  if (err instanceof Error) {
    const cause = (err as any).cause;
    return (
      cause?.code === "40001" ||
      cause?.message?.includes("serialization failure")
    );
  }
  return false;
};

// Retry loop
while (attempt <= MAX_RETRIES) {
  try {
    const result = await attemptMatch(orderId);
    return result;
  } catch (err) {
    if (isOCCCollision(err)) {
      const delay = Math.pow(2, attempt) * 10 + Math.random() * 50;
      await sleep(delay);
      attempt++;
      continue;
    }
    throw err;
  }
}

Dual-Database Architecture

The most important architectural decision: two databases, each doing what it's best at.

Aurora DSQL (Stockholm, eu-north-1) - Transactional Core

Orders, trades, ownership ledger, assets
Requires SERIALIZABLE isolation and global consistency
OCC collision resolution IS the product mechanic
Flat schema, no FK constraints (DSQL active-active requirement)
CREATE INDEX ASYNC (synchronous indexes not supported on distributed clusters)

Amazon DynamoDB (eu-north-1) - Activity Firehose

Trade events, OCC collision logs, stampede telemetry
Single-table design with 3 GSIs
30-day TTL auto-expiry
PAY_PER_REQUEST billing
No transactions needed - pure append-only throughput DSQL for correctness. DynamoDB for throughput.

Every trade fires a fire-and-forget write to DynamoDB:

// After successful DSQL match - never blocks critical path
writeFirehose({
  eventType: "TRADE",
  assetId: order.asset_id,
  payload: { sharesTraded, priceExecuted, latencyMs },
  timestamp: new Date().toISOString(),
}).catch(() => {}); // silent fail - DynamoDB is best-effort

System Architecture Flow

              ┌───────────────────────────────┐
              │      Client Web App (UI)      │
              └───────────────┬───────────────┘
                              │ HTTPS / SSE
                              ▼
              ┌───────────────────────────────┐
              │     Next.js Server Actions    │
              └───────┬───────────────┬───────┘
                      │               │
  Serializable Tx     │               │ Async Telemetry Write
 (Postgres pg client) │               │ (AWS SDK PutCommand)
                      ▼               ▼
              ┌───────────────┐┌───────────────┐
              │  OCC Engine   ││  DynamoDB     │
              │  (Aurora DSQL)││  Firehose     │
              └───────────────┘└───────────────┘

Stress Testing: The Stampede Load Simulator

To prove integrity under load, we built a stampede simulator:

Seed 100 sell orders from market-maker
Fire 100 concurrent buy orders simultaneously
DSQL generates serialization failures - engine retries with jitter
Audit: SUM(shares_owned) must equal total_shares exactly

Recent Run Results:

Settled: 100/100 orders
OCC Collisions Resolved: 432
Time elapsed: 1,450ms
Balance Delta: 0
Audit Verdict: ✅ PASSED - ZERO OVER-ALLOCATION

The mathematical proof: if two buyers both got share #47, delta would be nonzero. Delta is always 0. DSQL OCC makes this impossible.

Key Implementation Challenges

Drizzle wraps native errors: Drizzle puts the real Postgres error in err.cause, not err directly. Our retry loop initially missed all 40001 codes. Fix: inspect err.cause?.code.
DSQL DDL restrictions: No SERIAL types, no synchronous indexes. Required a custom migration runner using CREATE INDEX ASYNC.
Vercel timeout: 300 concurrent orders to Stockholm DSQL exceeded Vercel's 60s function timeout. Restructured to 3 batches of 20 with timeout escape hatch.

Live Demo & Code

Live Demo: sovereign-dex.vercel.app - click DEMO LOGIN, no signup needed.
Run Stampede: Hit RUN STAMPEDE in the telemetry console to see OCC collisions resolve live. Watch the DynamoDB activity feed populate on the market home page.
GitHub Repository: AmanM006/sovereign

I created this article for the H0: Hack the Zero Stack hackathon.