Sentinel-IR: Stop Making Your Agents Read Human Code. Give Them a Fact Layer

Live-benchmarked against gpt-6-astra. Raw scorecard and log included.

Part 1 — For everyone

The problem in one sentence

Every coding agent today does the same wasteful thing: to answer a simple question like "does this merge request touch the network?", it reads the entire source file — hundreds of lines of human-oriented code — and burns thousands of tokens on it.

Source code is written for humans. Comments, formatting, naming style — all of it is noise to an agent that only needs to know what the code does. So we built Sentinel-IR: a compact, machine-oriented intermediate representation that keeps the meaning and drops the noise.

What Sentinel-IR is (and isn't)

Sentinel-IR is not a new programming language. Nobody writes in it. It's a fact layer: a deterministic extraction of the security-relevant things in a file — the HTTP routes it exposes, the environment variables it reads, the files it writes, the processes it spawns, the surface it exports.

Think of it as the difference between handing your agent a 500-page novel and a one-page factual brief written by a parser that never gets tired and never guesses.

Key properties, in plain language:

• Deterministic. Same file → same IR, every time. No probabilities, no temperature. In our pipeline A/B testing, zero invariant drift — action, reason, risk, role, confidence and constraints were identical across every input mode.
• Parser-based, not substring-based. Built on a real tree-sitter AST, not content.includes("axios"). A call is a call_expression; a route is a call on app/router/server with a string path; an env read is a member access on process.env. That's how a false positive on regexp.exec got eliminated — name matching alone can't tell it from child_process.exec.
• Proven absence. When a file parses, a missing key means provenly absent, not "we didn't look". A raw source dump can never give you that.

The numbers — measured live, not offline

We ran the benchmark for real: 12 files, 87 questions, 267 actual LLM calls against gpt-6-astra. Same files, same questions, same ground truth for every variant.

Variant	Input tokens	Accuracy	Unresolved
Raw source	279,476	84/87 (96.6%)	0
IR only	58,549 (−79.1%)	82/87 (94.3%)	5
IR + raw fallback	80,340 (−71.3%)	87/87 (100%)	0

The headline: IR with fallback saves 71.3% of input tokens and is more accurate than reading the raw source — 100% vs 96.6%.

Two findings worth more than the headline:

• Less noise = better judgment. On 3 files (git-probe, token-signer, report-worker), the model answered the security_risk question correctly from the IR where it got the raw source wrong. The compact fact layer beat the full text it was derived from.
• The honest failure mode is visible. All 5 questions IR couldn't answer were "empty set" questions — no env vars, no dangerous constructs, no disk writes. Today's IR can't express "nothing is there", so it escalated to raw instead of guessing. Every escalation was recoverable; zero confidently-wrong answers in any variant.

Where it wins and where it doesn't — per file

IR has near-constant size, so savings scale with file size. Fitted break-even: ~303 source tokens (~34 lines).

File	Lines	Raw tokens	IR tokens	Savings
12-billing-platform	1,366	11,635	1,332	88.6%
10-analytics-kernel	951	7,186	498	93.1%
11-gateway-service	952	7,364	1,192	83.8%
09-report-worker	511	3,819	778	79.6%
08-inventory-api	393	2,861	734	74.3%
07-order-service	138	959	527	45.0%
04-polynomial	32	223	134	39.9%
02-cache-writer	33	209	287	−37.3%
05-git-probe	23	149	300	−101.3%
01-http-api-server	27	163	272	−66.9%
06-token-signer	23	163	224	−37.4%
03-status-client	26	172	199	−15.7%

Below ~34 lines, IR costs more than the source. We print that in our own output rather than hiding it — the per-file table is part of the scorecard.

What it costs in the real world

The full live run: 263 requests, 395,847 input / 9,239 output tokens, $4.93 total on the org account. Two honest notes:

• Our chars/4 token estimate predicted ~418k input tokens; reality was 396k (within 5%), so the ratios in this article hold.
• 70% of the cost was cache *writes* — the model bills cache writes more than input, and a benchmark sends 267 different prompts, so nothing is ever read back. In production Sentinel this doesn't apply: the ~20 KB system prompt is identical per call, so it hits cache. Benchmark cost is an upper bound on what the same volume costs a real deployment.

Against a real alternative — GitLab Orbit Local

	Orbit Local	Sentinel-IR
Answers correct	29/87 (33.3%)	87/87 (100%)
Context completeness	41.4%	100%
Confidently wrong	7	0

The gap is expression-level: Orbit's graph knows file structure, but not "this line spawns a child process" or "this MR adds a POST route reading an env secret". That is exactly the layer IR fills. (Orbit Remote is unmeasured — it needs a Premium group and a Knowledge Graph: Read token; we don't claim it.)

The honest part

• Token counts are chars/4 estimates applied identically to every variant — only the ratio is claimed (and it matched the provider's billing within 5%).
• One model, one run — no variance reported. The per-question rows are published so you can check the failures yourself.
• The corpus is ours. The external validation we do have: on 16 external repos / 140 merged PRs, the fact layer's critical gate blocked 3 PRs — all three genuinely executing external commands, missing none of the 28 files that run one. Precision 5/5, recall 85/85 on hand-verified findings.
• Inside Sentinel's own decision pipeline the saving is modest — the pipeline was already efficient. The product is the fact layer another agent consumes, not our own token bill.

---

Part 2 — For developers

The pipeline

JavaScript
    ↓
tree-sitter parser
    ↓
AstFacts    — routes / exports / imports / env / calls / risk
    ↓
Sentinel-IR — compact, flat, self-describing projection
    ↓
LLM (your agent)   [fallback: raw source on unresolved]
    ↓
Validator → Simulation → Commit

Everything below the parser is deterministic and local: no network, no LLM, no I/O. libs/core/ast-facts.js walks the AST and classifies real nodes — exec/fork only count as process-spawning when the callee resolves to child_process/execa/zx.

What an IR looks like

compressFacts in libs/core/sentinel-ir.js — deliberately boring:

function compressFacts(facts) {
    if (!facts?.ast) {
        return { ast: false, reason: facts?.error || "source did not parse; IR fell back to text heuristics" };
    }

    const compressed = { ast: true };
    const put = (key, value) => {
        if (Array.isArray(value) && value.length > 0) compressed[key] = value;
    };

    put("routes", facts.routes);
    put("exports", facts.exports);
    put("imports", facts.imports);
    put("env", facts.env);
    put("operations", Object.entries(facts.operations || {})
        .filter(([, enabled]) => enabled)
        .map(([name]) => name));

    const calls = {};
    for (const [bucket, entries] of Object.entries(facts.calls || {})) {
        if (Array.isArray(entries) && entries.length > 0) calls[bucket] = entries;
    }
    if (Object.keys(calls).length > 0) compressed.calls = calls;

    put("dangerous", facts.dangerous);
    put("riskSignals", (facts.riskSignals || []).map(s => `${s.signal}:${s.evidence}@${s.line}`));

    return compressed;
}

Three design choices worth stealing:

1. Flat and self-describing — enabled operations only, empty categories omitted. An agent reads a sparse object, not a tree it has to traverse.
2. Every risk signal keeps its evidence — signal:evidence@line, traceable back to the syntax that produced it.
3. ast: true is a completeness contract — a missing key is a proven absence. When the file doesn't parse you get ast: false with a reason, never silently wrong data.

The known gap this creates is the interesting part: explicitly-empty categories are omitted, so "are there env vars?" currently resolves to unresolved → escalate rather than provenly no. The fix — emitting explicit empty facts when ast: true — is the single change that would have turned 5 of our live misses into correct answers without touching the fallback. It's on the list.

The full compressed shape, field-for-field faithful to SentinelIR.compress():

{
  "mission":     { "target": "libs/api/server.js", "objective": "network_stability" },
  "world":       { "pressure": 0.5, "confidence": 0.78, "budget": 0.015, "risk": "high" },
  "constraints": ["preserve_api", "avoid_breaking_changes"],
  "forbidden":   ["eval", "child_process"],
  "summary":     { "size": 14203, "lines": 389, "hasCrypto": false,
                   "hasFilesystem": true, "hasNetwork": true },
  "facts": {
    "ast": true,
    "routes":  ["GET /health", "POST /orders"],
    "env":     ["DATABASE_URL", "STRIPE_SECRET_KEY"],
    "operations": ["inboundHttp", "diskWrite"],
    "calls":   { "process": ["spawn(node:child_process)@214"] },
    "riskSignals": ["process_spawn:spawn@214"]
  }
}

(Illustrative values; schema is exact.)

How the benchmark scores it

libs/ir-benchmark/runner.js defines "savings at retained accuracy": the best variant that is at least as accurate as reading raw source. If none is, the honest answer is 0% — not a smaller lie.

const candidates = [
    { variant: "ir", stats: ir },
    { variant: "ir+raw", stats: hybrid }
].filter(c => c.stats.correct >= raw.correct);

const best = candidates.sort((a, b) => a.stats.inputTokens - b.stats.inputTokens)[0] || null;
// → savingsAtRetainedAccuracyPct: best ? savings(best.stats) : 0

This run: ir+raw was the only variant at ≥ raw accuracy, so the claimed figure is 71.3% — not the prettier 79.1% that lost accuracy.

Why it beats feeding raw diffs

A diff shows what changed in text. IR answers what the change does: routes added/removed, env values newly read, fs/network/process operations appeared, exported surface changed, risk taxonomy movement. We dogfood it as a per-MR CI report across our own 44 merged MRs: 35 touched JS, 31 produced facts, median 13 facts/MR, 18 raised a file's risk level — and the job gates: an MR pushing a file to critical fails until acknowledged in .sentinel-gate.json.

Reproduce it

npm run ir-benchmark        # offline: info content, upper bound
node scripts/ir-benchmark.js --live --model gpt-6-astra   # what we ran: 267 calls, ~$4.9
npm run orbit-ab            # IR vs GitLab Orbit Local
npm run ir-pipeline-ab      # 0 invariant drift across input modes
npm run mr-report           # per-MR fact report over your own history

All of it is libs/core/sentinel-ir.js + libs/core/ast-facts.js + libs/ir-benchmark/. Tree-sitter is the only runtime dependency. No source leaves your runner.

ir-benchmark-live.json
212.26 kb
Download here:
https://app.devin.ai/attachments/b44c2c3c-9619-4914-a509-02a4e6c59a27/ir-benchmark-live.json