DEV Community: Saurav Niraula

When the agent writes the patch, who audits the tests?

Saurav Niraula — Sun, 15 Nov 2026 18:12:35 +0000

In the METR randomized trial published in July 2025, sixteen experienced open-source developers worked on 246 real issues in their own repositories. Half the tasks allowed AI tools, half did not. The AI-allowed tasks took 19% longer. Before the study the same developers expected a 24% speedup, and after living through the slowdown they still believed they had been sped up by 20%.

That gap between what happened and what the developers felt is the most useful result in the paper, and it is not really about coding speed. It is about measurement. People are bad at knowing whether a tool helped, and the tool is now producing the thing that tells them.

Apply that to tests and you get the quiet failure nobody is writing postmortems about yet. The agent writes the code. The agent writes the tests. The agent runs the suite and reports green. Every step is plausible, the diff is clean, and the review takes four minutes because the checks passed.

The ways a test suite gets laundered

None of these require an adversarial model. They are the shortcuts a model takes when the goal is "make the suite pass," which is the goal you wrote in the prompt.

A new test that asserts the code's current behavior instead of the intended behavior. It passes on the first run, it has a nice name, and it would pass if the function returned zero for everything.

A snapshot that got regenerated. -u is one keystroke away and the resulting diff is unreviewable by construction, which is the whole reason it slips through.

A broad except or a try/except: pass inserted in the module, converting a real failure into silence that the test reads as success.

An assertion loosened from equality to truthiness. assertEqual(total, 41.50) becomes assertIsNotNone(total). Both are green. Only one is a test.

A deleted case, or a skipped one with a TODO. The count of tests stays roughly stable so nobody notices.

A timeout raised from 2s to 30s, which hides a performance regression behind a green check.

Every one of these shows up in the diff as a small, defensible line. That is what makes it survive review: nothing looks wrong in isolation, and reviewers anchor on the code file being the interesting part.

Coverage is not the control you think it is

Coverage measures whether a line executed, not whether anything was verified. A test that calls a function and asserts nothing gives you 100% coverage of that function and zero protection. An agent optimizing for coverage will produce exactly that, because it is the cheapest way to satisfy the metric you put in the prompt.

The metric is the problem. If you tell an agent "increase coverage to 85%," you have defined success as execution, and the model will deliver execution.

Mutation testing on the changed lines

The cheapest real signal I have found is mutation testing scoped to the diff. Take the changed lines, mutate them (flip a comparison, change a constant by one, negate a boolean, remove a return), and check whether the test suite notices. If a mutant survives, the tests around that line do not actually verify it.

Running mutation testing on a whole repository is too slow to put in CI. Running it on the diff is not:

# Python: mutate only the files this PR touched
git diff --name-only origin/main...HEAD -- '*.py' \
  | xargs mutmut run --paths-to-mutate {} --tests-dir tests

Stryker does the same for JavaScript with --mutate limited to changed files, PIT with --targetClasses for Java. The report is a number: what fraction of injected faults did the suite catch. A patch with 40% mutation score is a patch with tests that exist and do not verify.

This catches the laundered test directly, because a test that asserts nothing kills no mutants. It also catches the deleted case and the loosened assertion, which coverage never will.

One piece of evidence to require

Every AI-assisted pull request should carry a failing-before proof for the behavior it claims to fix.

The mechanics are simple: run the new test against the code from before the patch, and attach the failure. In git terms, stash the source change, keep the test, run it, capture the output. Then unstash and run it again. The artifact is a diff of two test runs, and it takes the model four seconds to produce if you ask for it in the same session.

git stash push -- src/            # source change only
npm test -- --testPathPattern changed.behavior
git stash pop
npm test -- --testPathPattern changed.behavior

If the test passes both times, it does not test the change. Whatever else is true about the patch, the verification is decorative. That single check invalidates most of the laundered-test patterns in one step, and unlike a coverage threshold it cannot be satisfied by writing more nothing.

Separate the two capabilities

Here is the part that is an architecture decision rather than a process one. The agent should not be able to edit tests and source in the same commit with the same approvals.

Split them. A code change lands with tests unchanged, or a test change lands with code unchanged and a reviewer who is explicitly looking at it. CODEOWNERS on the tests directory is the mechanism, and "tests modified in this PR" as a required label is the signal. It is a five-line policy that makes the failure mode visible instead of silently green.

The agent can still propose test changes. It just cannot be the only witness to its own work.

The honest version of the productivity story

METR's follow-up work in early 2026 suggests developer experience has improved since that first trial, with an estimated speedup among returning participants, while also warning that selection effects make the number unreliable. I believe the newer estimate is closer to reality than the old one. I also think the perception gap is still there, and it matters more now that the tool writes the tests as well as the code.

You cannot feel your way to "the tests verify the change." You can measure it. Mutation score on the diff, a failing-before artifact, and test files that a different person signed off on. Three checks, none of them expensive, all of them boring, which is what a control is supposed to look like.

"Zero upload" is a claim. Show me the network tab.

Saurav Niraula — Sun, 15 Nov 2026 18:10:59 +0000

A PDF merge tool went around my team last month. The headline was "your files never leave your device." I opened devtools out of habit. The page loaded a Google font, a session recorder, and a Sentry bundle, and it POSTed a page_view with the document title in the payload. The PDF itself never left. The filename, the page count, and the fact that I was merging a contract at 11:40 p.m. did.

The tool was not lying, exactly. It merged the files locally, which is a real engineering achievement and the reason I use these things. But "zero upload" is a security claim, and a security claim is worth what its enforcement is worth.

The architecture is right, and that's the frustrating part

Moving file processing into the browser is one of the better ideas to come out of the last few years. PDF parsing via a WASM build of pdfium or qpdf, image resizing on a canvas, CSV work in plain JS, zip handling with a streaming library. No server means no retention policy, no data processing agreement, no S3 bill, no bucket that someone forgets to make private. For a 12 MB PDF, the whole job fits in the tab.

Then the marketing outruns the engineering. "Bank-grade privacy" appears next to a page that calls four third parties before the first paint.

The gap is always the same three things: analytics, error reporting, and fonts. Sentry is the worst of them because its payloads are designed to carry context. File names, breadcrumbs, and the last network request can all end up in an event, and nobody reads the scrubbing config until a customer asks.

Make the claim enforceable

If you want to say the files never leave, put it in a header where it is checkable:

Content-Security-Policy:
  default-src 'self';
  script-src 'self' 'wasm-unsafe-eval';
  connect-src 'none';
  img-src 'self' data: blob:;
  font-src 'self';
  form-action 'none';
  base-uri 'none';
  frame-ancestors 'none'

connect-src 'none' blocks fetch, XHR, WebSocket, EventSource, and navigator.sendBeacon. That is the actual guarantee. Not the copy, not the privacy policy, the header. 'wasm-unsafe-eval' is there because compiling a WASM module is blocked by a strict script-src without it, which is a detail people discover at 2 a.m. the night before launch.

form-action 'none' and base-uri 'none' close the two dumb paths around connect-src. And if you truly ship no telemetry, put a comment in the repo explaining that the header is a product promise, so the next person who adds a chat widget gets a failing test instead of a silent policy change.

If you cannot set connect-src 'none' because you need a support chat and a payment SDK, that is fine. Say "processed locally, no files uploaded" instead. That sentence is true and defensible. The other one is a promise your own CDN can break.

The constraints nobody puts in the demo

The privacy story gets all the attention. The engineering is harder than the landing page suggests, and the limits are predictable:

Memory. WASM32 gives you a 4 GB address space in theory and browsers give you far less in practice, especially on mobile, where the tab gets killed without a warning you can catch. A 300 MB scanned PDF will not fit if you decode every page at once. You need to process page by page, write intermediate results to OPFS, and free aggressively. If your demo file is 200 KB, you have not tested the product.

Streaming. File.stream() gives you a ReadableStream, and TransformStream lets you pass chunks to a worker without materializing the whole file. That works beautifully for line-oriented formats, hashing pipelines, and archives. It does not work for PDFs, because PDF structure is random access by design, the xref table can live at the end of the file, and incremental parsing gets ugly fast. Know which case you are in before you promise streaming.

Hashing. WebCrypto's crypto.subtle.digest takes a full buffer, which defeats the point of streaming a 2 GB file. If you need an incremental SHA-256 you will end up pulling in a WASM implementation or a pure-JS one that is slow enough to notice. I have lost an afternoon to this more than once.

Jank. Parsing 400 pages on the main thread freezes the UI and Safari will helpfully kill the page for you. Worker plus postMessage with transferable ArrayBuffers, always. If your worker copies instead of transfers, you will watch memory double for no reason.

What I'd want to see before trusting one

A page that merges my files locally and proves it. Three things make that checkable in under a minute: the CSP header, a network tab that stays empty while the file processes, and source you can read or a build you can diff. If the tool is closed source, the header plus an empty network panel is still most of the way there, and it is far more than the privacy policy tells you.

None of this is an argument against the architecture. It is an argument against describing a local-first tool with the vocabulary of a security product while shipping the telemetry of an ad-supported one.

The PDF merge tool from my team's Slack got the second half right. The files stayed on the machine. The metadata did not, and nobody had written down which one the promise covered.

Your cron job will fire twice. Design for it.

Saurav Niraula — Sun, 15 Nov 2026 18:09:18 +0000

Every tutorial for scheduled work has the same shape. A cron expression, a route handler, a loop over users, a call to an email API. It runs on the author's machine for two days and then it ships.

Then production happens. The platform retries a 502 that actually succeeded. A deploy lands mid-run and two workers process the same batch. The retry after a rate limit sends the digest a second time. A customer forwards the duplicate to support, and someone writes a ticket titled "we got this twice," and that is the first time anybody thinks about delivery semantics.

Cron is a trigger, not a scheduler. The moment your work leaves the process, you are in at-least-once territory, and at-least-once means duplicates are not an edge case. They are the contract.

Make the duplicate impossible, not unlikely

The fix is not a sent_at column you check before sending, because check-then-send has a race and you will find it at 3 a.m. with two workers that both read null.

The fix is a unique constraint that decides the winner atomically, before any side effect:

import pg from "pg";
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });

async function claim(eventId) {
  const { rowCount } = await pool.query(
    `insert into processed_events (event_id, claimed_at)
     values ($1, now())
     on conflict (event_id) do nothing`,
    [eventId]
  );
  return rowCount === 1; // true only for the first caller
}

if (!(await claim(event.id))) return; // someone else has it, or we already did

Two details matter more than the code. The key must come from the event, not from a timestamp or a random UUID you generate at send time, and it must be stable across retries. If your queue redelivers with a fresh ID, the constraint does nothing for you. And on conflict do nothing returning rowCount = 0 tells you the event was already claimed, which is exactly the information you need to skip quietly instead of erroring.

For the recurring job itself, the same idea applies at the job level. A single-flight lock beats a if (running) return flag in memory, which does nothing when you run three replicas:

const { rows } = await pool.query(
  `select pg_try_advisory_lock(hashtext('nightly-digest')) as locked`
);
if (!rows[0].locked) return;
try {
  await runDigest();
} finally {
  await pool.query(`select pg_advisory_unlock(hashtext('nightly-digest'))`);
}

Bound the concurrency or drown

The other half of this problem is what happens when your provider pushes back. A loop over ten thousand users with await inside is slow but polite. A loop with Promise.all is fast and gets you a 429 within seconds, and then the retry storm makes it worse.

Bound the in-flight requests, and treat 429 as an instruction rather than an error:

import pLimit from "p-limit";
const limit = pLimit(4); // provider allows ~5 concurrent, leave headroom

async function sendWithRetry(payload, attempt = 0) {
  try {
    return await limit(() => post("/v1/messages", payload));
  } catch (err) {
    if (err.status === 429 && attempt < 5) {
      const retryAfter = Number(err.headers.get("retry-after") ?? 0);
      const backoff = retryAfter * 1000 || 2 ** attempt * 1000;
      await sleep(backoff + Math.random() * 250); // jitter
      return sendWithRetry(payload, attempt + 1);
    }
    throw err;
  }
}

Three things in there that get skipped in tutorials. Retry-After is honored when the server sends it, because the server knows better than your exponential curve. Jitter is added, because identical backoff across workers synchronizes them into a thundering herd. And there is an attempt cap, because a permanent failure should land in a dead letter queue where a human sees it, not in an infinite loop that burns your quota.

Cap it, then route it. A row in a failed_jobs table with the last error and the payload hash is worth more than any amount of clever retry logic, because it makes the failure visible instead of ambient.

Don't enqueue from inside the transaction you can still roll back

One pattern bites every team once. You write to the database and publish to the queue in the same handler. The write commits, the publish fails, and the work is lost. Or the write rolls back after the publish, and the consumer acts on a record that does not exist.

Use an outbox: write the event to a table in the same transaction as the state change, then have a separate poller publish and mark it sent. It costs one table and one query per tick, and it converts a distributed consistency problem into a local one.

create table outbox (
  id          bigserial primary key,
  topic       text not null,
  payload     jsonb not null,
  created_at  timestamptz not null default now(),
  sent_at     timestamptz
);
create index on outbox (id) where sent_at is null;

The poller claims rows with for update skip locked, which lets you run it on every replica without coordination. That is the whole trick, and it is the same trick as the idempotency key: let the database decide who wins.

The test that would have caught it

Before any of this, write the test that fails on the tutorial version of the code.

Fire the same event twice into your handler with the same event ID. Assert that exactly one email exists in the fake provider, that the second call returns 2xx, and that no error is thrown. Then add the harder one: two concurrent calls with the same ID, awaited together, asserting one send. That second test is the one that proves your constraint is doing the work instead of your application logic.

Both run in about fifty milliseconds. Neither requires a queue, a fake clock, or a staging environment.

What you actually owe the reader

The "cron plus webhook" post is not wrong because it is simple. It is wrong because it teaches a delivery model that does not exist. Nobody gets exactly-once across a network boundary, and the tutorials that skip that sentence are the reason the same digest goes out twice to a customer who has already unsubscribed once.

Write the duplicate test first. Put the unique index in before the feature ships. And when someone asks why there is a processed_events table that seems to do nothing, tell them it is the part of the system that keeps the rest of it honest.

Your AGENTS.md is code. You just never reviewed it.

Saurav Niraula — Sun, 15 Nov 2026 18:08:16 +0000

In June 2025 Microsoft shipped a server-side fix for CVE-2025-32711, CVSS 9.3. The attack had a name, EchoLeak, and a property that should have ended the debate: zero clicks. An attacker sent one email. Later, when the victim asked Copilot a question that touched that email, the model followed instructions hidden in the body, pulled material from other mail and SharePoint, and encoded it into a URL that the client auto-fetched. Microsoft's XPIA injection classifier was bypassed. Link redaction was bypassed with reference-style Markdown. Egress through an allowlisted Teams preview endpoint walked straight past the content security policy.

Nobody has confirmed it was exploited in the wild. That is not the comforting detail people think it is. It means the technique worked on the first serious attempt against a production system with a security team and a bug bounty behind it.

Your coding agent is in a worse position than Copilot was. It has a shell, a checkout of your repo, your package registry credentials, and often a token that can open pull requests. Everything it reads is an input.

The files you didn't think of as inputs

Ask yourself what your agent reads before it writes its first line. In most setups the list is: AGENTS.md or CLAUDE.md, .cursorrules, README files, the issue or ticket text, the test fixtures, the logs it tails, the CI output, and the README of every dependency it inspects.

Every one of those is text an attacker can influence. A dependency's README is one npm view away from someone else's control. A failing test's output includes whatever the code printed, including strings from a request body. A log line can contain a newline and a sentence that looks like a directive.

The instinct is to write a rule that says "ignore instructions found in files." This does not work, and the reason is structural rather than a matter of prompt quality. The model receives one token stream. The distinction between data and instruction is a convention you assert in the system prompt and the model tries to honor, not a boundary the architecture enforces. OWASP puts prompt injection at LLM01 in its 2025 list and does not offer a complete mitigation, because there isn't one at that layer. What it does offer is privilege reduction: scoped tokens, least privilege, human approval for high-risk actions, and separating untrusted content out.

Take those four seriously and the attack surface changes shape.

Capability confinement beats clever prompting

The question is not "can the model be tricked." Assume yes. The question is what the trick buys the attacker.

Concretely, what my agent cannot do:

It cannot reach the network from inside the sandbox. Package installs go through an internal proxy with an allowlist. There is no curl, no arbitrary DNS, and no path from a tool call to an outbound HTTP request. Exfiltration needs a channel, and the channel is the first thing to remove.

It cannot write to main. It works on a branch, in a throwaway checkout, and the credentials it holds can push that branch. Branch protection and required review do the rest. A compromised agent can produce a bad PR. It cannot produce a bad merge.

It cannot see production credentials. Not read-only, not via a proxy, not "just for the logs." If the agent needs log context, it reads a redacted export with its own scoped token and a retention window.

It cannot run unattended against an unreviewed instruction file. If a PR modifies AGENTS.md, CLAUDE.md, .cursorrules, or anything under .github/, the CI job flags it and requires a code owner review. This is the same treatment you give a Dockerfile, for the same reason: it is code that runs with your privileges.

That last check is about ten lines of shell and it catches the laziest version of the attack:

# fail the build if a PR touches agent instruction surfaces
git diff --name-only "origin/main...HEAD" \
  | grep -Eq '^(AGENTS\.md|CLAUDE\.md|\.cursorrules|\.github/.*|scripts/agent/.*)$' \
  && echo "agent-config change: needs code owner review" && exit 1

Not a complete defense. It is a speed bump on the path of least resistance, and speed bumps are what supply chain security actually consists of.

Log the reasoning, not just the call

The thing that made EchoLeak interesting to me was the provenance question. The model's answer looked like a normal answer. The malicious part was the URL it constructed, and the only way to notice was to know which input sentence produced it.

Tool call logs should carry the justification: which file, which line, which retrieval hit. When an agent runs rm -rf build at 4 a.m. because a file said to, the log should say justified_by: notes/SETUP.md:41. That turns an unexplained action into a reviewable one, and it is the cheapest forensic control you can add.

def tool_call(name, args, justification):
    audit.write({
        "tool": name,
        "args": redact(args),
        "justified_by": f"{justification['path']}:{justification['line']}",
        "approved": False,
    })

Two fields. justified_by and the fact that approved starts false. Anything that mutates state flips that flag only through the gate you built on purpose.

The supply chain point

We spent a decade learning that a dependency can execute arbitrary code at install time, and we built lockfiles, provenance attestations, and allowlists for it. Agent instruction files are a dependency with no lockfile and no registry. They arrive through pull requests from strangers, they change without a version bump, and they execute with more privilege than most of your npm tree.

If you take one thing from the last six months of these disclosures, make it this: the model reading your workspace is executing instructions, not reading documentation. Review the instructions. Then take away the permissions you would not give a new hire on their first day.

References: CVE-2025-32711 (EchoLeak), disclosed by Aim Security, patched by Microsoft in June 2025. OWASP Top 10 for LLM Applications 2025, LLM01 Prompt Injection. Analysis in arXiv:2509.10540.

Human-in-the-loop at 3 a.m. is a rubber stamp

Saurav Niraula — Sun, 15 Nov 2026 18:08:05 +0000

03:12. A message in the incident channel: "Agent wants to roll back payments-api to v2.13. Approve?"

A thumbs up emoji. Two seconds. The rollback was wrong, and it took another forty minutes to find out, because the approver was half awake and had no idea what the agent had actually done.

Everyone building incident agents this month is adding exactly this feature: a human approval step before the agent touches production. Sixty-eight posts on DEV in the last week alone, most of them integrations with some chat product. It is the control OWASP recommends for high-risk agent actions, and it is the right instinct. The implementation is usually theater.

A gate is a decision, not a click

An approval is a security control only if the person clicking can answer four questions in under thirty seconds:

What exactly will change? Not "roll back the service." The plan: image tag, replica count, migration state, and whether anything is being reverted that cannot be re-applied.

What is the blast radius? Which services, which customer tiers, how many rows, whether the previous attempt at this is in the incident timeline.

Can we undo it? Reversibility, and the time it takes to reverse. A rollback across a schema migration is not the same action as a rollback of application code, and the prompt should not look identical for both.

Is this the plan that was reviewed, or a slightly different one? Hash it.

If the message in Slack contains none of those, you did not build a human gate. You built a notification with a button, and you gave the agent a path to production that costs one emoji.

The change envelope

I make the agent emit a fixed structure before it is allowed to ask for anything. If the structure is incomplete, the approval request does not render.

{
  "id": "chg_01J8Z6",
  "requested_by": "agent/incident-crew",
  "service": "payments-api",
  "env": "prod",
  "plan": {
    "kind": "rollback",
    "from_image": "payments-api@sha256:9f2c...",
    "to_image": "payments-api@sha256:41ab...",
    "commands": ["kubectl -n prod set image deploy/payments-api api=...@sha256:41ab..."],
    "migrations_in_scope": []
  },
  "blast_radius": { "services": ["payments-api"], "customer_tiers": ["all"], "est_requests_affected": 4100 },
  "reversible": true,
  "reverse_plan_eta_seconds": 180,
  "evidence": ["INC-2291", "deploy/4412"],
  "plan_sha256": "b7d1e0...",
  "expires_at": "2026-11-16T03:22:00Z"
}

Four fields do most of the work: plan_sha256 (the thing you approve is the thing that runs), reversible plus the reversal ETA, blast_radius, and expires_at. That last one matters more than it looks. An approval from 03:12 must not still be valid at 09:40 when the on-call has gone to bed and a different incident is running.

The executor verifies the hash before it applies anything. If the plan changed after the approval, the approval is void. Without that check, your "human approved" audit trail is a claim about an intention, not about an action.

Measure the gate or it isn't one

Two numbers, pulled from the approval log, will tell you whether you have a control or a formality.

Median time from request to approval. If it is under ten seconds, the reviewer is pattern-matching the name of the service, not reading the plan. That is fine for a canary bump and not fine for anything else, and the difference should be visible in the data rather than in your architecture diagram.

Reversal rate within twenty-four hours. How often an approved agent change gets undone by a human shortly after. A high number means the gate is approving things it should be rejecting, and the agent's plans are worse than its confidence suggests.

I also log the approver's identity and the incident they were paged for. When a postmortem says "the agent rolled back the wrong service and a human approved it," the useful question is not why the agent was wrong. It is why a tired person with three other tabs open was asked to be the last line of defense for an irreversible action at 3 a.m.

Default reversible, gate the rest

The better design inverts the usual instinct. Instead of gating everything and hoping the reviewer is sharp, make almost everything reversible and stop gating it:

Ship behind a flag with a kill switch. Canary at 5% with an automatic abort on error-rate burn. Cap the blast radius in the mechanism, not in the prompt: quota limits, a ten-minute change window, a replica ceiling. Then the agent can move fast without asking, because being wrong is cheap and self-limiting.

Keep the human gate for the short list that cannot be undone: schema migrations with data loss, deleting a bucket, rotating a credential that other services cache, force-pushing a branch, anything that touches billing records. That list is usually five items long. Write it down. Make the agent prove which one it is in.

That is a much smaller promise than "human in the loop," and it is the only version I have seen hold up under a real incident.

The part people don't want to hear

"Human in the loop" reads like a safety property. In practice it is a staffing assumption. Every approval queue needs someone awake, competent, and holding enough context to say no. If your incident rotation is three engineers and your agents file twenty change requests a night, you have not added a control. You have added a queue with a latency SLO nobody wrote down.

The gate should be a circuit breaker with a person on the other side of it, not a poll. Build the breaker first. Then decide which changes are expensive enough to wake someone up for, and be honest in the postmortem when the answer was "none of them, we just liked the feeling of approving."

One more thing worth saying plainly: if your approval message is a chat bubble with a service name in it, an attacker with write access to that channel, or to the agent's own context, can produce a plausible request. The envelope does not stop that. It just makes the request falsifiable, which is the most a prompt was ever going to do.

Your on-call agent doesn't need memory. It needs receipts.

Saurav Niraula — Sun, 15 Nov 2026 18:07:43 +0000

At 02:47 my agent said: "We already tried increasing the pool size. It didn't help."

I believed it for about ninety seconds. Then I checked. Nobody had tried that. Someone had suggested it in a thread six weeks earlier, and the agent had filed the suggestion away as a fact, then handed it back to me as institutional knowledge.

That is the failure mode nobody demos.

Sixty-nine posts about on-call agents with memory landed on DEV in the last week and a surprising number of them share a title. I read a stack. The architecture is usually the same: embed the postmortems, retrieve the top five chunks, paste them into the prompt, call it hindsight. It works in the demo because the demo has three documents and no history of being wrong.

In production you get a retrieval system that cannot tell the difference between something that happened and something someone once said. That is not memory. That is a rumor mill with embeddings, and it is worse than a blank slate, because a blank slate doesn't get cited.

What memory actually requires

An agent that remembers usefully needs five properties, and vector similarity gives you none of them:

A structured record of what was tried, on what, and what happened.
A pointer to evidence a human can open in one click.
A scope narrow enough that the fact is still true where you're applying it.
A way to be superseded.
A way to expire.

The record is the easy part. Everyone stops there. The other four are where the work is.

Here is the table I use. It is deliberately boring, and every column exists because a specific page went badly.

create table episode (
  id            bigserial primary key,
  service       text not null,
  env           text not null,
  symptom       text not null,        -- "p99 on /checkout above 4s"
  action        text not null,        -- "pool 20 -> 40"
  outcome       text not null check (outcome in ('fixed','failed','inconclusive')),
  evidence_url  text not null,        -- incident, PR, or log permalink
  version_from  text,
  version_to    text,
  observed_at   timestamptz not null,
  superseded_by bigint references episode(id),
  expires_at    timestamptz
);

evidence_url is not optional and there is no default value. If the agent cannot point at an incident, a pull request, or a permalinked log line, the fact does not get written. I would rather lose a real lesson than store an unattributable one, because the unattributable ones are exactly the ones that come back wrong at 3 a.m. sounding confident.

version_from and version_to are there because most operational facts have a half-life shorter than the documentation about them. "Raising the pool size helped" was true on Postgres 14 with the old pooler. It stopped being true when we swapped the pooler and the fix became "stop opening a connection per request." An agent that stores the first sentence and not the version range is storing a landmine.

Retrieval is a filter, not a search

The part I got wrong first was ranking. Semantic search over incidents will happily return the most similar-sounding incident from eighteen months ago on a service that has since been rewritten. Similarity is not relevance.

Retrieval should filter hard before it ranks:

select action, outcome, evidence_url
from episode
where service = $1
  and env     = $2
  and superseded_by is null
  and (expires_at is null or expires_at > now())
  and (version_to is null or version_to = $3)
  and outcome <> 'inconclusive'
order by observed_at desc
limit 5;

Five rows, not fifty. If the agent needs more than five past episodes to form a hypothesis, it does not have a hypothesis, it has a vibe.

Negative memory needs an expiry date

The most useful thing my agent learned to do was say "the rollback failed last time." The most dangerous thing it learned to do was say "rollbacks don't work."

Both sentences came out of the same table. The difference is scope. A failed action is evidence about one version, one environment, one failure mode. The moment you let the agent generalize it into a law, you have taught it to refuse the fix that would have worked.

So negative episodes get the tightest scope you can write and a mandatory expires_at, default ninety days. If the underlying cause is still true after ninety days, a human will re-record it, and that re-recording is a signal that the problem is structural rather than incidental. If it isn't re-recorded, the memory quietly stops influencing answers. Decay by default.

Humans write, agents propose

This is the part that gets skipped in every architecture diagram. Writing to memory is a privileged operation, and the agent does not have that privilege.

The flow I use: the agent proposes an episode with outcome = 'inconclusive' and the evidence it has. The human who closed the incident converts it to fixed or failed and confirms the scope. No human, no promotion. The agent's only write is a draft it cannot read back as fact.

That single rule killed most of my false recalls. It also produced a useful side effect: the promote step is where people noticed that half their "fixes" were actually coincidences, and started writing postmortems that said so.

What I measure now

Three numbers, all cheap:

Citation rate. Of the episodes the agent cited in the last thirty days, how many were opened by a human? Anything never opened is either irrelevant or wrong, and I'd like to know which.

Contradiction rate. How often two retrieved episodes disagree on the same service and action. A high rate means my scope columns are too loose.

Time to first correct hypothesis. From page start to the first theory a human says "yes, that's plausible" out loud. This is the only metric that matters, and it is the one nobody puts in the blog post.

The uncomfortable part

Memory is a liability with a maintenance cost. Every row you store can be retrieved, cited, and acted on by something with cluster write access at a moment when nobody is awake to check it. That is a different risk class from a search box over your docs.

If you are building one of these agents this month, and the trend suggests a lot of you are, spend the first day on the schema and the write path, not the embedding model. The model was never the hard part. Knowing whether the thing you just remembered is true is the hard part, and the only durable answer is a link to evidence and a date after which you stop believing it.

Jensen Huang is gone, and Nvidia has to explain itself on Tuesday

Saurav Niraula — Sun, 15 Nov 2026 17:30:00 +0000

I've been following Nvidia for years and I still don't have a clean way to describe the last two weeks. So here's the plain version.
Jensen Huang resigned as CEO on November 12. Not a handover, not a planned transition, just gone, effective immediately, after 33 years running the company he co-founded in 1993. Colette Kress, the CFO since 2013, is interim CEO and is somehow still doing the CFO job at the same time.
What ended it wasn't a product miss or a bad quarter. It started on October 29, when the New York Times reported on Huang's relationship with a businesswoman, Alice Huang, whom US intelligence suspects of being a Chinese spy. More reporting followed: an affair with a senior Nvidia employee, and another with someone who reported directly to him. The board opened an independent investigation, and by November 12 it was done.
I want to be careful here. A suspected spy is not a proven spy, and an affair is not automatically a national security breach. Nobody has shown that anything was leaked. But the board decided it couldn't wait for that answer, and once the second relationship with a direct report surfaced, sentiment turned fast. Someone familiar with the decision told CNBC the board worked late into the night and made the call that morning.

What the market did

The stock got hammered. Nvidia shed more than half a trillion dollars in market value at the worst point and closed November 13 at $162.85. That's about 24% below where it sat before the first story, and roughly 33% off its 52-week high of $243.37 from October 6. November is now on track to be Nvidia's worst month since ChatGPT kicked off the boom in late 2022.
Nine investment houses downgraded it. Bernstein halved its price target to $200 and moved to market-perform; Stacy Rasgon called the stock cheap but warned more allegations could still surface. Oppenheimer went to perform and floated a "brain drain," naming AMD, Marvell and Broadcom as the beneficiaries. Stifel kept its buy rating but cut the target hard, from $315 to $205.
Meanwhile Elon Musk and a pile of other tech execs signed a petition asking Nvidia to bring Huang back. It passed 115,000 signatures. Two investors holding about $2.13 billion of stock are pushing the board to at least consider his return. And Trump, predictably, praised him and floated Tim Cook as the replacement.

Tuesday's earnings call is the real test

Nvidia reports its fiscal third quarter on November 17, after the close. The numbers will probably look great: revenue around $109 billion, up roughly 50% from a year ago, and earnings per share near $2.47. But this is the first earnings call in the company's 27 years as a public company that Jensen Huang won't run. Kress has to make the case five days into the job, while still being the CFO.
I don't envy her. The quarter itself is almost beside the point. Everyone wants to know who runs Nvidia next, what happens to the roadmap over the next year or two, and whether the "Jensen premium" was a real thing that just got deleted. UBS also flagged that Nvidia might pause part of its funding of AI companies, which would pull something like $50 billion out of the ecosystem, plus the risk of tighter chip export rules on top.

What actually matters underneath

Huang has said he's willing to testify before Congress before Thanksgiving. Senator Elizabeth Warren's reply was blunt: resigning doesn't absolve him or the company of responsibility for putting national security at risk. There are already two congressional investigations into Nvidia's China sales. Separately, a former OpenAI researcher claims GPU fingerprinting shows models from DeepSeek, Moonshot, MiniMax and Ant were trained on Blackwell chips, despite the export controls meant to stop exactly that.
Take those together and the headline isn't really one CEO's personal life. It's whether the most important company in the AI supply chain can answer hard questions about who it sells to and what it knows. The leadership mess just dragged the question into the open earlier than anyone planned.

Where I land

I'm not going to pretend I know what the stock does next. The bulls have a real case: a forward P/E around 11 is cheap for a company growing this fast, and demand for AI compute hasn't gone anywhere. The bears have a real case too: no obvious successor, a board that didn't even have a chair until last week, and a national security cloud that one good earnings report won't clear.
What I'll actually watch on Tuesday isn't the revenue line. It's whether Kress sounds like a caretaker or a CEO, and whether she gives a straight answer about the roadmap and the buyback. That's the first real signal of what Nvidia is for the next two years. Everything else this month has been noise, just noise with an unusually large price tag.