DEV Community: Vikas Srivastava

Your Agent Benchmark Pauses the World. Production Doesn't.

Vikas Srivastava — Sun, 18 Oct 2026 09:58:10 +0000

Your evals assume the world holds still

Almost every agent benchmark in wide use is turn-based. The harness hands the model an observation, the model emits an action, the harness applies the action, and then — crucially — nothing happens until the model produces its next token. The environment is a pure function of the interaction history. Time is logical, not physical.

That is a deliberate and usually correct simplification. It isolates reasoning from scheduling noise, it makes runs reproducible, and it lets you compare two checkpoints without worrying that one got a slower GPU. It is also a bad model of deployment.

In production, the world keeps moving while your agent thinks. A price changes between the quote and the order. A lease expires. A human answers the email you were drafting a reply to. Another replica of your own agent takes the lock you were about to take. A recent preprint, "The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock," takes this mismatch as its starting point. The title names the problem precisely enough that the engineering consequences are worth spelling out on their own.

What wall-clock time actually breaks

Two distinct failures get conflated under "the agent is slow," and separating them is the first useful step.

Staleness. An observation is a claim about the world at some instant. Turn-based harnesses make it a claim about the world now, forever. Once you add a clock, every observation carries an implicit age, and the agent's context becomes a mix of fresh and expired beliefs with no visual difference between them. A log line, a search result, and a file listing taken ninety seconds apart look identical to a transformer. The model has no way to discount the stale ones unless you tell it.

Validity windows. Some actions are only meaningful within a time band. A session token, a signed upload URL, a pagination cursor, a reservation hold, a one-time code — all of these are affordances that exist at t and are gone at t + Δ. In a frozen environment, a failed tool call means the agent made a mistake. In a live environment, it might just mean the agent deliberated too long.

The concrete failure modes that show up once you stop freezing the world:

Read-modify-write races. The agent reads state, spends seconds reasoning, then writes back a value derived from a snapshot that no longer exists.
Duplicate side effects. A tool call times out. The agent retries. The first call had actually succeeded, and now the customer has two orders.
Expired affordances. The plan is right, but step four uses a handle that expired during step three.
Blocking serialization. The agent waits on a synchronous tool while an independent subtask could have progressed in parallel — the plan is fine, the schedule is not.
Deadline overrun. The answer is correct and arrives after the SLA it was supposed to satisfy. Under a turn-based scorer, this is a perfect score.

What a live-clock evaluation requires

Mechanically, there are three ways to build one, and they trade off differently.

Real time. Run the harness against actual concurrent processes and score on wall-clock. Highest fidelity, lowest reproducibility: your numbers now depend on your serving stack, your network, and your machine's load.

Virtual-but-live time. Introduce a clock abstraction, let the environment advance on its own simulated ticks, and give the agent a monotonic clock it can read through the tool layer. This is the interesting design point. If the agent can query time and always gets a frozen value, you have reverted to turn-based evaluation with extra steps. The clock must be observable to the agent but not controllable by it, exactly like environment state. Observations must carry timestamps and the agent must be able to act on them.

Recorded timelines with replay. Log real interactions with their timestamps, then replay them against a controllable clock for scoring. Cheap, reproducible, and it lets you sweep the agent's own latency without re-running the world.

Two scoring decisions follow. First, a live-clock eval cannot report a single number. It should report a latency-accuracy curve: what fraction of tasks complete correctly within a budget, at several budgets. Second, because the interleaving of agent deliberation and environment change is nondeterministic, you need seeded scheduling and a distribution over seeds rather than one run. Some of those seeds should be adversarial — a change landing just after the agent reads.

Teaching, not just measuring

Training under a live clock changes the objective, not just the test. Latency becomes part of the action space: how long to deliberate, when to re-check before committing, when to act on a stale-but-good-enough belief.

Concretely, that means putting a clock reading into the observation, charging wall-clock time into the reward rather than treating it as free, and being careful about credit assignment. The outcome of an action depends on the state at the moment the action landed, not the state at episode start. Standard advantage estimation over a trajectory silently assumes those are the same, and under a live clock they are not.

What to do before the benchmark exists

You do not need a research harness to get most of the benefit. Start here:

Timestamp every observation you put in context, and state freshness requirements in the tool description ("this cursor is valid for 60 seconds").
Prefer compare-and-swap and idempotency keys over blind writes in your tool implementations, so retries are safe by construction.
Make timeouts return a distinguishable unknown, not a generic failure, and teach the agent to re-check rather than assume failure.
Log wall-clock per step. The histogram usually has a long tail that dominates the total, and it is often a small number of tools.
Add one eval where the environment mutates on a timer independent of your agent, and measure whether the agent notices.

Open questions

Is live-clock performance a capability axis or a robustness axis? It plausibly needs to be reported separately, because a slower model with better staleness handling can beat a faster one on the same tasks. Reproducibility is unresolved: seeded nondeterminism may be the honest default, but the community has no convention for it yet. Transfer is unproven — latency distributions in production are heavy-tailed and machine-specific, so training on one clock may not survive another. And multi-agent settings turn the environment from a Markov decision process into a game, where the other players also own clocks.

When AI Reviewers Find More Bugs Than Teams Can Fix

Vikas Srivastava — Sun, 18 Oct 2026 09:57:29 +0000

The bottleneck moved from finding defects to closing them

For most of the history of software engineering, the hard part of quality assurance has been discovery. Static analyzers, fuzzers, and human reviewers all share a common failure mode: they miss things. Teams invested enormous effort in raising recall — more coverage, more instrumentation, more reviewers — because a bug you never find is a bug you never fix.

The PostgreSQL 19 release cycle reportedly slipped for the opposite reason. AI-assisted review surfaced more candidate defects than the maintainers could triage and fix inside the release window. That inverts the usual assumption. It also generalizes: any project that wires a high-recall machine reviewer into a process whose throughput is set by human triage will eventually hit the same wall.

A review pipeline is a queue, and the queue is the release gate

Model the review process as a queueing system. Candidate defects arrive at some rate. Each one costs a human some amount of time to triage, and the real ones cost additional time to fix and verify. If the arrival rate exceeds the service rate, the backlog grows without bound.

What makes this sharp in practice is that release criteria are usually defined over the queue, not over the arrival rate. "No known release-blocking defects" is a statement about the state of the backlog. So a tool that doubles discovery without changing triage or fix capacity does not make the software better faster. It makes the release later, because the gate cannot close until the queue drains.

This is a throughput mismatch, not a tooling failure. The tool is doing what it was asked to do. The process around it was sized for a world where discovery was the constraint.

Why machine review inflates the queue faster than it drains it

Several mechanisms compound.

Generation is cheap; falsification is expensive. A model can emit a plausible-sounding defect report in milliseconds. Deciding whether it is real requires reading surrounding code, understanding intent, building or reproducing, and often asking the original author. The cost asymmetry is several orders of magnitude.
Reviewers are tuned for recall, and recall buys precision loss. Any detector optimized not to miss things will produce a long tail of low-confidence findings. Each one still costs a human minutes to dismiss, and dismissal is invisible work that produces no artifact.
Duplicates multiply. Multiple detectors, multiple prompts, multiple runs on the same diff will surface the same root cause repeatedly. Without clustering by root cause, N tools produce roughly N times the triage load for one bug.
Findings without reproducers are not actionable. "This looks like a race" is a hypothesis. A failing input or a deterministic interleaving is a work item. The former consumes triage time; the latter consumes fix time. Conflating them hides where the capacity actually goes.
Severity is uncalibrated. Models have no shared notion of what blocks a release. Without a severity gate, everything is nominally a candidate blocker, and the queue never has a floor.
Some findings are not bugs. A reviewer can correctly identify that a code path is fragile, under-documented, or relies on an invariant nobody enforces. Those are real observations that cannot be resolved by a patch. They need a design decision, which is a different and much slower process.

The net effect: the queue grows in units of human attention, and human attention is the one input the tool cannot supply.

Designing the pipeline so the queue stays drainable

The goal is not to suppress findings. It is to convert findings into work at a rate the team can absorb, and to make the conversion itself cheap.

Deduplicate by root cause, not by text similarity. Cluster findings that point at the same line, the same invariant, or the same failing input, and present one entry per cluster.
Require evidence before human triage. A finding enters the human queue only with a reproducer, a failing test, or a concrete input. Everything else goes to a machine-triaged backlog.
Rank by reachability and blast radius. A defect in a hot path that ships to users outranks a defect in a rarely exercised error branch, regardless of how confident the model is.
Make the fix, not the finding, the unit of work. Auto-generated candidate patches are useful because they let a human verify rather than author. Verification is faster than construction, and it keeps accountability with the person who merges.
Cap work in progress. An unbounded queue destroys prioritization. Explicit WIP limits force the team to decide what actually blocks the release.
Measure precision per detector and turn off the noisy ones. If a detector's findings are dismissed more than some fraction of the time, it is consuming capacity, not adding safety.

Verification is the scarce resource, and that is the real story

The same pattern shows up wherever models generate artifacts faster than humans can check them: generated code, generated tests, generated pull requests. Generation has become abundant. Verification has not, and there is no obvious reason it will, because verification requires the very judgment that makes the work hard in the first place.

That reframes what these tools are for. A reviewer that finds more bugs is only useful if the pipeline can turn findings into merged fixes. Otherwise it is a machine for manufacturing guilt — a growing list of known problems that nobody has time to address.

The open questions are uncomfortable. Can models be calibrated to emit only high-precision findings, or is triage itself the thing to automate, with a second model filtering the first? Who owns a regression that an AI reviewer flagged and a human dismissed? Does a backlog of machine-found defects eventually get paid down, or does it become a permanent, growing liability that teams learn to ignore? And how do you measure any of this — findings per change, triage minutes per finding, precision at the release-blocking threshold — in a way that tells you whether the tool is actually helping?

The queue is the artifact. Everything else is upstream of it.

When AI Reviewers Find More Bugs Than Maintainers Can Fix

Vikas Srivastava — Sun, 18 Oct 2026 09:38:35 +0000

The bottleneck moved, and nobody re-planned for it

For most of the history of software quality, finding bugs was the expensive part. You wrote tests, you ran fuzzers, you paid a security firm, you waited for a user to file a weird crash report. Fixing was comparatively cheap: once a defect is isolated with a reproduction, a competent maintainer can usually reason about it and write a patch.

AI-assisted review breaks that assumption. A language model reading a diff, or an agent that can execute code and mutate inputs, can generate candidate defects far faster than a human reviewer can evaluate them. The output is not a finished bug report. It is a claim: this code may be wrong, for this reason, under these conditions. Some fraction of those claims are correct and important. Some fraction are correct but already known. Some fraction are technically true but irrelevant. And some fraction are confidently wrong.

The Postgres 19 release slipping because AI reviewers surfaced more issues than the project could work through is the visible version of a dynamic that is quietly arriving in every large codebase. The headline sounds like a success story for AI. It is closer to a queueing problem.

Why AI review produces a flood, not a fix

To see why, it helps to be precise about the mechanism. There are two distinct techniques getting lumped together.

The first is static review by a language model. You feed a diff plus surrounding context and ask for defects. This scales trivially — you can run it on every commit, with several models, at several temperatures, and get several different answers. Recall goes up. So does noise. These systems cannot run the test suite, cannot check whether a code path is reachable in production, and cannot know that the project deliberately made a tradeoff four years ago. They are pattern matchers over text that resembles buggy code.

The second is agentic or fuzzing-based discovery, where the model writes inputs, runs the program, observes crashes or assertion failures, and iterates. This is slower and more expensive per finding, but the findings tend to be reproducible, because the agent had to actually trigger the failure to report it.

The first technique is what generates the flood. And a flood of unreproduced claims is the worst possible input to a review process, because evaluating a claim costs almost as much as fixing the bug if the claim is real.

What makes a finding actionable

A bug report is a unit of work, and its cost is dominated by triage. The properties that make one cheap to process:

Reproducibility. A failing test case, a minimal input, a stack trace. Anything the maintainer can run in under a minute.
Reachability. Evidence the code path is actually exercised, not just theoretically present.
Severity framing. Data corruption and remote crashes are different from a misleading log line.
Deduplication. Ten reports of the same off-by-one are one report. Models are extremely good at rediscovering the same issue across runs and across files.
Provenance. Which model, which prompt, which commit — so you can calibrate trust in the source over time.

Most AI review output has none of these by default. It arrives as prose. Someone has to turn prose into a ticket, and that someone is a maintainer who would rather be writing code.

The practical implications

If you are running AI review on your own repository, the design decisions that matter are mostly about queue management, not model quality.

Run review in a branch or a shadow pipeline and do not auto-file issues. Auto-filing converts model output directly into human obligation, which is how you burn out a team.

Cluster before you surface. Embed the findings, dedupe by similarity, and rank by a signal you trust — reachability, whether the flagged line changed recently, whether the file has a history of defects. A ranked list of twenty beats an unranked list of two hundred.

Prefer the expensive agent that reproduces over the cheap one that speculates, at least for anything you intend to act on. A finding with a failing test attached is worth an order of magnitude more than a finding with a paragraph attached.

Budget for remediation, not detection. Detection is now nearly free and getting cheaper. The line item that grows is engineer hours spent reading, reproducing, and patching — plus the regression risk of landing a large batch of fixes under time pressure, which is its own kind of hazard in a project with a long backpatch history.

Open questions

Who is accountable for a defect that a model reported, a human deprioritized, and an attacker later exploited? Disclosure norms assume a reporter and a maintainer. Adding a third party that can generate findings at arbitrary volume, with no stake in the project, has no precedent.

Can the same tooling that finds bugs help fix them? Early experience suggests models are better at locating a defect than at producing a patch that respects a codebase's conventions and does not introduce a subtler problem. Reviewing a generated patch is cheaper than writing one, but not free.

And the calibration question: how do you know your reviewer's precision? Without a labeled set of known-real and known-false findings from your own repository, you cannot set a threshold, and you will either drown in noise or ignore real defects. Building that set is unglamorous work, and it is probably the highest-leverage thing a team adopting AI review can do first.

Build a Tiny Causal Transformer from Scratch

Vikas Srivastava — Sun, 18 Oct 2026 05:44:07 +0000

The concrete problem: predicting the next token

A language model has a simple training objective: given a sequence of tokens, predict the token that comes next. In a character-level model, a token is one character, so the task might be predicting e after th. The same objective applies when tokens represent words or subword pieces; only the vocabulary and sequence length change.

The engineering challenge is making each prediction depend on the useful parts of the preceding context without allowing the model to read the future. During training, the model receives many positions at once, but the prediction at position 10 must be based only on positions 1 through 10. This information boundary is what makes the model causal.

We can build a small causal transformer to make that mechanism tangible. It will read a text file, encode characters as integers, train on randomly selected context windows, and generate new text one character at a time. Because every component is visible, the relationship between the data, objective, architecture, and generated output is easier to inspect than in a large prebuilt system.

How it works

The model first maps each token ID to a learned vector. Token IDs themselves have no useful numeric meaning, so the embedding gives each character a trainable representation. It adds a learned positional vector because attention alone does not know whether a token appeared first or last. The resulting sequence is passed through transformer blocks, with one vector for each position.

Each attention head creates queries, keys, and values by multiplying the token representations by learned matrices. Query-key dot products measure which earlier positions are relevant to the current position; softmax turns those scores into weights; the weighted values are combined into a new representation. Different heads can learn different relationships, such as nearby character patterns or recurring delimiters.

Causal masking is the critical constraint. Before softmax, positions to the right of each token receive negative infinity, so their probability becomes zero. During training, every position can therefore predict its next token in parallel while still obeying the same information boundary used during generation. Without this mask, the model could use the answer it is supposed to predict and would not be learning the intended task.

A block adds the attention result back to its input, then applies a position-wise feed-forward network and another residual connection. Layer normalization stabilizes these transformations. The final linear layer converts each hidden vector into logits over the vocabulary, and cross-entropy compares those logits with the known next characters. The training target is the input sequence shifted one position to the left: each character supplies context for the character that follows it.

Here is a complete miniature implementation. Put training text in input.txt; the code uses only PyTorch and the Python standard library.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(1)
device = "cuda" if torch.cuda.is_available() else "cpu"
text = open("input.txt", encoding="utf-8").read()
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for c, i in stoi.items()}
data = torch.tensor([stoi[c] for c in text], dtype=torch.long)
n, split = len(data), int(.9 * len(data))
train, valid = data[:split], data[split:]

block_size, batch_size = 128, 32
def batch(source):
    starts = torch.randint(len(source) - block_size, (batch_size,))
    x = torch.stack([source[i:i+block_size] for i in starts])
    y = torch.stack([source[i+1:i+block_size+1] for i in starts])
    return x.to(device), y.to(device)

class Head(nn.Module):
    def __init__(self, dim, head_size):
        super().__init__()
        self.key, self.query, self.value = [nn.Linear(dim, head_size, False)
                                            for _ in range(3)]
        self.register_buffer("mask", torch.tril(torch.ones(block_size, block_size)))
    def forward(self, x):
        k, q, v = self.key(x), self.query(x), self.value(x)
        scores = q @ k.transpose(-2, -1) * k.size(-1) ** -0.5
        scores = scores.masked_fill(self.mask[:x.size(1), :x.size(1)] == 0, float("-inf"))
        return F.softmax(scores, dim=-1) @ v

class Block(nn.Module):
    def __init__(self, dim, heads):
        super().__init__()
        size = dim // heads
        self.attn = nn.Linear(heads * size, dim)
        self.heads = nn.ModuleList([Head(dim, size) for _ in range(heads)])
        self.ff = nn.Sequential(nn.Linear(dim, 4 * dim), nn.ReLU(),
                                nn.Linear(4 * dim, dim))
        self.n1, self.n2 = nn.LayerNorm(dim), nn.LayerNorm(dim)
    def forward(self, x):
        a = torch.cat([h(self.n1(x)) for h in self.heads], dim=-1)
        x = x + self.attn(a)
        return x + self.ff(self.n2(x))

class Model(nn.Module):
    def __init__(self, vocab, dim=128, layers=4, heads=4):
        super().__init__()
        self.tok = nn.Embedding(vocab, dim)
        self.pos = nn.Embedding(block_size, dim)
        self.blocks = nn.Sequential(*[Block(dim, heads) for _ in range(layers)])
        self.norm, self.lm_head = nn.LayerNorm(dim), nn.Linear(dim, vocab)
    def forward(self, idx, targets=None):
        t = idx.size(1)
        x = self.tok(idx) + self.pos(torch.arange(t, device=device))
        logits = self.lm_head(self.norm(self.blocks(x)))
        loss = None if targets is None else F.cross_entropy(
            logits.reshape(-1, logits.size(-1)), targets.reshape(-1))
        return logits, loss
    @torch.no_grad()
    def generate(self, idx, steps):
        for _ in range(steps):
            context = idx[:, -block_size:]
            logits, _ = self(context)
            probs = F.softmax(logits[:, -1], dim=-1)
            idx = torch.cat([idx, torch.multinomial(probs, 1)], dim=1)
        return idx

model = Model(len(chars)).to(device)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
for step in range(2000):
    x, y = batch(train)
    _, loss = model(x, y)
    opt.zero_grad(set_to_none=True)
    loss.backward()
    opt.step()
    if step % 200 == 0:
        print(step, loss.item())

start = torch.zeros((1, 1), dtype=torch.long, device=device)
print("".join(itos[i] for i in model.generate(start, 500)[0].tolist()))

The batch function selects independent windows and creates the shifted targets. The generate method uses only the most recent block_size characters, computes probabilities for the final position, samples one character, and appends it before repeating. That loop is the training objective made operational: predict one next token, then use it as part of the next context.

What the evidence actually shows

This experiment demonstrates the architecture, not a competitive language model. On a small corpus, the model can learn spelling patterns, punctuation, local syntax, and recurring sequences; with enough capacity and training, it may mostly memorize the data. A convincing-looking sample can therefore reflect familiarity with the corpus rather than broad language understanding.

The useful evidence is in the loss and generated samples. Training loss should generally decline, while validation loss reveals whether the model is learning reusable structure or simply fitting the training text. The two views answer different questions: training loss describes optimization on seen examples, while validation loss tests behavior on held-out positions from the same data source.

If generated output becomes coherent only on familiar phrases, the model has learned distributional regularities but not robust generalization. Conversely, imperfect samples do not necessarily mean the mechanism is wrong; the corpus, number of updates, vocabulary, and context length all shape what the small model can learn. Samples are most useful when read alongside the loss.

The original transformer result established that attention-based sequence models could replace recurrent computation for major sequence tasks. Subsequent systems showed that increasing data, parameters, context, and training quality can produce much broader capabilities, but those results do not imply that a four-layer toy model will reason like a production model.

Using it

Start with a corpus large enough to fill many context windows, but small enough that iteration is quick. Plain text is useful because character tokenization removes vocabulary and tokenizer dependencies; it is also inefficient, since common words consume many positions. The character vocabulary is easy to inspect, but the model must spend several predictions spelling a word that a subword tokenizer might represent more compactly.

Watch the loss rather than treating a single sample as a metric. Add a validation batch and periodically evaluate it with model.eval() and torch.no_grad() in a real experiment. Save checkpoints containing model.state_dict() and the optimizer state so an interrupted run can resume. A checkpoint also lets you compare generations from the same trained state instead of accidentally comparing different training runs.

Change one variable at a time. Increase dim or the number of heads to test capacity, increase block_size to test longer dependencies, and adjust the learning rate before simply training longer. The attention matrix is also worth inspecting: it can show whether heads focus locally, attend to delimiters, or spread attention across the available context. Such inspection connects an abstract matrix operation to recognizable patterns in the text.

For reproducible comparisons, fix the random seed, keep the train/validation split stable, and record the parameter count, corpus, context length, and number of optimization steps. Generation temperature belongs to sampling, not training: lower values make output more predictable, while higher values explore lower-probability tokens. Keeping sampling settings fixed prevents a change in temperature from being mistaken for a change in model quality.

Limitations and failure modes

This implementation omits several production necessities. It uses character tokens, no dropout, no learning-rate schedule, no mixed precision, no distributed training, and no efficient attention kernel. Its feed-forward activation and normalization order are also deliberately simple. These omissions keep the implementation readable, but they also mean that training behavior and performance should not be generalized to larger systems.

Small datasets make memorization likely, and random context sampling can hide problems with document boundaries or duplicated text. A model may achieve a lower loss by exploiting formatting artifacts without learning the behavior you care about. Always inspect held-out examples and check whether generated text reproduces sensitive training material. Random windows can also mix contexts in ways that are convenient for training but do not represent the original document task.

The code assumes input.txt is longer than the context window and that its encoding is valid UTF-8. Larger models also make the quadratic attention cost visible: a sequence twice as long requires roughly four times as many pairwise attention scores per layer, before memory and optimization overheads. The context window therefore affects both what the model can connect and how expensive each forward pass becomes.

What to watch

The next useful step is not adding arbitrary layers; it is measuring what each change buys. Compare character and subword tokenization, visualize attention under controlled prompts, and test whether validation loss improves when the corpus contains longer-range structure. A controlled prompt makes it easier to distinguish a stable attention pattern from an appealing interpretation of one random sample.

From there, implement checkpointing and evaluation before scaling. A transformer becomes easier to reason about when its data pipeline, objective, masking rule, and sampling procedure are all small enough to inspect. Each change should have a corresponding observation: a loss curve, a held-out example, an attention visualization, or a generation under fixed sampling settings.

That discipline turns the miniature model into more than a toy. It provides a compact environment for asking concrete questions about context, capacity, optimization, and data quality, while keeping the code short enough that the answer can be traced back to a particular component.