DEV Community: JaviMaligno

Coding Agents and Teamwork: Social Skills, or Structure?

JaviMaligno — Sun, 12 Jul 2026 12:23:19 +0000

A complete study on a 19-task slice of CooperBench's 652. Every condition ran to completion, the headline cells have three seeds, and I audited the eval harness itself — all on a subset I could run cleanly across two model tiers. The ceiling is coverage, not confidence: within this slice the pattern replicates; whether it holds across the full benchmark is the open question.

A recent Stanford benchmark, CooperBench ("Why Coding Agents Cannot be Your Teammates Yet"), reports a striking result: when you pair up two strong coding agents to split a task, their success rate roughly halves versus a single agent doing the same total work. They call it the curse of coordination, and their reading is that the missing ingredient is social intelligence — agents don't use language for coordination reliably, so, they argue, this needs to be trained, not prompted away.

It's a careful paper, and I want to be fair to it: they deliberately run agents with almost no scaffolding, precisely to measure intrinsic coordination ability. That's a legitimate scientific choice. But it left me with an engineering question they explicitly set aside:

Human teams don't coordinate well with zero process either. We don't fix a team that keeps stepping on each other with therapy — we use structure: code ownership, interface contracts, sequential PRs, a reviewer who integrates. So how much of the "coordination gap" is really about social skills, and how much is just the absence of structure the agents were never given?

The setup

I started from CooperBench's own benchmark (it's open source) and reproduced the baseline: two agents, each assigned one feature of a task, working in separate containers but connected by a message channel they can use to coordinate at any time (the benchmark gives them this from the start), with their patches merged afterwards. That channel matters for everything below: the agents can always talk — the open question is whether talking is enough. Then I added a ladder of coordination structures, ordered from "advisory" (the agent can ignore it) to "enforced" (the scaffold guarantees it):

Handshake (C1): agents must exchange a plan before the scaffold lets them edit code. (Analogy: a design review before coding.)
File ownership (C2): each file is owned by one agent; edits to another agent's file are reverted. (Analogy: CODEOWNERS.)
Line-range ownership (C2b): finer version — both may edit the same file, but only in disjoint regions. (Analogy: not stepping on each other's functions.)
Sequential pipeline (C3): agent A implements feature 1 and commits; agent B starts from A's code and adds feature 2. No concurrency. (Analogy: small, sequential PRs on trunk.)
Integrator (C4): A and B work independently; a third agent reconciles both patches. (Analogy: an integration engineer / PR reviewer.)

The point of the ladder is to separate "did they talk?" from "did the process stop them from colliding?" — and to find the minimum structure that helps.

Results

Caveats first, because they matter: these are 19 task-pairs across 5 repositories — a subset I could run cleanly — not the full benchmark. But I ran the whole ladder on two capability tiers (a mid-tier gpt-5.4-mini and a stronger gpt-5.4), and the four headline conditions were run three times each so I can show a range instead of a single noisy number.

condition	mini	gpt-5.4
solo (one agent, both features)	14% [11–16]	30% [21–37]
coop (two agents, free-form)	7% [0–11]	7% [0–16]
coop + fair conflict resolution*	7% [0–11]	9% [0–16]
team — the paper's own (lead + member, shared task list)†	0%	0%
sequential (C3)	26% [21–37]	35% [32–37]
integrator (C4)	23% [21–26]	33% [26–42]
file ownership (C2)	0% [0–0]	14% [11–16]
handshake (C1)†	0%	0%
line ownership (C2b)†	0%	0%

Headline cells and C2 are mean [min–max] over three seeds; †team, C1 and C2b are a single run each — I only replicated the conditions with any signal to defend (all three sat at 0% for the weak model and never beat plain coop, so a second seed wasn't going to change the story). *The paper's eval auto-resolves trivial merge conflicts with a small model before declaring failure; the open-source release omits that step. I added an equivalent (if anything stronger) resolver and re-scored the concurrent conditions — details in "Under the hood".

Four things stand out:

The coordination gap reproduces — and it's wider for the stronger model. Two agents working concurrently (coop) score below one agent doing the whole task (solo): 14%→7% for mini, 30%→7% for gpt-5.4. Pooled across tiers and seeds the drop is solid (paired McNemar p=0.002); split out, it's clearly significant for the strong model (p=0.002) and not for the weak one (p=0.39) — simply because mini's solo baseline is already low, so there's less to lose. The curse of coordination bites hardest where a single agent was actually competent.
Two structures recover it — and sequencing does more than recover. The sequential pipeline (C3) and the integrator (C4) both beat coop decisively at both tiers (p<0.0001). They share a mechanism: one agent ends up owning the final integrated state — B builds directly on A's finished work, or a reviewer reconciles both patches in a single workspace. But the two aren't equivalent. Sequencing doesn't just claw back to solo — for the weak model it beats solo (mini 26% vs 14%, ahead in all three seeds, p=0.039), while merely tying it for the strong one (large 35% vs 30%, p=0.45). The integrator recovers over coop but never significantly beats solo (p=0.23). Read plainly: decomposing the task so agent B faces "add feature 2 to a codebase that already has feature 1" is a smaller problem than "build both at once" — and a smaller problem helps most whoever was struggling with the big one.
So structure interacts with capability — in both directions. That's the genuinely interesting part, and it cuts two ways. Sequencing helps the weak model more (it lifts mini above its own solo baseline; it only matches solo for the strong one). Enforced file ownership (C2) is the mirror image — across three seeds it recovered nothing for the weak model (0% in all three) but reached a stable 14% [11–16] for the strong one. Two opposite-signed interactions: lightening the cognitive load (sequencing) rescues the weaker model, while a rule it must actively exploit (ownership) only pays off for the stronger one. One honest deflation, though: C2's 14% is not significantly better than coop once coop's merges are scored fairly (14% vs 9%, McNemar p=0.73 across the three seeds), and it stays far below solo (0-of-13 tasks where they differ, p<0.001). So the fence's value over a decent merge pipeline is real-looking but statistically a wash — the capability interaction shows up in who can use the rule at all, not in a scoreboard win.
Forcing communication does nothing — because they already communicate. The handshake gate (C1) never even fired: agents already message each other before touching code (first message on turn 2, first edit on turn 6). Making them talk more changed nothing, at either tier. This echoes CooperBench's own "communication doesn't help" result — and suggests the problem isn't whether they exchange information, but what they do with it.

A smaller, slightly uncomfortable note: finer line-range ownership (C2b) did worse than coarse file ownership (C2, 16%) for the strong model. I first suspected an artifact of my v1 enforcement, which reverted the whole file on any overlap — so I built a v2 that reverts only the violating hunks and re-ran both tiers: 0% at both. The binding constraint isn't revert granularity. It's that CooperBench features intentionally overlap on shared lines (the same import list, the same function signature), so any exclusive line ownership starves whichever agent arrives second.

One logged run shows the failure in miniature. Agent 1, blocked by the territory guard, escalates exactly the way a person would — announces, asks, and finally delegates — while agent 2 says its piece once and then goes quiet:

agent 1 → agent 2

I'm going to touch other.py and init.py for an IsMac validator. Are you touching either of those? …so I can keep the patch merge-friendly?

agent 2 → agent 1

I'm about to add an IsEmail validator. I expect to touch _other.py and init.py too. …can we coordinate to avoid conflicts?

agent 1 → agent 2

The guard reverted my first attempt — claimed regions. Can you avoid the import block and all_, and the top of _other.py?

agent 1 → agent 2 · delegating

The guard is blocking me from the import/export regions. Could you add the IsMac re-export there, since you own that region?

agent 2 never replies. In its own private reasoning it wrote: "We do overlap. I should inspect and respond with precise coordination." It never did.

I checked the plumbing on both sides: all of agent 1's messages landed in agent 2's context with turns to spare, and agent 2's outbound channel demonstrably worked — it sent the first message of the whole thread. It read the requests, even planned to coordinate in its own reasoning, and then never replied, never added the one-line export it owned, and closed out its task. The bottleneck isn't the communication channel; it's follow-through — precisely the commitment failure CooperBench's taxonomy describes, reproduced here with the channel verified blameless.

Under the hood: collaboration dies at the merge

Digging into how each condition fails changes the picture more than the pass rates do.

Free-form coop fails at the merge, not at the code. 14–15 of 19 coop failures (both tiers) are outright merge conflicts — the two agents' patches can't even be combined. And here's the striking part: enforced ownership drives merge conflicts to zero — 0 of 19 for both C2 and C2b, at both tiers. The failure then moves downstream: patches combine cleanly but the integrated code doesn't pass both suites.

But is dying at a merge conflict too cheap a death? (Isn't resolving conflicts what PR review is for?) It's a fair objection — and it turned out to be aimed at a real gap: the paper's own eval resolves trivial conflicts with a small trained model before declaring failure, but the open-source release omits that step, so my first table scored every conflict as an instant fail. I checked it both ways. First, I triaged all 29 conflicted pairs by hand: roughly half the conflicts aren't real logic collisions — two agents adding a different import, or a different keyword argument, to the same line; git flags it, the code wouldn't care. Then I added a trivial-conflict resolver (using a stronger model than the paper's, so coop gets every benefit of the doubt) and re-scored the conflicted pairs across all three seeds. It rescued plenty of merges — but almost none of them became passes: fair-merge coop lands at 7% for mini (unchanged) and 9% for gpt-5.4 (up from 7%), a single extra task. The conflicts were mostly masking semantic failures further down: in one rescued pair, one feature passed 71/71 tests and the other failed a genuinely broken assertion the conflict had been hiding.

So the causal chain reads:

unstructured concurrency → ~75% merge conflicts → the syntactic layer can be fixed two ways — prevention (enforced territory: zero conflicts) or cure (a trivial-conflict resolver) → what's left is a semantic integration problem → and neither fences nor resolvers fix that. Sequencing or an integrator do.

This rhymes with CooperBench's own observation that communication reduces conflicts without improving success — the syntactic layer is simply not where the task is won.

The "0%" rows hide half-done work. Scoring requires both features to pass. Count partial success (at least one feature working) and the weak model's picture shifts: free-form coop lands only a handful of single-feature passes, while enforced ownership roughly doubles them (mini reaches 5/19 under line-range enforcement). Enforcement turns total collapse into "half the job done" more often — it's just not enough to clear the both-features bar.

The paper's own structured team is the worst of all — and that's the point. CooperBench ships a richer coordination mode than plain coop: a lead and a member with a shared task list, the lead responsible for integrating both features and shipping one patch. It scored 0% at both tiers — below even free-form coop. The partial credit is the tell: the lead passes its own feature in 7/19 (weak) and 11/19 (strong) pairs, and never both. It does its half and drops the partner's — the same follow-through failure as coop, except the richer protocol adds a manual hand-off step (the lead pulls the member's patch from a shared workspace and stitches it in) that turns out more fragile than letting the merge tool do it. More coordination structure, more places to fumble the hand-off. (A rigor note, since a flat 0% deserves suspicion: the open-source eval routed team like coop and merged both patches — double-applying the member's work, which is already inside the lead's patch. I fixed it to score the lead's shipped artifact instead; the number didn't move, but now it's measuring the right thing.)

Both tiers hit the fence equally often; only one recovers. The enforcement hook fired a similar number of times for both models (≈18–29 violations per run). The strong model isn't "tidier" — it violates territory just as much. The difference is what happens after the revert: the strong model re-plans and routes around it; the weak one keeps banging. All three of the strong model's C2 passes had enforcement events — the mechanism was active in every one of them, they weren't conflict-free freebies.

Statistics, honestly. The four headline conditions and C2 have three seeds each; C1 and C2b are single runs (both flat at 0% for the weak model). Paired McNemar tests (matched per task and seed) give: the coordination gap solid pooled (p=0.002) and clearly significant for the strong model alone (p=0.002); sequencing and the integrator each beat coop overwhelmingly (p<0.0001); sequencing beats *solo* pooled (p=0.021) and for the weak model alone (p=0.039, ahead in all three seeds), while only tying solo for the strong one (p=0.45); the integrator recovers but never significantly beats solo (p=0.23). One honest caveat on the method: pooling discordant pairs across seeds treats each (task, seed) as independent, which mildly overstates significance — so I lean on the per-seed direction (mini seq > solo in 3/3 seeds) as much as on the p-value. With three seeds on a 19-task subset these are still directional; I'd want a larger repeated run before hardening any single number.

Teamwork has a price tag. For the strong model, per run: solo cost ~\$3.60; sequential ~\$7.10 and the integrator ~\$9.73. Sequencing roughly doubles the bill to modestly beat solo; the integrator nearly triples it to match. On small, coupled tasks, if you must collaborate, sequence; if you can avoid it, one capable agent is the cheapest strong option.

Can you stack the fixes?

If enforced territory kills syntactic conflicts and an integrator owns the merge, surely combining them beats either alone? I tried three combinations: territory + integrator, sequencing + reviewer, and handshake + territory. Short answer: none robustly beats its own components.

The instructive one is territory + integrator. At one seed it looked like the star of the whole experiment — 42% for the strong model, seemingly ahead of the integrator alone. Then I ran two more seeds and it collapsed to 33% [21–42]: the 42% was a lucky draw, and paired against the plain integrator across all three seeds it comes out at p=1.0 — indistinguishable. The replicates existed precisely to catch that kind of over-claim, and they did.

But a cleaner measurement rescues something real. For each pair I re-scored the tree without the extra stage, on the very same two agent patches — so the only variable is the extra stage itself, no run-to-run noise. The integrator, measured this way, rescued 8 pairs and broke 0; the reviewer (on top of a sequential pipeline) rescued 3 and broke 3, a net zero. So the integrator's contribution is genuine and one-directional — it only ever helps. The reason territory + integrator still doesn't win on the scoreboard: the territory-constrained development phase hands the integrator no better raw material than plain independent development does, so a real but modest boost starts from equally noisy inputs and washes out across seeds. The reviewer nets zero for a simpler reason — a sequential pipeline already integrated the tree, so there's nothing left to reconcile, only code to accidentally break.

The lesson isn't "structure doesn't stack." It's narrower: stacking composes only when an earlier stage improves the inputs to a later one. Sequencing→review fails that test (nothing to improve); territory→integrate fails it too (territory doesn't make better patches, just non-overlapping ones). The one lever that keeps working is the same one from the single conditions — one agent owning the integration — and doubling down on it by combination didn't buy more.

What this does and doesn't say

It does not say CooperBench is wrong — I reproduced their gap at both tiers. What it adds is the piece their setup deliberately left out: once you give agents structure, does the gap close? On this subset, partly — and which structure works depends on both the structure and the model.

The honest headline isn't "structure beats social skills." It's narrower and, I think, more useful:

The most reliable lever wasn't communication or ownership — it was giving one agent ownership of the final integration: hand off sequentially, or appoint an integrator/reviewer to own the merge. Both recover solo-level performance at both tiers, and sequencing does best of all.
The structure has to match the model. Sequencing — which shrinks the task rather than adding rules to obey — is the one intervention that lifts the weak model above its own solo score; concurrent collaboration otherwise stays dead for it no matter what you fix (fair merging, fences, handshakes all leave it flat). Enforced ownership, a rule the agent has to actively exploit, only paid off for the strong model. Below some capability threshold, handing an agent a coordination protocol is like handing a process manual to someone who can't yet do the underlying job — but handing it a smaller job still helps.

So "they can't be teammates yet" reads, from here, less like they lack social intelligence and more like they were dropped into an unstructured free-for-all no sane engineering team would run — and the fix that generalizes is process (decompose + sequence, or appoint an integrator), not exhortations to communicate.

Are humans actually better at this?

It's tempting to read the paper's framing as "agents lack the social intelligence humans have." But look at the three failure modes I found and ask honestly whether we are immune to them:

Concurrent edits collide at the merge. Two humans editing the same file in parallel produce merge conflicts too — constantly. We didn't solve that by getting more socially intelligent; we solved it with process: small PRs, code ownership, trunk-based development, "rebase before you push."
Follow-through fails even when the message arrives. The sharpest failure here was an agent that read a teammate's request, wrote "I'll need to coordinate with them," and then simply didn't. If you've ever watched a Slack request get a 👍 and then nothing, you know humans do this too. Our fix isn't willpower — it's tickets, assignees, standups, and "who owns this?" made explicit.
What rescued the agents is what we already do. One agent owning the final integration — sequential handoff, or a designated integrator/reviewer — is just trunk-based development and the release-manager role. We didn't invent those because we're bad at talking; we invented them because unstructured concurrency doesn't scale for anyone.

So the fair comparison isn't "humans vs agents at coordination." It's "humans with decades of accumulated process vs agents dropped into a process vacuum." Put a human team in the agents' exact setup — no ticketing, no ownership, no review, just a chat channel and "go" — and it would step on itself too. The honest difference that remains: when process is missing, a good human improvises it (pings the right person, swivels their chair, calls a five-minute huddle); the agents, handed the very same channel, mostly didn't. That gap — improvising coordination when none was imposed — is real, and it may be exactly what "social intelligence" is pointing at. But it's a smaller, more specific gap than "they can't be teammates," and the practical lesson cuts the same way for both species: don't rely on anyone, silicon or carbon, to invent process in the moment — give them the process.

Caveats and what's next

19 pairs, 5 repos. The headline conditions and C2 have three seeds (hence the ranges); C1 and C2b are single runs at 0%/near-0%, so those two are the softest numbers here.
Passes concentrate in a few repos. Every pass, in every condition and tier, lands in two Python repos (plus one Pillow task); four of the nine tasks were solved by nothing, ever — including solo. The effectively discriminative set is closer to ~10 pairs than 19.
The subset skews toward what I could run natively; broadening it is the obvious next step.
The C2b number survived a v2 with per-hunk reverts (0% at both tiers on a re-run) — it is not a revert-granularity artifact.
A caution from building this: two conditions initially scored a false 0% due to eval-composition bugs (a stacked patch evaluated against the wrong base). I then ran two independent adversarial code audits over all five condition implementations and the eval routing; the published numbers survived, and the audit is what surfaced the missing conflict resolver behind the fair-merge row above. In agent harnesses, the scoring plumbing deserves as much testing as the conditions themselves.
One thing I couldn't test here: restart-vs-iterate. There's a related question — when an attempt is going badly, does a capable agent throw it away and start fresh, or keep patching a sinking ship? I mined all 1,806 trajectories and found zero voluntary restarts — but that's an artifact of this setup, not a finding: CooperBench agents make a single pass and never see the test results, so they're never shown the failure signal that would prompt a restart in the first place. The closest thing I can see is adaptation after an imposed restart (when territory enforcement reverts an edit) — a cousin of the real question. Measuring restart-vs-iterate properly needs a different harness — visible tests, several iterations, an explicit restart affordance — and that's the next experiment.

All five conditions, the combos, the LLM conflict resolver, and the eval fixes are implemented on top of the open CooperBench harness — the full fork is public at github.com/JaviMaligno/CooperBench (branch experiment/structural-conditions), so you can reproduce or poke holes in any of this.

Responding to CooperBench (Khatua, Zhu, et al., 2026). The headline cells are three-seed means on a 19-task subset; broadening the task set is the next step.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Bootstrap the Environment, Not the Agent

JaviMaligno — Wed, 08 Jul 2026 15:52:27 +0000

I recently found myself repeating the same loop with Claude Code in a remote session:

Ask it to make a change.
Watch it try to run the build or tests.
Watch the environment fail because a system dependency was missing.
Tell it: "use apt-get and install this."
Repeat in the next clean session.

The problem was not Claude. It was not LaTeX, Python, or the build itself either. The problem was that part of the project's operational knowledge lived in a conversation, not in the repository.

The concrete case was Code World Models. The project has Python tests, but it also builds a LaTeX paper. On my laptop, everything was installed. In CI, everything was installed too, because GitHub Actions installed TeX Live before compiling the paper. But a remote Claude Code session starts from a fresh environment and has no reason to know that pdflatex is part of the real repository contract.

The fix was small: a session-start hook that prepares the environment before the agent needs it.

The broader principle is more useful: bootstrap the environment, not the agent.

The Antipattern: Dependencies in the Prompt

When we work locally, we accumulate state without noticing. We have system packages, CLIs, credentials, caches, extensions, fonts, browsers, native binaries, and toolchains that do not appear in package.json, pyproject.toml, or requirements.txt.

That state becomes invisible because the project "works on my machine."

Cloud environments break that assumption. AI agents break it faster, because the agent tries to close the full loop:

change the code;
run tests;
compile docs;
open browsers;
generate screenshots;
publish artifacts;
verify the final result.

If a binary is missing, the agent often does the reasonable thing: diagnose the failure, find an install command, try again. But that burns time, tokens, and attention. Worse, if the fix only lives in chat, the next environment does not improve.

Telling an agent "install TeX Live before compiling" is manual operational memory. Putting it in the repo is infrastructure.

The Real Case: LaTeX and Python in a Remote Environment

In Code World Models, the paper lives in docs/paper/main.tex and is checked with:

bash scripts/check_latex.sh

In CI, the dependency was declared in .github/workflows/ci.yml:

- name: Install TeX Live
  run: |
    sudo apt-get update
    sudo apt-get install -y --no-install-recommends \
      texlive-latex-base texlive-latex-recommended texlive-latex-extra \
      texlive-fonts-recommended texlive-science texlive-bibtex-extra biber

That protected GitHub Actions, but not necessarily a remote work session.

So I added a Claude Code SessionStart hook:

{
  "hooks": {
    "SessionStart": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/session-start.sh"
          }
        ]
      }
    ]
  }
}

The script does three important things:

if [ "${CLAUDE_CODE_REMOTE:-}" != "true" ]; then
  exit 0
fi

echo '{"async": true, "asyncTimeout": 300000}'

if ! command -v pdflatex >/dev/null 2>&1; then
  export DEBIAN_FRONTEND=noninteractive
  apt-get update
  apt-get install -y --no-install-recommends \
    texlive-latex-base \
    texlive-latex-recommended \
    texlive-latex-extra \
    texlive-fonts-recommended
fi

python -m pip install -e ".[dev]"

First, it is a no-op locally. My laptop already has what it needs, and I do not want an agent hook mutating my machine unnecessarily.

Second, it runs asynchronously. The first version of this hook was synchronous, but in this repo the paper build usually happens at the end, after the code and text edits are done. I do not need to block session startup while TeX Live installs; I need it ready by the time compilation happens.

Third, it is idempotent. If pdflatex already exists, it does not reinstall the toolchain.

Fourth, it uses python -m pip, not plain pip, so the install targets the same interpreter that later runs python -m pytest. That small detail avoids a tedious class of bugs where pip and python point at different environments.

The result is not dramatic. That is why it is good. The next time the agent opens the repo remotely, the environment prepares itself and the agent can spend its reasoning on the actual change.

This Is Not Just About Agents

The same principle applies to any ephemeral environment:

GitHub Codespaces;
Dev Containers;
Gitpod;
CI runners;
Vercel-like sandboxes;
temporary demo machines;
internal onboarding environments;
remote agent sessions.

The difference with AI is that the cost of not doing it becomes more visible. A human might remember, "this repo always needs libpq-dev." An agent can infer it, but it should not have to rediscover it in every session.

Cloud workspaces make environments disposable. Agents make the build-test-fix loop much more frequent. Together, they make reproducible setup part of the repository contract rather than a nice-to-have.

What Belongs in the Bootstrap

Not everything belongs in a startup script. If you put too much there, the environment becomes slow, opaque, and fragile. But there is a clear category of things that do fit:

system packages required to compile or test;
CLIs used by repository scripts;
browsers or native dependencies for end-to-end tests;
documentation toolchains such as LaTeX, Pandoc, or Graphviz;
editable install of the local package;
quick version checks;
cache or temporary-directory setup.

The practical rule I use:

If the agent, CI, or a new developer has to ask for it manually to run the basic workflow, it should be declared in the repo.

It does not have to be a Claude hook. It could be a devcontainer.json, a Dockerfile, scripts/bootstrap.sh, a Makefile, a Nix definition, a reusable workflow, or a combination. The tool matters less than the ownership: the repository should know how to prepare its own environment.

A Good Bootstrap Has Six Properties

1. It Is Idempotent

It should be safe to run many times. Check before installing:

if ! command -v pdflatex >/dev/null 2>&1; then
  apt-get install ...
fi

Agents retry. Humans retry. Design for that.

2. It Separates Local from Remote

Not every environment needs the same treatment. My hook exits immediately unless it is in a remote session:

if [ "${CLAUDE_CODE_REMOTE:-}" != "true" ]; then
  exit 0
fi

That condition keeps the bootstrap from becoming a source of local surprises.

3. It Reuses the CI Contract

If CI installs TeX Live, the remote environment should install TeX Live. If CI runs pip install -e ".[dev]", the agent should prepare the same package. Consistency matters more than elegance.

In the best case, you factor the shared setup into one script. In some repos, the CI workflow and the remote hook duplicate a few lines. That is acceptable as long as they do not drift semantically.

4. It Chooses Sync vs Async Deliberately

Some setup can run in the background, but not all of it. In my case, the hook can start asynchronously because the PDF is built at the end of the workflow, long after the session starts. That would not be acceptable for a dependency needed by tests in the first few seconds.

The rule is simple: if the likely next command depends on the setup, keep it synchronous and fail fast. If the setup prepares a tool that will be used later, it can run asynchronously and save startup latency.

The bootstrap should not hide errors that make the environment invalid.

5. It Fails Visibly When It Blocks the Workflow

Async does not mean silent. If a dependency is mandatory, the workflow that uses it still needs a clear failure when setup did not finish or could not complete. In this case, scripts/check_latex.sh remains the real guard: when compilation time comes, it validates that the toolchain exists.

6. It Is Conservative with Privilege and Network Access

A startup hook is not a good place for arbitrary downloads. Avoid curl | bash unless you have a strong, controlled reason. Use system packages where they are enough. Pin versions when reproducibility matters. Separate privileged OS setup from user-level project setup.

The Mental Shift

Many repositories still treat setup as documentation:

Install these things before running the project.

That was reasonable when the main reader was a human sitting at their primary machine.

Now the reader might be an agent, an ephemeral runner, a temporary workspace, or a collaborator joining from another machine. For those readers, documentation is not enough. They need an executable contract.

The question is not "can the agent figure out how to install this?"

It probably can.

The better question is: why should it figure it out every time?

Every dependency that lives only in a conversation is friction that will repeat. Every setup step the repo can execute by itself is one less piece of context you have to load into your head, or into the agent's.

For teams working with AI, this becomes a form of DevEx: not just human developer experience, but agent experience. The best environment for an agent is not the one with the longest prompt instructions. It is the one that needs the fewest explanations before it can reach an honest build.

Bootstrap the environment. Let the agent work on the problem.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

The Forgetting You Don't Measure

JaviMaligno — Mon, 06 Jul 2026 08:37:02 +0000

There's a recent paper from the Qwen team — Qwen-AgentWorld — that turns an LLM into a world model: instead of answering as an assistant, the model learns to predict what an environment returns after an action. Run a shell command, and it predicts the exact terminal output, exit code and all. They build this by taking a finished, general-purpose model and training it further on millions of interaction trajectories — a step called continual pre-training (you keep pre-training an already-trained model on new data to teach it something new).

The catch with that kind of training is well known: pour enough narrow data on a model and it starts to forget things it used to know — the classic "catastrophic forgetting." The standard defense is replay (also called data mixing): while you train on the new narrow data, you keep mixing in a fraction of general text, so the model rehearses its old knowledge instead of letting it wash away. The Qwen paper does exactly this — during continual pre-training they blend in general-knowledge corpora (law, medicine, current events). They frame it as a capability move (you can't simulate a hospital without medical knowledge), but it doubles as anti-forgetting insurance. It's a sensible choice. What they never do is measure it: there's no before/after on general benchmarks, no number on how much gets forgotten or how much the replay saves.

So I ran the experiment at toy scale. Not to one-up a 397B model with a 0.5B one — that would be silly — but because the question is about a training dynamic, and dynamics show up at small scale too. Honestly, I mostly wanted to see the thing this blog keeps circling back to: not whether a system degrades, but whether you can tell when it does.

The setup

I took Qwen2.5-0.5B-Instruct, generated a narrow "terminal simulator" dataset (real shell commands and their real outputs), and continually pre-trained the model to predict those outputs — teaching it to be a tiny world model. Before and after training I measured general ability on standard benchmarks (ARC, HellaSwag, WinoGrande), and I measured whether it actually learned the new task (held-out next-output accuracy). Then I swept the one knob that matters: what fraction of each training batch is replay — general text (Wikipedia) — instead of terminal data. Zero, 10%, 25%, 50%, and a 100% control. Three seeds each. The whole thing ran on a single spot T4 on Azure for about the price of a couple of coffees.

Result one: yes, it forgets, and a little mixing buys it back

With no replay, the model learns the terminal task well (exact-match on held-out outputs jumps from ~0.36 to ~0.90) — and it forgets. General accuracy drops across the board: on average −0.078, and −0.15 on the worst-hit benchmark (ARC-Easy). Then I dialed replay up. The amber line below is the story:

Ten percent replay recovers about 73% of the average forgetting — and roughly 82% of the worst case — at no cost to the new task (task accuracy stays at ~0.90). Past 25% you get diminishing returns. And a 100%-replay control (not plotted) confirms the bracket from the other side: train only on general text and the model never learns to simulate anything (task accuracy collapses back to baseline), while general ability stays flat.

I want to be upfront: this is not a new discovery. That a small replay fraction suppresses catastrophic forgetting is well-established — the continual-learning literature has been saying "mix in 1–5% of the old distribution" for years. What I did was put a number on the specific claim the world-model paper makes by design but leaves unquantified. That was the easy part. The blue line — LoRA — is where it got interesting.

The more interesting part: how much you forget depends on how you specialize

Replay is one lever. The training method is a bigger one. Everything above used full fine-tuning — every weight in the model is free to change. So I re-ran the whole sweep with LoRA instead: the cheap, popular parameter-efficient method that freezes the original model and trains only a small set of add-on "adapter" weights. That's the blue line in the chart above — flat, near zero, at every replay fraction, including zero.

LoRA barely forgets, even with no replay at all, while learning the terminal task just as well (same ~0.90 accuracy). This isn't mysterious: because the original weights are frozen, the general knowledge they hold sits untouched by construction. LoRA is known to forget less precisely because it changes so little. Full fine-tuning, by contrast, lets the optimizer overwrite any weight — including the ones holding general ability.

So I pushed on what full fine-tuning actually costs, beyond the reasoning benchmarks. Two more probes, at zero replay:

Probe (at 0% replay)	Full fine-tuning	LoRA
Instruction-following (IFEval)	−0.071	+0.033
World model on unseen commands	0.00	0.15

Instruction-following. My reasoning benchmarks couldn't see this, so I added IFEval — instructions a script can grade automatically (obey a format, a length, include a given word). Full fine-tuning drops it 37% relative (0.194 → 0.123); LoRA actually nudges it up. This is the concrete thing that "general benchmarks barely moved" was hiding on the full-FT side: an instruction-tuned model quietly losing the instruction-following it was tuned for.
Depth of the learned task. Here I expected the opposite — that LoRA, changing so little, would learn a shallower world model. It was the reverse. Both score ~0.90 predicting outputs for the kinds of commands they trained on, but on genuinely unfamiliar commands (things like sed, awk, grep and pipes, which the training set never contained) full fine-tuning collapses to zero while LoRA still gets 0.15. Full fine-tuning memorized the exact command shapes it saw; LoRA, riding on the frozen base, kept some ability to generalize.

So the honest read is simpler than a "gotcha": full fine-tuning pays three costs — general reasoning, instruction-following, and out-of-distribution robustness — that a single benchmark undersells, and LoRA mostly doesn't pay them, because it perturbs the model far less. That's not LoRA hiding anything; it's LoRA doing less damage. The mild version of the lesson still holds, though, and it's the through-line of Results-Oriented Programming and How Much Should You Still Know?: one number ("my general accuracy held") can quietly stand in for several you didn't check.

The trade-off is the usual one: LoRA learns the new task a bit less aggressively than full-FT. Here the terminal task was easy enough that both hit the same ~0.90, so LoRA looks strictly better — but on a harder target where you need every point of task performance, full fine-tuning's willingness to overwrite is exactly what you're paying for. There's no free lunch, just a dial between "learn more, forget more" and "learn less, forget less."

Two nuances, honestly reported

Base versus instruct. I expected the instruction-tuned model to forget more — more to lose. On reasoning it didn't: base and instruct 0.5B forgot almost identically (−0.14 vs −0.17 mean at zero replay). Where the instruct model does bleed is instruction-following itself (the IFEval drop above) — a base model has little of that to lose in the first place. So "more to lose" is true, but only on the axis the instruct model was tuned for.

Size. The larger model forgot a bit less: full fine-tuning 1.5B lost −0.060 on average versus 0.5B's −0.078, leaning toward "smaller models forget more." But the effect is soft and not uniform — clear on ARC-Easy, reversed on ARC-Challenge — and comparable to the noise between seeds. (Getting 1.5B full fine-tuning to fit on a 16 GB T4 at all took an offloaded 8-bit optimizer and a batch size of one; it overshot memory by 200 MB on the first try. A craft note, not a finding.)

The takeaway

None of this is new physics — replay mitigating forgetting, LoRA forgetting less, and the rest are all things the literature already knows. This was a small reproduction you can run overnight for the price of a few coffees. What it made concrete for me is how much the answer to "did it keep its general ability?" depends on what you measured and how you trained.

Full fine-tuning on a narrow task looked fine on reasoning, until IFEval showed the instruction-following it had shed and an unfamiliar-command probe showed how narrowly it had actually learned. Same model, same data — the losses were real, they just weren't in the first place you'd look. The fix isn't exotic: when you specialize a model, check more than one capability, pick the training method deliberately (LoRA and full fine-tuning are genuinely different trades, not interchangeable defaults), and don't read one flat benchmark as proof that nothing was lost.

Code and full results: github.com/JaviMaligno/language-world-model-forgetting. This is the third in a loose series on delegating to models and staying able to tell when they're wrong — see also How Much Should You Still Know? and Results-Oriented Programming.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

The Only Software You Need Is a Subscription

JaviMaligno — Tue, 30 Jun 2026 15:41:45 +0000

There's a small piece of software I use every day that doesn't exist as a product. It watches my calendars — personal, work, the consulting one — and copies the busy blocks across all of them, so that anyone looking at any single calendar sees when I'm actually blocked, even when the block lives somewhere else. Paid tools do exactly this. I looked at them, then spent an evening with Claude Code writing a script and a routine that runs on a schedule. It's been running quietly for months. I barely remember it's there.

That last sentence is the whole article, but let me get there slowly.

Because the tempting conclusion right now — the one the moment keeps pushing you toward — is the maximalist one: with agents like Claude Code and Codex, building almost anything is suddenly within reach, so why pay for anything? Taken to its limit, the only software you need is a subscription to Claude or ChatGPT, and everything else you just build. There's enough truth in that to be dangerous, which is exactly why it's worth slowing down on.

What getting cheap actually reveals

It's easy to read the agent moment as "building got cheap" and stop there. True, but not the interesting part. The interesting part is what cheap building exposes — because buy vs build was never really a contest about the cost of building.

When you paid for software, the price tag was mostly not for the code. The code was the least of it. What you were buying was three things the code quietly carried:

Someone else absorbs the maintenance. The thing keeps working when an API changes, when an edge case shows up, when the platform shifts under it. That liability is the provider's, not yours — forever, or until you cancel.
Someone else already paid the trial-and-error tax. A built product has been through the loop of "this looked right and wasn't" hundreds of times before it reached you. You skip that. You buy the far side of a learning curve.
It doesn't cost you your attention. This is the one that matters most and gets named least.

Agents collapsed the first cost everyone talks about — the writing — and in doing so they pulled the curtain back on the other three. The bill you were paying was never really for the software. It was for not having to care about it.

The resource that doesn't parallelize

Of those three, attention is the one I keep coming back to, because it behaves differently from the rest.

Time, agents genuinely give back. You can fan five of them out at once and compress a week of work into an afternoon — I've written before about what that does and doesn't buy you. But attention isn't time. You can run five agents in parallel; you cannot attend to five things in parallel. Every tool you build is a small standing claim on your attention — a thing that can break, drift, surprise you, and quietly ask to be looked at. Build enough of them and you haven't bought yourself freedom; you've appointed yourself the on-call engineer for a sprawl of one-person software nobody else can page for.

So the real build-vs-buy question, now that code is nearly free, isn't "can I build this?" The answer is almost always yes. It's: who ends up holding the maintenance, and who ends up holding my attention?

When I build anyway

Framed that way, two situations make building the obvious call.

When it's trivial and then goes silent. The calendar sync is the clean case. I built it once, it runs on a schedule, and it makes no further claim on me. The attention cost was a single evening and then approximately zero. This is the sweet spot the maximalists are actually right about: small, well-scoped, set-and-forget. When something can be built quickly and then left to run without watching, building wins easily — you skip the subscription and you pay almost nothing in the currency that's actually scarce.

When nothing on the market fits well enough. VitaminD Explorer — my solar vitamin-D calculator — started as a Claude artifact and grew into a real PWA precisely because what I wanted didn't exist in a form I'd have paid for. When the market's answer is "close, but not the thing," the build/buy math stops being about cost at all. You're not undercutting a product; you're filling a gap a product never filled.

Notice that neither of these is "build because building is free." Free is the precondition that puts building on the table. It isn't the reason.

The trap, and the limits

The flip side is where the maximalist position quietly breaks.

The trap is everything that doesn't go silent. A tool that needs babysitting, that breaks in ways you can't predict, that demands a decision every time something upstream moves — that's not a saving, it's a second job you assigned yourself. Here the provider's offer is honest and worth taking: pay us and the maintenance, the drift, the 3 a.m. failure mode become our problem. For anything that would otherwise sit on your attention indefinitely, that's often the better deal even when you could build it in a weekend. Could and should part company exactly here.

And then there are the plain limits that the "just a subscription" framing waves away. Usage ceilings are real. Some things genuinely shouldn't run unattended, no matter how capable the agent — anything with money, identity, or irreversible consequences on the other end wants a human in the loop, which means it never fully leaves your attention. The reductio — the only software you need is a subscription — is a useful provocation and a bad operating plan. The subscription is the workshop, not the warehouse.

Same logic, at company scale

This isn't only a personal calculus, and the enterprise version is the part that's been talked to death — "the death of SaaS," every company becoming a software company, the great unbundling. I don't have much to add to that discourse except to point out that the axis it usually argues over is the wrong one. The debate fixates on cost and capability: now that we can build internal tools cheaply, should we stop buying SaaS? But the deciding variable is the same one as in the personal case — attention, just spread across a team instead of a person.

I've built internal monitoring systems that run on their own, watching logs and doing automated performance and code-quality reviews. Tools like that absolutely exist on the market. The reason to build wasn't a gap — it was a thin layer of customization the off-the-shelf option couldn't match cheaply, on top of something that, once built, mostly runs itself. Low standing attention cost, modest effort, a fit you couldn't buy: the personal sweet spot, scaled up.

The same trap scales up too. A company can build something that it then has to staff forever, pulling a team off the actual line of business to babysit internal software — which is just the on-call sprawl problem with a payroll attached. The question a company should ask isn't "can we build this instead of buying it?" It's the same two questions: who carries the maintenance, and whose attention does it spend? A dedicated internal-tools team is sometimes exactly the right answer to that — and sometimes it's the most expensive way a company ever learned it should have just bought the thing.

Where I land

I build by default now, more than I used to, and I'm not embarrassed about it. Code got cheap; that genuinely changed the math. But the version of "build everything" worth holding is narrower than the slogan: build the things that are trivial enough to leave running, or that the market never built for you — and especially the ones that, once built, go quiet and stop asking for you.

For everything else, the most useful thing a paid tool sells isn't the software. It's the right to stop paying attention. That's a real product, and some days it's the only one worth buying.

A companion to Human Limits in Managing AI Agents and Results-Oriented Programming. VitaminD Explorer lives at getvitamind.app.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Results-Oriented Programming

JaviMaligno — Wed, 24 Jun 2026 16:47:14 +0000

There's a line I dropped almost in passing in a recent piece, and it's been nagging at me ever since: programming keeps drifting from verifying how the code works to verifying what it produces. I want to pull on that thread properly, because it's one of the quieter but more consequential shifts in how this job actually works now.

To be clear about what's drifting: it isn't that we never checked results before. Tests have always existed, and writing them has always been good practice — even in the long stretches where the industry didn't write nearly as many as it should have. The shift is one of center of gravity. For most of my career the weight of "reviewing" sat on the implementation: you read the diff, followed the logic, decided whether the path the code took was sound, and the tests rode alongside as a safety net. The path was the primary artifact you scrutinized.

That balance is inverting. More and more the result-check is the main event, and I let the path be the model's problem as long as the result holds. Call it results-oriented programming, for lack of a better name. It isn't a methodology I went looking for; it's what the work quietly turned into.

The interesting part isn't the slogan. It's what "verify the result" turns out to mean, because it means wildly different things depending on what the result is.

The easy end: when the result is a number

The cleanest version is when the output pins to a value. My pricing system quotes a number: either it's the right one for that material, origin, and client, or it isn't. I don't read the code that computed it — I keep a golden set the client signed off on and tests that assert the output matches. The oracle that decides "correct or not" is exact and auditable, no one's judgment in the loop at the moment of truth; Claude can refactor the engine three times over a weekend and I won't care, as long as the set stays green. When you can get here, get here — it's the strongest form of the idea.

The hard end: when the result is a behavior

But most of what we build doesn't reduce to a number.

A rules engine that has to reach the right decision across a thousand interacting conditions. A classifier sorting a business into a regulatory category. A scheduling routine whose output is "reasonable" or "not" in ways no single assertion captures. An agent choosing, mid-task, whether to ask for clarification or just proceed. A summary that has to be faithful without being a transcript. Some of these are pure deterministic code, some are model-driven, but they share the property that matters here: "the result" is a behavior or a decision, and there's no golden number to diff against. Two perfectly good outputs can look nothing alike.

The temptation is to retreat to the old habit: if I can't pin the output, I'll go back to reading how it got there. And often you can read it. A rules engine's branches are right there; even a model leaves a trail — its reasoning, its tool calls, the intermediate steps. The path isn't the sealed black box it's convenient to call it, and inspecting it is genuinely useful for debugging and for calibrating how much to trust the thing. But it tells you how it arrived, not that it arrived somewhere correct. A clean-looking trace can land on a wrong answer and a tangled one on a right answer — which was always true of code, and is precisely why we wrote tests instead of just re-reading the function. At this end of the spectrum, reading the path informs you; it doesn't settle correctness. That still has to happen at the result.

So results-orientation doesn't break here, it just changes oracle. Instead of an exact match you build a richer check, and it climbs a ladder of subjectivity: behavioral specs and property tests and a scenario suite for the rules engine; an eval set of representative cases with the calls you'd defend; and for the genuinely judgment-bound outputs, sometimes an LLM scoring against criteria you wrote. I've argued before about when an LLM-as-judge is even trustworthy enough to lean on; this is exactly where it matters, because at the far end your judge is your verification. The result is still the thing under test — you've just traded a deterministic comparator for a defined-but-subjective one, and the engineering work moves into making that check as principled and repeatable as you can.

The spectrum is the whole point. From "assert equals 4,200.00" to "a panel of criteria says this triage decision was reasonable," it's the same move at different resolutions: specify the result, build something that checks it, stop policing the path.

Why this is happening

None of this is a philosophy I adopted. It's a response to the path getting cheap.

When a model writes the implementation, two things flip at once. The path becomes nearly free to produce — and expensive, sometimes pointless, to read. A 600-line diff that Claude generated in a minute can take me an hour to review line by line, and at the end of that hour I often know less about whether it's correct than a five-second glance at the test suite would tell me. The economics of attention have inverted: reading the how no longer scales with the rate at which we produce it.

Meanwhile the result is the only thing the user or client ever actually touches. They don't experience your control flow; they experience the price, the decision, the summary. Verification migrates to where the value and the risk actually live — the boundary between input and output — because that's the part that didn't get cheaper. Defining what "correct" means is still hard, human work. The model just took over the part in the middle.

The nuance that ruins the slogan

Here's where results-oriented programming gets dangerously easy to misread. If all that matters is the output, does the code in the middle matter at all? Can it be a sludge of duplication and clever hacks, as long as the tests pass?

No. And the reason is sharper than "good engineers care about craft."

Internal quality — modularity, clear boundaries, security, conventions — still matters, arguably more than before. But the reason why it matters has shifted, and so has the audience. We used to keep code clean for the humans who'd read it next. Increasingly, the next reader is an agent. Maintainability is drifting from maintainability-for-humans to maintainability-for-agents. A model produces better results, more reliably, in a codebase with clean seams, honest names, and tight modules — for roughly the same reasons a human does: less context to hold, fewer ways to go wrong, clearer places to make a change. Messy internals don't just offend taste; they degrade the very results you're optimizing for, just on a delay. The bad price you ship next quarter is often paid for by the tangle you let the model leave behind in this one.

So results-orientation isn't "internal quality doesn't matter." It's "internal quality matters because of its effect on results, not as an end you inspect for its own sake." That's a shift of priority and framing, not a license to abandon it. You stop reviewing the diff for elegance. You don't stop caring whether the system stays workable — you just hold that bar through different means: linters, architecture tests, conventions encoded where the agent will actually see them.

Where this is heading

Follow the line far enough and the job starts to look different.

If verification lives at the boundary, and internal quality is encoded as rules the agent has to respect, then the thing you're really authoring is no longer the implementation at all. It's an objective plus a set of constraints: the result you want, and the rules the system must stay inside while reaching it — the stack, the business rules, the security requirements, the conventions, the working definition of "good." Then you hand that to a model inside a good harness and let it optimize toward the objective without stepping outside the constraints.

That's the logical endpoint of results-oriented programming: you specify the target and the guardrails, and the model searches the path. Your leverage stops being the code you write and becomes the quality of the objective you define and the harness you put around it — the tests, the evals, the judges, the rules, the checkpoints. Get those wrong and a capable model optimizes confidently toward the wrong thing. Get them right and you've built something that improves the result without you touching the middle.

Which is why I don't think "results-oriented" means "lazy." It means the hard part moved. It used to live in the writing. Now it lives in defining — precisely enough that a machine can be held to it — what you actually wanted, and in building the apparatus that holds it there. The path got cheap. Saying what "right" means got more important than ever.

A companion to How Much Should You Still Know?, which ran into this same idea from the angle of delegation and responsibility.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

How Much Should You Still Know?

JaviMaligno — Tue, 23 Jun 2026 22:20:02 +0000

I'm building a pricing system for a client right now, and I have to admit something: I couldn't tell you most of how it prices things.

Not because the code is opaque — I wrote it, or rather Claude and I wrote it. Because the business logic is enormous. Prices depend on the client, the material, the origin, and a long tail of extras and exceptions that each have their own rules. There are more of these than I can hold in my head, and they change. Every time I hit a gap — a price whose logic I can't find, a case the documentation doesn't cover — Claude Code and I log it in a "pending data" document. That document gets reviewed and sent to the client.

Here's the part that should bother me more than it does. A few times the client has come back asking me to clarify how something is priced, and to answer I've had to ask Claude first. I'm the external consultant. I don't carry the business knowledge, and honestly I don't think I should try to. Claude has access to all the documentation and is far more reliable at retrieving the right rule than I would be at internalizing the whole thing.

The instinct is to feel guilty about that. I've decided the guilt is mostly aimed at the wrong question.

Offloading knowledge isn't new, and it isn't automatically debt

We've been outsourcing memory for a long time. Most of us stopped memorizing phone numbers two decades ago. We don't memorize facts we can look up; we remember where to look. Psychologists have a name for it — the "Google effect" — and the sky did not fall. Knowing how to find something is itself a kind of knowledge, and it's often the more useful kind.

An LLM is the next rung on that ladder. It doesn't just tell you where the answer lives; it retrieves it, applies it to your specific case, and explains it. For something like a sprawling, client-specific pricing rulebook, that's not a degradation of my work. It's the only sane way to do it. Memorizing thousands of volatile rules that belong to someone else's business would be a waste of the one thing I'm actually paid for.

There's a term going around — "cognitive debt" — borrowed from technical debt, and a much-shared MIT study ("Your Brain on ChatGPT") that measured reduced neural engagement in people who wrote essays with an LLM. It's a useful phrase. But I think it gets applied too broadly. Not all offloading is debt. Buying groceries instead of farming isn't "agricultural debt." The question is never whether you delegate knowledge. It's how much, when, and — the part everyone skips — what you remain on the hook for.

What actually changes: responsibility

A student who delegates their learning to an LLM is only hurting themselves. If they don't understand the essay, the cost lands on them, later, privately. That's a clean trade they're free to make.

I don't have that exit. I'm responsible for a pricing system that will quote real numbers to real customers. If it prices something wrong, "Claude told me" is not an answer I can give the client. The knowledge is delegable. The responsibility is not.

That reframes the whole thing. The question I kept asking myself — how much of this should I know? — is the wrong one. The right question is: do I know enough to answer for it? And those are very different bars.

Three questions, and where the line actually falls

So when is it enough to just be able to ask in the moment? What should you know without asking? And how deep does that understanding need to go? Let me answer with the pricing system, because the line falls in a specific place.

The individual data points — ask away. What's the surcharge for this material from that origin? I will never hold that in my head, and I shouldn't. It's volatile, it's the client's domain, and the model retrieves it more reliably than I would. Being able to ask in the moment is genuinely sufficient here. This is the layer where delegation is not just acceptable but correct.

The shape of the system — know it without asking. What categories of rules exist, how they compose, which inputs can move a price and which can't, where the dangerous interactions are. I can't recite every surcharge, but I have to know that surcharges-by-origin are a thing that exists, or I won't even know to ask. You can't query a space whose existence you're unaware of. This is the map, and the map is mine to hold.

Deep enough to catch a wrong answer — and honest about what that means. This is the real threshold, and it's not about how much you know but about what kind. Here's where I have to be honest, though: I can't validate every output by hand, and pretending I could would just be a different flavor of dishonesty. I'm not going to compute a price manually, and often I don't even know what the correct price is — that's the client's business, not mine. What I can catch directly is the gross stuff: a number that's an order of magnitude off, a rule applied to a case it should never touch, two extras stacking that shouldn't. The basic-logic violations.

The exact application — the part I can't eyeball — doesn't get validated by my judgment at all. It gets validated by tests, and by a golden set the client signs off on. That's the move that actually scales: I don't hold the right answer in my head, I hold a mechanism that checks it. Validation itself gets delegated — but to something deterministic and auditable, not to vibes. It's part of why programming keeps drifting toward being results-oriented: increasingly you specify and verify the behavior, not the path the model took to produce it.

So the line is a little wider than "can I personally tell it's wrong." It's: do I have some way to catch a wrong answer — my own eye for the crude errors, a test or a golden set for the exact ones? If I have none of those, I'm not delegating — I'm abdicating. Delegation keeps a human, or a check the human owns, in the loop. Abdication removes both. They look identical right up until the answer is wrong, and then they could not be more different.

The quiet symptom: documentation only the agents read

Here's where I notice the line getting blurry in my own work. More and more of the knowledge about my systems lives in files written for machines. CLAUDE.md. AGENTS.md. Skills. And in this project, that "pending data" document — half spec, half question-list, written so the agent and I can navigate a domain neither the client nor I have fully written down anywhere else.

I wrote about this drift from another angle in Software Is Dissolving Into the Model: the layer we used to call software is collapsing into instructions on one side and generated output on the other. The flip side of that convenience is this — when documentation is written for the agent, who's keeping a map for the humans who remain responsible? It's easy to end up with a system that's beautifully legible to Claude and slowly going dark to everyone accountable for it.

The interesting thing is that my "pending data" document accidentally does the right thing. It's not just context for the agent; it's a human-in-the-loop checkpoint. It forces a moment where a person — me, then the client — has to look at what's missing and decide. That's the pattern worth keeping. The answer to "docs only agents read" probably isn't writing fewer of them. It's deliberately designing the points where a responsible human has to stop and validate, instead of letting the loop close without one.

The part nobody wants to admit out loud

There's a social layer to all of this that the technical framing skips. Telling you I sometimes ask Claude before answering my own client costs me something. It brushes right up against impostor syndrome — the quiet fear that the moment people see how the sausage is made, they'll decide I'm not the expert they hired.

The strange thing is that transparency is the cure for that feeling, not the trigger. Impostor syndrome grows in the gap between the image you project and how you actually work. Saying out loud "I asked the model and then verified the rule applies" closes that gap. There's nothing left to be found out.

But I won't pretend the cost is imaginary. With some clients you genuinely will come off worse for being open about it — not because you did anything wrong, but because they don't yet understand what good AI use looks like. They hear "I asked Claude" and pattern-match it to "didn't know," when it should pattern-match to "retrieved the authoritative rule and checked it." The irony is sharp: those are often exactly the clients I'm hired to help integrate AI into their own processes. The same gap in understanding that makes transparency risky is the reason the work exists.

That's why the validation line matters here too, not just technically. Transparency isn't confessing "I don't know anything, Claude does it." It's being precise about what you delegate and, crucially, that you kept the ability to check it. "I asked Claude and confirmed the rule applies to this case" is a completely different sentence from "I asked Claude." One describes a professional using a tool well. The other describes the thing the client is afraid of. Being able to validate is what makes the honesty defensible — and over time, modeling that distinction is part of how clients learn to tell the two apart.

Where I've landed

I'm not going to tell you to delegate less. In the pricing project, delegating the business knowledge is the right call, and pretending otherwise would just make me slower and no more trustworthy. The line I'm drawing isn't between "implementation you can offload" and "knowledge you must keep." It's between knowledge you can still validate and knowledge you've gone blind to.

Three things I'm actually doing about it:

Hold the map, not the territory. Don't memorize the data; do know what categories of things exist and how they interact. You can't ask a question about a space you don't know is there.
Treat "can I check this?" as the real test. Before I let the model own a decision, I ask whether I have a way to catch it being wrong — my own eye for the crude errors, a test or golden set for the exact ones. Note that "check" rarely means "compute it myself"; usually it means owning the mechanism that does. If there's no mechanism at all, that's not delegation, and I either build one or don't ship it unattended.
Design the checkpoints, not just the prompts. When knowledge moves into agent-readable docs, add the human moment back on purpose — a review step, a pending-questions doc, a sign-off — so the loop can't silently close without someone accountable in it.

The honest answer to "how much should you still know?" turns out not to be a quantity at all. It's a capability: enough to know when the answer is wrong. Everything above that line, ask freely. Below it, you're not saving effort — you're just postponing the moment you find out.

There's a more radical version of this idea, where instead of trusting the model's judgment you force it to externalize its reasoning into code you can actually read and test. That's the experiment in the next post — delegation you can audit, line by line.

This is the first of two pieces on delegation and understanding. The next one gets concrete: an experiment in making a model write down its world as verifiable code instead of keeping it in its head. Related reading: Software Is Dissolving Into the Model.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Demos That Wow vs Demos That Work

JaviMaligno — Wed, 17 Jun 2026 10:01:27 +0000

I've worked on two teams with opposite philosophies of how to demo to a client, and I still go back and forth on which one is right.

One builds demos for impact. Everything is mocked: the screens are polished, the flow is choreographed, and the result that lands on screen is the best possible version of itself. The backend isn't really doing the work — and crucially, the AI isn't really being called. It's a beautiful film of the product.

The other builds functional proof-of-concepts. As much as possible is real and wired up — real services, real model calls, real data — and then you iterate on top of it. The first version is rougher and less choreographed, but what you're looking at is actually happening.

Both "work" in the sense that both can win a room. They just optimise for different things, and the gap between them gets a lot wider the moment AI is involved.

The case for the mock

It's easy to be snobby about mocked demos, so let me steelman them first, because the reasons are good ones.

A mock is fast. You're not blocked on infrastructure, model access, data pipelines, or the ten unglamorous things that have to exist before a real flow runs end to end. You can demo a product that doesn't exist yet.

A mock is controlled. Demos fail in stupid, memorable ways — a timeout, a rate limit, a model having a bad day in front of the one person you needed to impress. A mock removes that variance. The story you rehearsed is the story they see.

And a mock sells the vision, not the current state. Early on, what you're really validating is desire: do people want this? A crisp mock answers that question without you having to build the thing first. For a non-technical stakeholder deciding whether to fund the next phase, a polished mock can be exactly the right artifact.

None of that is dishonest. It's a legitimate strategy with real upsides.

The case for the functional PoC

The functional PoC gives up some polish and a lot of control in exchange for one thing: what you show is true.

That truth compounds. A functional PoC isn't thrown away after the meeting — it's the first commit of the product. You iterate on it instead of rebuilding from a slide deck. The feedback you get is real feedback, because people are reacting to real behaviour, not to your best-case storyboard. And the hard parts surface now, while they're cheap, instead of after a contract is signed and the timeline is fixed.

I've felt this directly. In one live demo, someone asked the assistant about errors that had occurred, and its answer blended what had gone well in with what had gone wrong. The person watching reacted on the spot: this needs to be more concise. In the same session, someone asked whether it could do a particular thing that — because of an internal constraint — it simply couldn't. Two concrete pieces of feedback and two new tickets, in the span of one demo. With a mock, you can't even attempt those questions: the script answers what it was scripted to, and nothing real is being tested.

It's slower to first wow. But it never has to walk anything back.

Where AI widens the gap

For ordinary software, the distance between a mock and the real thing is mostly polish: the real version will be a bit slower, a bit less pretty, a few edge cases will misbehave. Manageable.

With AI, the distance is substance — because a mock hides the two properties that define how the product will actually feel:

Latency. In a mock, the answer appears instantly (or after a scripted, tasteful little spinner). In reality, the model call takes time — sometimes a second, sometimes ten, sometimes longer if it reasons or chains tool calls. "Instant" is the single easiest thing to fake and one of the hardest to deliver. I've written before about a demo where the real fix wasn't UI at all — it was lowering the model's reasoning effort so responses came back in ~1.5s instead of 10–20s. You only discover that when the model is actually in the loop.
Non-determinism. A mock always returns the perfect answer, because someone wrote it. The real model returns a answer — usually good, sometimes wrong, occasionally confidently wrong, and different each time. The mock demonstrates a certainty the product doesn't have. Everything I keep harping on about evaluating LLM outputs exists precisely because real results vary and you have to measure that variance. A mock makes it disappear.

So when you demo AI with a mock, you're not just smoothing over rough edges. You're selling away the two risks the project actually has. The client falls in love with an instant, always-correct assistant, and then the team has to build something that is neither of those by default. That gap doesn't close itself — someone pays for it later, usually in eroded trust during the build.

So which one?

Honestly, it depends on what you're trying to learn from the demo:

If you're validating desire — "would anyone want this?" — and talking to people who think in outcomes, not architectures, a mock is often the efficient, correct tool. Don't build a backend to answer a marketing question.
If you're validating feasibility — "can we actually build this, and will it feel good?" — a mock answers the wrong question convincingly. That's where you want the real thing wired up.

My own bias leans toward the functional PoC, and it's leaned further the more I work with AI — because with AI the risk lives exactly in what the mock paints over. A functional PoC doesn't have to be ugly or slow to build, either: the AG-UI demo I put together recently is real end to end — real model, real latency, real rendered widgets — and it's about a hundred lines. The "real is too expensive to demo" assumption is often less true than it looks.

But I hold it loosely. A mock that honestly sells a vision, followed by a team that closes the gap, is a perfectly good way to build a company. The failure mode isn't mocking — it's mocking the risky parts and then quietly hoping reality will cooperate.

Or run both

It doesn't have to be binary. I've worked on projects that ran both at once: a mocked path to guarantee a clean end-to-end walkthrough — the story you can always tell without something breaking mid-meeting — and the real system alongside it, to probe live with different cases. The mock de-risks the narrative; the real part invites the hard questions. You get the controlled wow and the honest feedback in the same session, as long as everyone in the room knows which half is which.

The question I'd open up

Maybe the real question isn't "mock or functional." It's which moment you're optimising for: the signature at the end of the demo, or the trust at the end of the first sprint. Sometimes those point the same way. With AI, more often than I'd like, they don't.

How does your team demo — and have you ever been bitten by the gap between the demo and the thing you shipped? I'd genuinely like to hear it.

Related: When the Chat Builds Its Own Interface (a functional demo, end to end) and LLM-as-Judge Is Three Decisions (on measuring the variance a mock hides).

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

When the Chat Builds Its Own Interface

JaviMaligno — Tue, 16 Jun 2026 10:49:02 +0000

At AI Signals x LangChain Community London #32, Sofía Sánchez-Zárate from CopilotKit gave a talk titled "AG-UI & The Generative UI Spectrum." The demo that stuck with me: you ask for help working out a loan repayment, and instead of answering with a single number — one you'd have to ask it to recompute every time you change the rate or the term — it renders an actual working calculator inside the chat. Inputs you can edit, a total that updates live. Not a screenshot of one; a real widget the agent assembled for that question, on the fly.

That is generative UI: the chat stops being a text box and becomes a surface the agent can compose.

I've already been building this — by hand

The reason the talk landed is that I've been collaborating on a conversational KYC project that does exactly this. The agent doesn't answer onboarding questions with paragraphs. When it needs a beneficial-ownership structure, it renders a form to add people and their relationship to the company. When it has extracted data from an uploaded document, it shows an editable confirmation panel, not a wall of text. It renders file uploads, a signature pad, a shareholder chart — all in the conversation.

Under the hood, the pattern is straightforward. The agent (built on LangGraph) pauses with a typed interrupt — a payload whose type is person_form, confirm_summary, signature_pad, shareholder_chart, and so on. The React front end keeps a registry that maps each type to a component, renders it, lets the user interact, and sends the result back so the graph resumes from exactly where it paused.

It works really well. It is also entirely homegrown and welded to our stack: our interrupt format, our widget registry, our front end. Which is precisely why Sofía's talk hit — what we built vertically by hand is the thing AG-UI standardises.

What AG-UI actually is

AG-UI (Agent-User Interaction) is a lightweight, event-based protocol for the channel between an agentic backend and the front end a human looks at. Instead of returning one final text blob, the agent emits a stream of typed events the UI renders as they arrive:

Lifecycle — RunStarted, RunFinished, RunError, with optional StepStarted/StepFinished pairs.
Text messages — streamed token by token.
Tool calls — so the UI can show what the agent is doing.
State — snapshots plus deltas to keep front end and agent in sync.

Our interrupt-plus-registry mechanism is a private dialect of this. AG-UI is the attempt to make it a shared language — so a conversational backend isn't permanently married to one bespoke front end, and so human-in-the-loop (pause, let the user confirm or edit or sign, resume) is a native primitive rather than something you wire up yourself.

The generative UI spectrum

The framing I found most useful was the spectrum — how much of the interface the model dictates, from least to most:

Typed components. The agent picks from a fixed catalogue of front-end components and fills them with data. This is exactly what our KYC widgets are: the model doesn't invent the form, it chooses person_form and populates it.
Declarative UI. The agent emits a description of the interface and the client renders it. A2UI (Google's spec) streams a UI tree as JSONL — a serialised layout of components and bindings the client builds, rather than a fixed catalogue the client already has. Open-JSON-UI (OpenAI) is the same idea from the other camp; it's the lineage behind the interactive charts ChatGPT has started rendering inline — an equivalent protocol, though not strictly the one CopilotKit proposes. (Worth clearing up a common confusion: A2UI is not AG-UI. A2UI is one UI representation; AG-UI is the transport that can carry it.)
Iframe-embedded. MCP-UI ships self-contained UI inside a sandboxed frame.
Generated pixels. At the far end, the model renders the interface itself — the territory I explored in Software Is Dissolving Into the Model, where Flipbook and Project Genie generate every frame with no DOM at all.

AG-UI's bet is to be the neutral pipe across that whole range: standardise the channel, and let the representation slide from typed → declarative → generated without rewriting the backend each time. The generated-pixel end is where I'd previously argued software is heading; generative UI is the pragmatic middle of the same line — structured enough to be reliable, generated enough to be flexible.

Why forms are the killer case

Sofía pointed out that conversational AI over forms is where this shines — and that DocuSign, whose entire business is documents, agreements and signatures, has a partnership with CopilotKit. That clicked immediately, because forms are exactly where plain text falls apart.

A conversation is a great way to gather information and a terrible way to structure it. "List every shareholder with more than 25%, their nationality, and their relationship to the directors" is painful as a back-and-forth of messages and trivial as a small rendered table you fill in. Confirming fields pulled from a passport, drawing a signature, reviewing an ownership tree — none of that wants paragraphs; it wants widgets. That's the whole premise of the KYC project: replace the rigid step-by-step wizard with a conversation that still produces structured, validated, signable data. Generative UI is what makes the conversation and the structure the same surface.

What standardising would buy

Our homegrown protocol works, but it's coupled. A standard like AG-UI would let the same conversational backend talk to any front end that speaks it, mix in declarative UI specs where a fixed widget catalogue is too rigid, and get human-in-the-loop pauses as a first-class part of the protocol instead of a convention we maintain. The cost is the usual one for adopting a young standard: betting on its event model and its trajectory.

Given that LangGraph — which our project already runs on — is among the frameworks with AG-UI integration, the migration path from "our dialect" to "the standard" is unusually short. That makes it a real option, not a rewrite.

Trying it: from hand-built pattern to SDK

To feel the difference between my bespoke version and the standard, I built a tiny demo with the AG-UI SDK — a Pydantic AI agent on Azure OpenAI, a CopilotKit React frontend, and the two widgets declared as frontend tools. Ask it to estimate a loan repayment and it renders an interactive calculator in the chat; say you want to apply and it renders a form. The Python side is essentially "wire a model to AG-UI" — the generative UI lives on the client, in about a hundred lines total.

That's the whole point in one screen: the same loop as the KYC project — model picks a widget, client renders it, the user interacts, the result flows back — but the wiring is a protocol instead of a registry I maintain by hand. Code on GitHub.

The middle of the spectrum

The honest takeaway is that generative UI answers a question I left open in Software Is Dissolving Into the Model: if the rendered surface is drifting from hand-coded components toward model-generated pixels, where do you stand today, on a product that has to actually work? The answer is the middle of the spectrum — let the model choose and compose structured UI, keep the rendering reliable — and a protocol like AG-UI is what lets you pick your spot on that line and move along it later. I've been standing there by hand for months. It's good to see it getting a name.

Inspired by Sofía Sánchez-Zárate's talk "AG-UI & The Generative UI Spectrum" at AI Signals x LangChain Community London #32. Protocol details: the AG-UI docs and the AG-UI Dojo. Related reading: Software Is Dissolving Into the Model.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

LLM-as-Judge Is Three Decisions

JaviMaligno — Sun, 14 Jun 2026 15:47:47 +0000

Last week I went to AI Signals x LangChain Community London #32, and one idea from Bilge Aksu's talk on evaluation has been rattling around my head since. She put it on a slide that read "Context comes first" — with a subtitle I keep quoting: you can't even choose the right unit or the right dimension until you know the context.

That sentence reframed how I think about LLM-as-judge. We tend to treat "use an LLM to grade outputs" as a prompt-engineering problem: write a good rubric, ask for a score from 1 to 10, parse the number. But a judge that produces a number is not the same as a judge that produces a useful number. The useful part is decided long before the prompt — in three choices that most people make implicitly and therefore make badly.

The trap: a score that nobody validated

Here is the failure mode Bilge described that landed hardest for me. Your judge says a conversation was "good" — but the user never came back. Or it flags a conversation as "bad" — and that user is your most loyal returning customer. The judge's criteria and the actual signal from users point in opposite directions.

When that happens, the instinct is to tweak the rubric. The real problem is usually upstream: the judge was scoring the wrong unit, on the wrong dimension, with too little context to know better. A 1-to-10 "overall quality" score hides all three mistakes behind one confident-looking integer.

So before writing a single line of judge prompt, I now force myself through three decisions, in order.

Decision 1 — Context: the situation, decided first

Context is what the judge gets to see, and it is logically prior to everything else. Who is the user? What kind of request is this? What information was actually available when the answer was produced?

A judge with the wrong context will be confidently wrong. If I ask "was this diagnosis correct?" but only show the judge the final chat message — not the logs, not the tool outputs the agent actually saw — then the judge is grading prose, not correctness. It will reward a fluent, plausible answer over a terse, right one.

Getting context right means deciding deliberately: does the judge see the original question, the retrieved documents, the tool calls and their results, the full thread? Each of those is a knob, and leaving them at the default ("just the output") is how you end up with scores that correlate with verbosity instead of truth.

Decision 2 — Unit: a turn, a conversation, or across sessions

The unit is what you score. The same system can be evaluated at three very different granularities, and they answer different questions:

Unit	What it answers	Good for
Turn (one-shot)	Was this single response correct?	Questions with a clear, checkable answer
Conversation (thread)	Did the whole interaction reach the goal?	Iterative, multi-step problem-solving
Session / across sessions	Did the user's underlying problem get solved over time?	Retention, trust, real-world outcome

This distinction became concrete when I started evaluating my production DevOps agent — a Slack bot that triages infrastructure questions by querying real tools (Kubernetes, cloud CLIs, CI/CD, logs).

Some of its requests are genuinely one-shot: "where do I find the logs for service X?" has a right answer, and a turn-level judge is the correct unit — did it point to the right place, yes or no. Others are iterative: a 500-error investigation that takes ten reasoning rounds and a dozen tool calls. Scoring only the final turn there throws away the most important information — how it got there.

And a third category isn't really a quality judgment at all. Tool-call count, tool-call type, token usage, number of reasoning rounds, wall-clock time — these are efficiency metrics. They belong on their own axis, not blended into a "quality" score. Conflating "was it right" with "was it cheap" is one of the fastest ways to produce a meaningless number.

Decision 3 — Dimension: which aspect of "good"

The dimension is which property you are measuring. "Good" is not one thing — it's accuracy, helpfulness, warmth, safety, conciseness, and a dozen others, and they trade off against each other. A single judge that scores "overall good" is silently averaging incompatible properties.

The fix is boring and effective: one judge, one dimension, ideally a binary verdict plus a short written critique rather than a number on a scale. "Is this factually correct? yes/no, and why" gives you something you can act on and audit. "Rate the quality 1–10" gives you noise with a decimal point.

Putting it together: grading the DevOps agent

When I validated the DevOps agent against real support history, the three decisions made the evaluation legible instead of hand-wavy.

Context: I took real requests from an internal support channel and gave the judge the original human resolution thread alongside the bot's answer — so it was comparing like with like, not grading the bot in a vacuum.

Unit: per request (turn-level for the one-shot ones, thread-level for the investigations), plus a separate efficiency track.

Dimension: correctness first (is the answer right?), then thoroughness (did it find what the human found — or more?), with efficiency kept on its own axis.

The headline numbers, across ten real cases:

Metric	Value
Correct responses	10/10 (100%)
Better than the human resolution	6/10 (60%)
Worse than the human	0/10
Average response time	~2.5 min vs hours/days for the human cycle

Then I shipped a persistent memory system and re-ran the same ten cases as a before/after benchmark — and this is where keeping efficiency on its own axis paid off. Quality (correctness) stayed flat at 100%, which is exactly what you want: the change wasn't supposed to make answers more correct, it was supposed to make them cheaper to produce. On that axis the gain was clear — total time dropped 57%, and one "known-answer" case went from 16 tool calls to 2 (about 7× faster) because the bot remembered the answer instead of re-investigating. Two cases didn't improve, and one actually over-investigated — which the efficiency metrics surfaced immediately, where a blended "quality" score would have hidden it.

The dimension you forgot to validate

Now back to Bilge's trap, because my own numbers contain it. "Better than the human in 60% of cases" — better on which dimension, and who decided? I decided, reading threads, using my own judgment as the rubric. That's a perfectly good starting point. It is not the same as the dimension that actually matters in production: did the person who asked get unblocked and trust the answer enough to act on it?

That gap — between the criterion your judge optimizes and the outcome your users actually experience — is the whole game. The three decisions don't close it for you. What they do is make it visible: when you've named your context, unit, and dimension explicitly, you can point at exactly which one is drifting from reality, instead of staring at a single number and wondering why it disagrees with your retention chart.

A checklist before you write the judge prompt

Context — What exactly does the judge see? Does it have everything a fair grader would need, and nothing that just rewards fluency?
Unit — Am I scoring a turn, a conversation, or a session? Does that match the question I'm actually asking?
Dimension — One property per judge. Is "efficiency" sneaking into my "quality" score?
Verdict shape — Binary + critique over a 1–10 scale, wherever I can.
Validation — Have I checked the judge against a real-world signal (human labels, retention, did-they-come-back) — or am I trusting a number nobody validated?

The prompt is the last 10% of building a good eval. The first 90% is deciding what you're measuring, on what, with what in view. Context comes first.

Inspired by Bilge Aksu's talk at AI Signals x LangChain Community London #32. The DevOps agent it's applied to is described here. Further reading: Hamel Husain's guide to LLM-as-judge and the MT-Bench paper that kicked off the field.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Building a Production DevOps Agent: From Slack to Kubernetes

JaviMaligno — Thu, 30 Apr 2026 09:37:13 +0000

At SimpleKYC, the #devops_requests Slack channel is where every team goes when something breaks — or when they think something might be about to break. Pod stuck in CrashLoopBackOff? Pipeline failed? 502 from a service that was fine yesterday? It all lands there.

A colleague analyzed the channel and found that 70% of requests could be self-resolved by querying the same tools the DevOps team uses: kubectl, Azure CLI, Bitbucket pipelines, health checks. The knowledge wasn't secret — it was just scattered across terminals, runbooks, and people's heads.

We were already using Claude Code to investigate issues ourselves. The next step was obvious: put an agent in the channel that could do the same thing, for everyone, automatically.

The Agent Architecture

The bot is a generic LLM agent — not a collection of pre-built workflows. When someone mentions @DevOpsBot in a thread, GPT-5.4 (via Azure OpenAI) decides which tools to call, in what order, with what arguments. It can chain up to 20 tool calls per request, building up context as it goes.

This matters. A pod crash investigation might require:

kubectl get pods to find the failing pod
kubectl describe pod to check events
kubectl logs to read the error
A Bitbucket API call to check the last deployment pipeline
A health check on the service endpoint

No pre-built workflow can anticipate every combination. The LLM constructs the investigation path dynamically, adapting based on what each tool returns.

Six Tools, All Read-Only

The bot has access to six tools:

Tool	What it does	Examples
kubectl	Kubernetes diagnostics across 9 AKS clusters	`get pods`, `describe`, `logs`, `top`
az cli	Azure resource queries across 5 subscriptions	VM status, Postgres info, storage, APIM
Bitbucket	CI/CD pipeline status, repo content, config files	Pipeline logs, `values.yaml`, branch diffs
Health check	HTTP endpoint testing	Status codes, response times, body preview
Jira	Issue search and context	JQL queries, ticket details, comments
Confluence	Documentation search	Runbooks, architecture docs, onboarding guides

Every tool is strictly read-only. kubectl blocks delete, apply, patch, scale, exec, and edit. Azure CLI only allows show and list. There's no way for the agent to modify infrastructure — by design.

Read-Only First: A Deliberate Choice

This wasn't a technical limitation — it was a trust-building strategy. The agent's first job is to prove it can understand the infrastructure correctly before anyone considers letting it change anything.

The roadmap is progressive:

Phase 1 (current): Read-only diagnostics — investigate and report
Phase 2: Safe actions in dev/test — restart pods, rerun pipelines
Phase 3: Approval workflows for staging/production — the agent proposes, a human approves via Slack reaction
Phase 4: Full traced autonomy — the agent acts independently with complete audit trails

Escalation to humans isn't a phase — it's always present. The agent knows when it's out of its depth and routes to the right person. That's not a feature you add later; it's part of the human-in-the-loop design from day one.

This mirrors how you'd onboard a new team member. You don't give someone production access on day one. You let them observe, ask questions, and demonstrate understanding first.

Infrastructure as Knowledge

The agent's system prompt encodes the team's accumulated infrastructure knowledge: cluster naming conventions, namespace patterns, GitOps workflows, config repo structures, and common troubleshooting paths.

For example, the agent knows that:

Deployments follow a GitOps pattern — editing config files in environment branches triggers ArgoCD reconciliation
Kubernetes labels follow consistent naming patterns that map services to environments
The promotion chain goes through multiple stages from development to production
Some services run on VMs with different config structures than the K8s-based ones

This knowledge didn't come from a single source. I had context from working with the infrastructure, the LLM explored the actual clusters and repos to fill gaps, and then a DevOps engineer shared the documentation he personally uses with Claude — that was the biggest single jump in accuracy. After that, I refined edge cases through testing.

The result is that when someone asks "why is service X failing in UAT?", the agent already knows which cluster to check, which namespace, what label selector to use, and where the config repo lives. It doesn't need to be told.

Output Compaction: Saving 79% on Tokens

kubectl and Azure CLI return verbose JSON. A kubectl get pods across a namespace can easily be 8,000+ characters. Feed that raw into the LLM context and you burn tokens fast.

The bot compacts tool outputs on the fly — parsing JSON, extracting relevant fields, and discarding noise. A raw kubectl response of 8,000 characters becomes ~1,600 characters after compaction, with no loss of diagnostic value. That's a 79% token reduction that directly translates to lower costs and faster responses.

The compaction is tool-aware: it knows which fields matter for kubectl (status, restart count, images, events) vs. Azure CLI (provisioning state, FQDN, SKU) vs. Bitbucket (pipeline state, step results, error messages). If JSON parsing fails, it falls back gracefully to truncation.

Security Guardrails

Building an agent that runs shell commands against production infrastructure requires paranoia-level security:

Command validation: Every tool call goes through shlex.split with an array-based subprocess call — never shell=True. This prevents injection by construction.

Blocklists: Shell metacharacters (;, |, &, backticks, $()) are blocked at the input level, before any command reaches the shell.

Allowlists: Each tool defines exactly which subcommands are permitted. For kubectl, that's get, describe, logs, top, config, version. Everything else is rejected.

Output limits: Tool results are capped at 8,000 characters (before compaction) to prevent context window flooding.

Private IP blocking: The health check tool blocks requests to localhost, 10.x, 192.168.x, and 172.16-31.x to prevent SSRF.

The Moment It Clicked

Someone asked the bot to check resource usage for a service. The Grafana query was failing, and the bot started investigating — as usual. But then something unexpected happened: it found its own bug.

The bot's Grafana tool was sending step: '1h' as a string, but the implementation tried to parse it as an integer. ValueError: invalid literal for int() with base 10: '1h'. The bot traced the error to its own code, reported it honestly — "the error was mine" — and then pivoted: instead of giving up, it checked what it could verify (the deployment, replicas, pod status, resource definitions), gave an honest assessment of what it couldn't answer yet, and proposed a fix for its own Grafana tool.

That's meta-debugging. An agent that not only investigates infrastructure issues, but catches and reports its own failures with the same rigor. No hallucinated numbers, no pretending everything is fine — just "here's what I can confirm, here's what I can't, and here's how to fix it."

The First Test Went Wrong

The first live demo with the DevOps team exposed two UX problems I hadn't anticipated.

The first: someone tagged @DevOpsBot in an existing thread — just the mention, no additional text. They expected the bot to read the thread context and help. Instead, it silently dropped the message. I'd built a guard against empty inputs as a safety measure, but "empty mention" and "useless input" aren't the same thing. In a thread with 15 messages of context, a bare @DevOpsBot is a perfectly valid request for help. This guard made the bot useless in one of the most common Slack patterns: jumping into an ongoing conversation.

The second: requiring an @mention at all is friction. In a fast-moving incident, typing @DevOpsBot before every question feels like ceremony. But it's an acceptable tradeoff as part of the trust-building process — no accidental triggers, no noise, clear audit trail of who asked what. The alternative, auto-responding to everything in the channel, risks unintended tool calls and noise that erodes trust faster than the friction does.

Both were quick fixes in code, but they taught me something about building agents for real teams: defensive defaults that make sense in isolation can break the most natural interaction patterns. Test with the actual users early — your assumptions about how people will talk to the bot are probably wrong.

What I Learned

Tools first was the right call. Getting kubectl and Azure CLI working reliably was the quick win that let me start experimenting immediately. The system prompt refinements came naturally from watching the agent struggle with real questions — you need the tools running to know what knowledge is missing.

Compaction should be built in from day one. I added it after noticing token costs climbing. In hindsight, every agent that calls tools returning structured data should compact by default.

Design for thread context early, even if you implement it later. The bot reads up to 20 previous messages in a thread to maintain conversation continuity. Without this, every follow-up question ("what about the UAT environment?") would lack context. You don't need it for initial testing, but keep it in mind from the start — retrofitting conversational state is harder than designing for it.

The Stack

LLM: GPT-5.4 via Azure OpenAI (function calling)
Slack: Slack Bolt + Socket Mode (async Python)
Runtime: Python 3.11, asyncio throughout
Deployment: Docker → AKS via ArgoCD (GitOps)
Observability: Prometheus metrics, Application Insights, structured logging
Testing: pytest (unit + integration + e2e against real Slack)

Is It Worth Building?

If your DevOps team spends significant time on questions that amount to "look at this thing and tell me what's wrong" — yes. The investment is primarily in encoding your team's infrastructure knowledge, not in the LLM plumbing.

The read-only constraint makes the risk profile low. The agent can't break anything. The worst case is a wrong answer, which is the same worst case as a human answering from memory without checking.

The best case is that 70% of your #devops_requests channel gets answered in seconds, and your DevOps engineers focus on the 30% that actually needs human judgment.

Built with Python, Azure OpenAI, Slack Bolt, and a lot of infrastructure knowledge. Deployed on AKS via ArgoCD. View the project details.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

From Claude Artifact to Production PWA: Building VitaminD Explorer

JaviMaligno — Wed, 29 Apr 2026 20:19:26 +0000

It started with a simple question: can I synthesize vitamin D today in London?

I'd been reading about how latitude, season, and skin type determine whether sunlight can actually trigger vitamin D production. The science is clear — below a certain solar elevation angle (~45–50°), UVB radiation is too weak for synthesis regardless of how long you stay outside. But figuring out when that window opens for a given city on a given day? That's a calculation involving solar declination, hour angles, and spherical trigonometry.

So I asked Claude.

The Artifact That Started Everything

Claude didn't just answer the question — it generated an interactive artifact with a solar elevation curve, a day-of-year scrubber, and a threshold line showing when synthesis was possible. It used Spencer's 1971 formula for solar declination and the equation of time, projected onto a clean SVG chart.

I expected a paragraph with some numbers. Instead I got a working prototype.

I played with it for a few minutes — scrubbing through the year, watching the synthesis window expand in summer and vanish in winter — and thought: this is actually useful. Not just for me, but for anyone living at high latitudes wondering if they should bother going outside for their vitamin D.

That's when I decided to build it for real.

From Toy to Product: The Technical Decisions

Next.js App Router as the Foundation

I went with Next.js 16 and the App Router. The app needed client-side interactivity (D3 charts, geolocation, animations) but also server-side API routes (weather data proxying, push notification delivery, city search). App Router gives you both in one project without stitching together separate services.

D3.js on Canvas: Performance Over Convenience

The original artifact used SVG for everything. That works for a single chart, but I wanted three visualizations: a daily solar curve, an interactive world map, and a latitude × day-of-year heatmap (130 latitudes × 365 days = 47,450 cells).

SVG can't handle that on mobile. So I moved to Canvas rendering with an SVG overlay for interactive elements — hover tooltips, click targets, threshold adjusters. The world map uses an equirectangular projection with pan and pinch-to-zoom, rendering 195 countries from TopoJSON on every frame. Device pixel ratio scaling keeps it sharp on Retina displays.

The tradeoff is complexity. Canvas means manual coordinate transforms, hit-testing, and redraw management. But it was necessary — the heatmap alone would have created 47K DOM nodes as SVG rectangles.

The Vitamin D Math: Not Just Solar Angles

The artifact only calculated when synthesis was possible. I wanted to answer how long you need to stay outside for a given dose — say, 1000 IU.

This required modeling the actual photobiology. The rate depends on:

UVI (UV index at the current hour)
Skin type (Fitzpatrick scale I–VI, each with a different MED — Minimal Erythemal Dose)
Exposed body area (face and hands vs. t-shirt and shorts)
Age (synthesis efficiency drops ~1.3% per year after 20, per Holick et al. 1989)

And critically, it's not linear. Previtamin D3 doesn't just accumulate — it photodegrades under continued UVB into inert compounds (lumisterol and tachysterol). This is your body's built-in overdose protection, and it means there's a ceiling on how much you can produce per session.

I modeled this as a saturating exponential:

IU(t) = IU_sat × (1 - e^(-R×t / IU_sat))

Where IU_sat is the photodegradation ceiling (19,200 IU for full-body 1 MED, per Holick 1982, scaled by area and age) and R is the production rate based on Dowdy et al. 2010 MED tables. At low doses, it matches Holick's linear rule. At higher doses, diminishing returns kick in. The model agrees with Young et al. 2021 (n=75, in-vivo validation).

Real Weather Data: Open-Meteo

Solar geometry gives you clear-sky estimates, but clouds matter enormously. A 70% cloud cover can reduce UVB by 60%. I integrated the Open-Meteo API for hourly UV index and cloud cover — no API key needed, free for non-commercial use.

The tricky part: Open-Meteo's UV index already includes cloud cover. If you apply a cloud penalty on top, you double-count it. I caught this bug after noticing that exposure times on cloudy days were absurdly high. The fix: only apply the cloud factor to theoretical (solar geometry) estimates, never to real API data.

Why a PWA and Not a Native App

This was a deliberate choice. A vitamin D calculator needs to work on any device — iOS, Android, desktop — and I'm a solo developer. Building and maintaining native apps for two platforms plus a web version was out of the question.

PWAs close most of the gap that used to matter:

Installation: "Add to Home Screen" works on both iOS and Android — no app store, no review process, no $99/year Apple Developer fee
Push notifications: Supported on Android since 2015 and on iOS since 16.4 (March 2023) — the last major blocker is gone
Offline: Service workers cache everything needed for core functionality
Updates: Deploy to Vercel, users get the new version on next visit — no "please update the app" friction

What you don't get: background location tracking, health app integrations (Apple Health, Google Fit), and a presence in the App Store for discoverability. The first two could matter for future features (wearable integrations, automatic daily tracking). The third is a real distribution disadvantage — people search app stores, not the web.

For now, the tradeoff is clearly worth it. If the app grows to need native APIs (wearables, health data), a hybrid approach with Capacitor or a thin native wrapper around the existing web app would be the natural next step — not a rewrite.

Offline Support

A vitamin D calculator is most useful when you're already outside — potentially with no signal. The service worker caches all pages and static assets at install time, with a network-first strategy for dynamic content and an offline fallback page.

Push Notifications

This was the feature that felt most "app-like." A Vercel cron job runs daily at 8 AM UTC, fetches real UV data for each subscriber's saved location, calculates their personal synthesis window, and sends a Web Push notification:

"London · 12 min for 1000 IU · Best hour: 13:00 (UVI 4.5) · Window: 11:00–15:00"

The implementation uses VAPID keys with the web-push library, and subscriptions are stored in Supabase. One gotcha that cost me 53 days of broken push notifications: echo "$KEY" | vercel env add appends a newline that Vercel stores literally inside the VAPID key. The fix is printf '%s' instead of echo. A subtle shell behavior that silently corrupted a cryptographic key.

Six Languages and 5,000+ Cities

The app supports English, Spanish, French, German, Russian, and Lithuanian via next-intl. Each language file is ~480 lines covering UI labels, Fitzpatrick skin descriptions, and a full educational FAQ about vitamin D science.

City search combines a built-in database of 50+ major cities with Supabase full-text search (pg_trgm) over GeoNames' 200K+ city dataset. City names are localized — "Londres" in Spanish, "Лондон" in Russian.

What I Learned

Claude artifacts are underrated as prototyping tools. The artifact wasn't a mockup or a diagram — it was working code with real solar math. That head start meant I could focus on product decisions (what visualizations? what user flow?) instead of re-deriving spherical trigonometry.

Canvas D3 is worth the pain for data-dense visualizations. If you're rendering fewer than 1,000 elements, stick with SVG. Beyond that, Canvas is the only way to keep mobile smooth.

PWA push notifications are surprisingly powerful and surprisingly fragile. The Web Push spec works well, but the operational surface area (VAPID keys, cron timing, subscription cleanup, error handling for uninstalled apps) is larger than the code suggests.

Photobiology is more nuanced than "go outside for 15 minutes." The difference between Fitzpatrick I (pale, Celtic) and VI (deeply pigmented) is a factor of 6× in MED. Body area exposure matters more than most people think — face and hands alone (10% body area) can require 5× longer than a t-shirt and shorts (25%).

Try It

VitaminD Explorer is free, open source, and works on any device. Search your city, set your skin type and exposure, and see exactly when and how long you need for your daily vitamin D.

The original Claude artifact is still live if you want to see where it all began.

Built with Next.js 16, D3.js, Supabase, and Vercel. Source on GitHub.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.

Software Is Dissolving Into the Model

JaviMaligno — Sat, 25 Apr 2026 18:42:37 +0000

Two repositories caught my eye this month. The first is google/agents-cli, Google's official tooling for building agents on Google Cloud — a CLI plus a bundle of markdown "skills" that any coding assistant (Claude Code, Codex, Gemini CLI, Cursor) can pick up. The second is Flipbook, an experimental "browser" launched by ex-OpenAI researchers two days ago. Flipbook has no HTML, no DOM, no rendered components. Every pixel you see — including the text — is generated frame-by-frame by a video diffusion model streaming over a WebSocket.

They look like they belong to different conversations. They don't. Put them next to each other and a pattern emerges: the layer we used to call "software" is being squeezed from both ends. What we write is becoming markdown. What users see is becoming a model output. The middle keeps getting thinner.

What we write is becoming markdown

Skills are repositories of instructions, not deployed services. The runtime is whatever coding agent the user happens to have installed. The artifact is a SKILL.md file with maybe a couple of helper scripts.

The last few weeks alone tell the story:

OthmanAdi/planning-with-files is a Claude Code skill that gives the agent a Manus-style persistent planning workflow — a few markdown files defining when to write task_plan.md, findings.md, progress.md. It has 9.2k stars. Manus, the company that built that workflow into a full product, was reportedly acquired for $2B. The IP was the pattern, not the implementation.
VoltAgent/awesome-agent-skills curates 1000+ portable skills across Claude Code, Codex, Gemini CLI, Cursor and 40+ other agents. 18.7k stars.
GitHub shipped gh skill on April 16 — a first-class CLI primitive for installing, pinning and publishing skills directly from GitHub repositories.
Google Workspace's official CLI now ships 100+ SKILL.md files, one for every supported API, plus 50 curated recipes for Gmail, Drive, Docs, Calendar and Sheets.

A year ago, "ship a tool for X" meant building an SDK, a service, or at least a wrapper library. Today it increasingly means writing a folder of markdown that any agent can load on demand. The trend isn't "AI helps you write software". It's "what we used to call software is now an instruction set".

What users see is becoming a model output

Now look at the other end of the pipeline. It's doing the same thing.

Flipbook launched on April 23 from Zain Shah's team (ex-OpenAI, ex-Humane, ex-Apple), out of South Park Commons. The interface is generated pixel-by-pixel by LTX Video, an open-source diffusion transformer for video, optimized to stream at 1080p/24fps via WebSocket from Modal Labs serverless GPUs. There is no DOM. There are no buttons in the traditional sense. When you "click", the model generates the next frames.

DeepMind has been doing the same thing at world-model scale. Project Genie rolled out to AI Ultra subscribers in January 2026, powered by Genie 3. You type a prompt. You walk around a generated world that is being generated as you move, at 24fps, with consistency held for a few minutes.

These are demos. Flipbook is barely useful as a product yet. Genie's worlds last a minute or two before the model loses coherence. But the direction is clearer than the products: the rendered output of an application is becoming a model inference, not a tree of components.

We're the ones who should adapt

A friend told me this week that he'd been trying to get an LLM to write internal reports following his company's exact style guidelines. After enough iteration he realized the prompt-engineering effort to keep the model on rails was taking him longer than writing the reports himself — and his company has multiple report types like that.

What he said next stuck with me: "Maybe the problem isn't that the AI can't adapt. Maybe we're the ones who should adapt. We're making it less efficient with all our constraints, when what it produces on its own is good enough for the purpose."

That insight isn't really about reports. It's the thing happening on both ends of the software stack right now.

A markdown skill is what you get when you stop trying to force the model into the shape of a typed SDK with versioned interfaces and start writing in the medium the model actually thrives in: natural language with examples. Flipbook is what you get when you stop trying to force the model to emit valid React components and start letting it render the pixels directly.

The friction we were adding — typed APIs, deterministic UIs, hand-coded glue — was buying us correctness, but at a cost we hadn't priced in. Every constraint also costs throughput, and a lot of those constraints existed because humans needed them, not because the model did.

That doesn't mean correctness goes away. It means we re-locate it. Evals replace types. Skill instructions replace SDK documentation. Frame-level objectives replace pixel-perfect Figma specs. We're trading one kind of rigor for another that fits the tool better.

What survives in the middle

So if what we write is becoming skills and what users see is becoming pixels, what's left for those of us building things in the middle?

I've been thinking about this in the context of my own work.

The MCP servers I've built — mcp-server-bitbucket, postgres_mcp, langfuse-mcp-server — already live in this new shape. They're not products with UIs. They're protocols any agent can pick up. The interface is the spec, not the screen.

The compliance classifier and the medical document parser depend on something that doesn't compress into a markdown file: a curated taxonomy, edge cases extracted from real documents, an evaluation suite that proves the model handles the long tail. The orchestration around them increasingly does compress into instructions, but the domain context doesn't.

The data-source automator's value is in a specific human-in-the-loop sequence we worked out the hard way — where the agent stops, what it asks, what it logs. Two years ago that lived in code. Today it lives more naturally as a skill plus an eval set.

What survives in the middle is the stuff that wasn't really code in the first place: the data, the taxonomy, the eval harness, the orchestration decisions. The stuff that was just "code" — the wrapper, the form, the boilerplate — heads to the skill layer on one end or the generated render on the other.

Engineers can't hide in the niche anymore

There's a corollary for those of us who write the code being squeezed: the gap between idea and shipped product is getting short enough that we can't stay tucked into a development niche and still add value.

When the middle layer was thick, you could be "the backend person" or "the LLM ops person" and contribute through a slice of the stack. The product owner had the idea, designers handled the surface, engineers handled the plumbing, and the handoffs were the work. When the middle is thin, those handoffs cost more than they coordinate. The model renders a lot of the surface. The agent carries a lot of the plumbing.

What's left for engineers looks more like product judgment than implementation — which problem to solve, what "good enough" means in this domain, where the model needs to be on rails and where it doesn't, what failure looks like and how to catch it. Even when you're not building your own product, you can't hide in your niche. You have to take ownership of the outcome to be trusted with it — because the part you used to be paid for, the implementation, is the part that's compressing.

Practical takeaways

Three things I'm acting on:

Ship the skill, not the wrapper. If you're building anything that lives behind an LLM, the wrapper code is the part with the shortest half-life. The skill — instructions, examples, evals — is what you actually own.
Stop fighting the medium. If you're spending more time constraining the model than the model is spending producing useful output, you're solving the wrong problem. Either accept what it produces or pick a different tool.
Invest where the moat is. The new defensible thing isn't "we built it well". It's "we have the data, evals and orchestration that prove this works in our domain". That's where I'm putting time on my own products.

Software isn't disappearing. It's redistributing itself across the stack — and the middle, the part we used to call "the application", is getting a lot thinner than I expected even six months ago.

Repositories referenced: google/agents-cli · OthmanAdi/planning-with-files · VoltAgent/awesome-agent-skills. Demos: Flipbook · Project Genie.

Originally published on javieraguilar.ai

Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.