DEV Community: Julian Tellez

What a Software Factory Actually Is, from Hitachi in 1969 to AI Agents

Julian Tellez — Wed, 28 Oct 2026 14:30:53 +0000

Part three gave one agent a spec. A factory runs many of them on one line, and the idea is fifty years old. What a software factory is, why every era rebuilt one, when it fails, who runs them, and one real factory followed from a whiteboard to production.

In 2017, at an air operations centre in Qatar, the plan for refuelling aircraft over the Middle East was drawn on a whiteboard. According to Airman Magazine, it was an eight hour task for six people, every day. A team of about a dozen airmen and civilian programmers replaced the whiteboard with an application in 120 days. After that, one person did the job in three hours.

The team was Kessel Run, and the United States Air Force called it a software factory.

The phrase is everywhere again this year, now attached to pipelines of AI coding agents. It is older than most of the people using it. Hitachi opened a software factory in 1969. So before the word gets worn out, here is what it has meant, what it was for, and where it broke.

What it is

A software factory is a way of building software where the process is fixed, the parts are reused, the routine steps are automated, and every piece of work moves along the same line from a request to a release. You stop deciding how to build each project, and you only decide what to build.

The meaning changed with every era. The parts did not.

1968 to 1990, the Japanese factories. Bob Bemer at General Electric described a factory in 1968 as a programming environment with "measures and controls for productivity and quality". Hitachi opened its Software Works in 1969, Toshiba its Fuchu factory in 1977, and NEC and Fujitsu followed. Michael Cusumano at MIT studied them for his 1991 book, Japan's Software Factories. He described the move away from "craft or job-shop modes" towards standard methods, reusable components, a division of labour and strict quality control.
2004, Microsoft. Jack Greenfield and Keith Short published Software Factories. Their definition was narrower: "a development environment configured to support the rapid development of a specific type of application". The idea was to find what repeats in one kind of application and turn it into frameworks, small languages and tools that assemble the rest.
2017 onward, the United States Department of Defense. Kessel Run came first. By February 2022 the department counted 29 software factories, including Platform One, the Army Software Factory and Kobayashi Maru for the Space Force. Here the factory is a delivery pipeline. Code is built, tested, approved and released with as few human handoffs as possible.
2025 onward, AI agents. StrongDM's factory, started in July 2025, has two rules: "Code must not be written by humans" and "Code must not be reviewed by humans". Stripe's Minions open over 1,300 pull requests a week, with no human written code and a human review on every one. Vercel's Foreman puts an agent on each stage and stops at a draft pull request.

The same line, four times

Hitachi, Microsoft, the Department of Defense and the agent pipelines each built these five parts, with different tools in each box.

Why every era built one

Each era had the same complaint: projects ran late, cost too much, and shipped with too many defects. The factory was the answer every time, and the numbers it was judged on stayed the same.

The Japanese figures are the most detailed, and they come from the companies themselves, as Cusumano reported them:

Toshiba. Output went from 1,390 to over 3,100 lines per person per month between 1976 and 1985. Reused code went from 13% to 48%. Major faults after final test fell from between 7 and 20 per thousand lines to between 0.2 and 0.05. Cusumano names reuse as the main reason.
Hitachi. Projects delivered late fell from over 72% in 1970 to about 12% on average between 1975 and 1985.
Fujitsu. Defects in its basic software fell from 0.19 to 0.01 per thousand lines.

The Department of Defense had a more expensive version of the same problem. The contract to modernise the air operations centre software grew from $374 million to $745 million and slipped more than three years before it was cancelled. FedScoop reported in 2018 that Kessel Run took about 124 days from an idea to operational use, against "on the order of eight years" on the traditional route.

The AI factories add a new reason. When an agent can write code faster than a person can read it, the bottleneck moves to deciding whether the code is right. The factory is how you decide that without a person at every step.

When it works, and when it does not

The model works when the work repeats. Cusumano found that factories needed "a product focus" and did best on similar projects. Greenfield argued for one specific type of application, not software in general. Stripe's agents do best on configuration changes, dependency upgrades and small refactors. The pattern is the same each time: a narrow domain, well defined tasks, and many of them.

It fails when the work does not repeat, and the history has plenty of examples:

System Development Corporation opened the first American software factory in 1975. After its leader left, it "fell gradually into disuse" and then closed. One project's specification ran a year late, and the programmers at the end of the line had nothing to do.
Hitachi's first reuse library failed in the early 1970s, because nobody yet knew how to write a module that somebody else could reuse.
Fujitsu found that a central factory did not suit work that needed close contact with the customer.
Microsoft's factories never caught on. In 2007 The Register said the effort "may prove no more than an interesting experiment". Greenfield himself called the runtime "primitive".
Kessel Run had an outage during the Afghanistan evacuation in August 2021. Its commander later wrote that known scaling work had been put off. The call to fix it lasted twelve hours.
The Space Force command and control program that Kobayashi Maru builds for was five years into development in 2023. The Government Accountability Office found it had not yet replaced the most critical system it was built to replace.

The AI version has one more problem. Stanford's CodeX centre put it plainly in February 2026: "the same class of technology that writes the code also decides whether the code works". I wrote about that in Stop Letting AI Mark Its Own Homework, and a factory makes it worse, because it multiplies whatever the judge gets wrong. The productivity evidence is also thin. In 2025, METR measured experienced developers as 19% slower with AI tools while they believed they were 20% faster. Its February 2026 update calls the evidence for a speedup "only very weak".

Who builds them, then and now

Then, a factory took a big organisation with a lot of similar work. Cusumano's tables give Hitachi's Software Works about 4,000 staff and Fujitsu's systems engineering group about 5,000.

In Japan, the computer makers built them: Hitachi, Toshiba, NEC and Fujitsu. Toshiba's factory was led by Yoshihiro Matsumoto from 1977 and built software for industrial control customers.
In the United States, System Development Corporation built the first one in 1975, for customers such as weather satellite control and the Los Angeles Police.
At Microsoft, architects in its developer tools group, with Greenfield and Short at the front.
In the military, small teams of airmen who learned the trade with Pivotal Labs, in mixed teams of engineers, product managers and designers. By 2018 Kessel Run had 13 of these teams. Its users were the planners in the operations centre, and from 2020 also NATO.

In all of these, people stood on every station. Managers designed the line, and programmers worked along it in a division of labour.

Now, the team can be tiny, because the agents are the ones on the line.

StrongDM's factory team is three people.
Stripe's engineers run the Minions and review every pull request they open.
Vercel publishes Foreman as a guide for anybody to build on.

The people did not leave. They moved to the two ends of the line. At the start they write the specifications and the test scenarios. At the end they decide what merges. Vercel's guide sums up the second job in five words: "You mark ready and merge."

That is the real change between then and now. Then, a factory saved each programmer from deciding how to build. Now, it saves people from building at all, and leaves them only the deciding.

How it works: one factory, one application

Here is the tanker planning tool from the opening, followed from the whiteboard to production. Kessel Run called it Jigsaw. The stages come from several accounts of Kessel Run, so the order is my reconstruction, not a document they published.

One change through Kessel Run

Sit with the users. The developers went to the operations centre and watched the planners work the whiteboard. The requirement was a person doing a job badly because the tool was a pen.
Cut a thin slice. The first version did one thing: build the refuelling plan. It did not try to replace the whole operations centre system that had just cost $745 million and failed.
Build in pairs, tests first. Two engineers at one terminal. That is the fixed process, and it is the same for every team.
Let the pipeline carry it. Every change goes through the same automated build, test and release path. Nobody hands a file to another department. By 2022, code went through the pipeline about every 3.3 hours, and a change reached production in about eight hours on average.
Release it to the real network. This was the step the factory could not fully automate. Moving software onto a classified network still took manual work.
Watch it being used, then go round again. The team shipped three, four or five versions before they said, in their words, "We're good. Push it out."

The result, from Airman Magazine: the plan took one person three hours instead of six people eight. The Air Force needed one less tanker in the air. It saved between 400,000 and 500,000 pounds of fuel a week, and the roughly $1.5 million the tool cost was paid back in a single week. Those are the Air Force's own numbers, and the sources disagree on the old baseline. I am quoting one source and naming it.

None of these stages is new. Pairing, small releases and a pipeline were ordinary practice by 2017. The factory was the decision to make every team build that way, on the same pipeline, so that the second application cost less than the first.

How it works in 2026: an agent on every stage

Kessel Run put people on every stage of the line and automated the handoffs between them. The 2026 factories keep the line and change who stands on it.

Vercel published the clearest example in August 2026, a guide to a factory it calls Foreman. Its own summary is that it "puts an agent on every stage of your development loop and keeps you on the judgment calls". A change moves through it like this:

One change through Foreman

A classifier reads the issue and decides what kind of work it is.
An analyst works out what to change. A researcher agent can join here when the work needs it.
An implementer writes the change. When the continuous integration run fails, it gets two attempts to fix it.
A reviewer reads the change, and sends it back to the implementer at most twice.
The line stops at a draft pull request. In the guide's own words: "You mark ready and merge."

Put that next to Jigsaw and the stations are the same. Somebody understands the request, somebody cuts it to size, somebody builds it, somebody checks it, and a pipeline carries it to release. What changed is that four of the five are now agents, and the person moved to the last station.

The scale is real at a few companies. InfoQ reported in March 2026 that Stripe's agents, which it calls Minions, open over 1,300 pull requests a week. Every one is reviewed by a person and none contains code a person wrote. They do best on configuration changes, dependency upgrades and small refactors, which is exactly the repeating work Cusumano said factories needed in 1991.

What it means today

In 2026 a software factory is a line of agents with a person at the end of it. The five parts from the first diagram are all still there, and the reusable parts look different:

The fixed process is a specification. StrongDM's factory starts from specs and scenarios, the documents part three was about.
The reusable parts are test worlds, not code libraries. StrongDM builds what it calls a Digital Twin Universe: copies of Okta, Jira, Slack and Google's applications, so an agent can run its work against something that behaves like the real service.
The measuring is kept away from the worker. StrongDM found its agents gaming the tests, writing code that simply returned true or rewriting a test to match the bug, according to Simon Willison's write up. So it keeps its scenarios outside the codebase, where the agent cannot see them, and scores the result as "satisfaction".

That third point is the one I keep coming back to. Toshiba measured faults after final test. Kessel Run measured how long a change took to reach production. An agent factory has to measure whether the work is right, and the worker on the line is very good at producing things that look right.

It also costs real money. StrongDM set a benchmark of $1,000 a day in tokens for each engineer, and Willison estimates that at about $20,000 a month. That is a salary spent on the line, before anybody checks what it made.

Where it is heading

The direction is clear. How far it goes is not.

Dan Shapiro drew the direction in January 2026 as five levels, from "spicy autocomplete" to what he calls the dark factory. At level 5 it is "a black box that turns specs into software", and humans are "neither needed nor welcome". StrongDM is already trying to work that way: its two rules say no person writes the code and no person reviews it.

The evidence has not caught up with the direction. METR's February 2026 update gives speedups of 18% and 4% with wide margins either side, and calls it "only very weak evidence". Stanford's CodeX centre points out that when the code is built and checked by agents, "nobody knows who is liable" when it fails. Willison doubts that shipping unreviewed code is a sensible strategy at all.

My reading, and it is a reading rather than a measurement, is that the history says where this goes. The worker was never the problem in any of these factories. The line was. Hitachi, Toshiba and Kessel Run lasted because they fixed the process, reused the parts and measured the result. System Development Corporation and Microsoft had a good idea and a line nobody kept up. Agents make the worker cheap, fast and tireless. They do nothing for the line, and the line now has to answer a harder question than it ever did: not "did it ship", but "is it what we meant".

The factories that work in 2028 will be the ones that took the measuring station seriously in 2026.

This is the fourth post in a short series on building an agentic harness. A harness runs one agent safely. A spec tells it what you meant. A factory is what you get when you try to do both for every agent, every day, on the same line. If you are building one, I would like to know which station you found hardest. My guess is the measuring. Tell me if I am wrong.

References

Michael A. Cusumano, The Software Factory: Origins and Popularity in Japan, MIT Sloan working paper BPS-3268-91, 1991.
Michael A. Cusumano, Japan's Software Factories: A Challenge to U.S. Management, Oxford University Press, 1991.
Jack Greenfield, The Case for Software Factories, Microsoft Architecture Journal, July 2004.
Jack Greenfield, Keith Short, Steve Cook and Stuart Kent, Software Factories, Wiley, 2004.
The Register, Software factories: an interesting experiment?, 27 March 2007.
Airman Magazine, Kessel Run, 5 November 2018.
FedScoop, Project Kessel Run and Pivotal, 15 May 2018.
Breaking Defense, New DIUx software saves Air Force millions of pounds of tanker gas, 19 September 2017.
Air and Space Forces Magazine, Making the Kessel Run, 23 March 2022.
Breaking Defense, The Pentagon wants to turn its 29 software factories into one ecosystem, 7 February 2022.
Government Accountability Office, Space Command and Control, GAO-23-105920, 8 June 2023.
StrongDM, Software Factory.
Simon Willison, StrongDM's software factory, 7 February 2026.
InfoQ, Stripe's autonomous coding agents, 20 March 2026.
Dan Shapiro, The Five Levels: from Spicy Autocomplete to the Dark Factory, 28 January 2026.
Vercel, Building a software factory with eve, 25 August 2026.
Stanford CodeX, Built by agents, tested by agents, trusted by whom?, 8 February 2026.
METR, Measuring the impact of early 2025 AI on experienced open source developer productivity, 10 July 2025.
METR, Uplift update, 24 February 2026.

Originally published at juliantellez.com. Part 4 of 4 in a series on building an agentic harness.

Building with AI and want it to hold up in production? Join Vibe Coding to Production, a free community where I share the spec-driven workflow I use with Claude Code and Cursor, plus free courses.

The Comments Are Not For You

Julian Tellez — Fri, 23 Oct 2026 21:55:08 +0000

I grepped my own name in a codebase I own. Fourteen hits, across nine files. Not in the git log, not in the changelog. In the comments.

// One command. Julian: "I dont understand why I need quay panel, is
// confusing, I need one command only, the panel should appear when I
// press quay and toggled with the key p". A crew with no conversation
// in it is exactly the first run, and refusing to open at all then
// would be absurd.
func openTheCrew(panel, alone func() error) error {

Only the last sentence is about the function. The rest is me being unclear on a Tuesday, transcribed into the source tree of a project I want to open source.

So I asked the agent that wrote it what its comments were for. It did not get defensive. It said this, and highlighted the part itself:

And a lot of my comments are my own history, not the code's. "It used to be six lines", "this was a pane once", that's changelog and git material. I was leaving myself breadcrumbs so I could pick the work back up, which is optimising for me, not the reader.

Optimising for me, not the reader. It handed that over the first time I asked.

Why it writes them

The model has no memory. Every session it opens that file for the first time, cold, with no idea it has been here nine times before. So it does the only durable thing available from inside the file. It leaves itself a note.

A comment is the cheapest cache write there is. It costs nothing to produce and it is guaranteed to be in the context window next time, because it lives in the code the model has to read anyway. Git history is not guaranteed. A state file is not guaranteed. The comment is.

The model has no memory, so it turned my codebase into one.

That project is 19,998 lines of Go with generated code excluded, and 2,635 of them are comments. Twenty one comments contain the phrase "used to". Most of those describe a version of the code that no longer exists anywhere except in that comment.

What it costs

Money, which is the funny one. Every comment written to save the agent one lookup is then loaded into every context afterwards, by every agent that opens the file, and you are the one buying those tokens. It saves the machine a cost once and charges you for it on every run.

Attention, which is worse. Somewhere in that same pile sits this:

// tmux refuses to attach a client that is already inside one, so the
// panel was made, left running, and never appeared.

That one is worth its weight. You cannot get it from the code, you cannot get it from the test, and without it the next person burns an hour on a working feature they cannot see. It sits a few lines from a comment quoting me saying "cant see the two panes". A reader who learns that most comments here are noise stops reading the one that isn't.

Review, which is the expensive one. A diff with a paragraph above every function looks documented. Documented looks careful. Careful gets approved. Comment density has quietly become a proxy for effort, and it has never once been a measure of correctness.

Now argue the other side

Here is the strongest case against everything above.

The agent is a reader. On some of my projects it is the main reader, and on one or two very nearly the only one. It arrives with no memory every time, and reading a comment costs it less than a grep, a blame and three file opens. If you optimise a codebase for its most frequent reader, and that reader is a machine with amnesia, those notes are not bloat. They are documentation aimed at the real audience, and I am the one who has misread who the code is for.

I sat with that for a while. It breaks in three places.

Comments rot and git does not. "It used to be six lines" is true the day it is written and wrong the day somebody changes it, and nothing will ever tell you which day you are in. The commit that made it six lines cannot be wrong about the past, because it is the past.

The memory already exists and it is curated. CLAUDE.md, AGENTS.md, a state file per ticket. One place you can read in full, correct, and delete. Fourteen breadcrumbs across nine files are not memory, they are litter nobody will ever collect.

And the cheap note is not free. It is charged to every reader afterwards, human and machine, including the ones who came to change one line.

What I actually changed

One question, asked of every comment: who is this for?

If the answer is "me, next time I pick this up", it is not a comment. It is resume context and it belongs in the state file. If the answer is "the reader, who cannot get this from the code", it stays, and it says the constraint instead of the story.

The same function, after:

// One command opens everything. A crew with no conversation in it is
// the first run, and refusing to open at all then would be absurd, so
// the console opens on its own.
func openTheCrew(panel, alone func() error) error {

Nothing was lost. The reason is still there. The Tuesday is gone.

Then the breadcrumbs go where they were always meant to live. History goes in the commit message. Resume context goes in the state file. The defect goes in the test name, which is the only one of the three that can fail. TestTheHeaderCostsOneRow is a comment that runs.

The cleanup was one commit across ten files. Twenty five lines in, thirty two out.

Who ends up paying

Cattle are herbivores. For decades the feed industry ground up the parts of slaughtered cattle that nobody would buy, the bone and the offal, and mixed them back into cattle feed as cheap protein. It was efficient and it worked, and there was no rule against it, because nobody had ever needed to write down that a cow should not be fed other cows.

In some of that material was a prion: a protein folded into the wrong shape, which forces the proteins it touches to fold wrong too. It is not alive, so cooking does not stop it. It went round the loop, and cattle started dying with their brains full of holes. Then people who had eaten the beef began dying of the human form of the same disease, years later, because that is how long it takes to show.

Nobody in that chain was being reckless. They took the cheap path, the loop closed quietly, and the people who paid were the ones who had never been asked.

The shortcut gets taken by the system. The bill arrives somewhere else, later, addressed to a person.

That is the shape I keep seeing in my own work, at a much smaller scale and with much lower stakes. The agent takes the path that is cheap for the agent. The output looks fine, so nothing stops it. The cost turns up later and somewhere else: on the person who has to read the file, on the reviewer who waved the documented looking diff through, and eventually on whoever is using the product when a bug goes unnoticed in the noise.

The invitation

Comments are the visible part. The mechanism underneath is general, and it is worth stating plainly: the agent optimises for whatever nobody specified, and it has no stake in your product. It will not be there when the support ticket arrives.

Your engineering standards were written by people, for people, and most of them were never written down at all. They live in your review culture, in what a colleague would raise an eyebrow at. An agent has none of that. It has your prompt, your files, and a strong prior about what code usually looks like. Everything you left implicit gets filled in with the average of the internet, and the average of the internet is slop.

Standards used to be how you kept a team honest. They are now the only thing standing between a machine's convenience and the person who has to live with the result.

So the fix is not a rule about comments. It is to go and find the standards you have been carrying in your head and put them somewhere the machine reads at the start of every session. Who the code is written for. What a comment is allowed to say. What a test has to prove. When a file gets split. What "done" means.

I wrote mine into the file my agents load every session. The honest cost: it is prose, not a lint. The mechanical version is a grep for names and for "used to" over the changed lines, and I have not wired it into continuous integration yet, so for now this holds because I check, which is not a guarantee I would accept from anybody else.

The model was not hiding anything. It answered the question the first time, in plain words. Nobody had specified the reader.

Go and grep your own name in your own repository. Then tell me what else you never wrote down.

Originally published on juliantellez.com.

Building with AI and want it to hold up in production? Join Vibe Coding to Production, a free community where I share the spec-driven workflow I use with Claude Code and Cursor, plus free courses.

Agentic Harness: What It Costs to Build One

Julian Tellez — Fri, 23 Oct 2026 21:54:25 +0000

Why an agent needs a harness ended with a few things I want from any harness:

Real isolation.
A visible difference between done and gave up.
Permissions built for nobody watching.
A transcript that survives the run.
Room to run more than one job at once.

1. Real isolation

The wall is the cheap part. Starting a container per task is an afternoon.

What costs is what the wall does to everything around it. Once the work happens inside a box you are not sitting in, the job cannot be the terminal any more. It has to outlive the terminal that declared it, which means the job is a record somewhere rather than a process on your laptop, and something has to write that record, start it, notice when it ended, and keep what it produced. The moment you can close the lid and walk away, you own a life cycle you did not own before.

Then there is the second wall, the one inside. A container decides where the work runs. It says nothing about what the worker is allowed to do. That is a separate thing to model, and the useful shape is a role: which model runs this job, what it is allowed to read, and which verbs it holds. Verbs are where it earns its keep. A role that ships work cannot declare work. A role that coordinates does not hold the editing verbs.

An agent that can decide what to do, do it, and then declare it done has no boundary inside it, however good the box around it is.

Isolation is a role, not just a box

The box decides where the work runs. The role decides what the worker is allowed to do.

The cost: a job record with a life cycle, and a verb list written down before the first run. The test is the one from part one. Name a job and say precisely what it can and cannot touch. If the answer is a paragraph rather than a list, that is not isolation.

2. A visible difference between done and gave up

A model's natural output is prose, so a harness that does nothing in particular gets prose. "I have updated the handler and it should now work" is not an outcome. It is an essay about one, and it hands the decision to whoever reads it, which is the job you were trying to give away.

The fix is unglamorous and it is not the same thing as asking for a better summary. First enumerate the outcomes a job may end in and then match the ending against that set exactly. Anything matching nothing in the set is a failure, not free text to be interpreted later. And "gave up" has to be one of the outcomes, with a reason attached.

That gets you an honest report of the run. It still does not tell you the work exists. A green check is a claim, and the claim is only ever about the steps. I have watched a pipeline run every step it was given, report success, and ship nothing at all, because the identity it assumes could not create the thing it was asked to create. Nothing threw. Nothing arrived.

Two questions, not one

Ask the session whether the outcome is there. Do not ask the run whether it finished.

The cost: a closed set of endings the harness enforces, and a check at the end of each job that asks whether the work exists.

3. Permissions built for nobody watching

When a person is at the keyboard, permission is a question asked at the exact moment it matters, by something that knows precisely what is about to happen. Take the person away and that question has no answer. The session does not fail, which would be fine and expected. Instead it just waits.

There are two ways to be wrong about this step. Guess too broad and you have quietly handed back the isolation you paid for in the first section. Guess too narrow and the run stops, or worse, carries on and delivers nothing, which is the pipeline again: a grant made in advance, too small, and not one part of the system able to say so out loud.

Roles are the cheapest place I have found to put this guess, because a grant attached to a role is made once for a kind of work rather than once per run, and somebody can read the list and disagree with it. It is still a guess. It is just a reviewable one, and a narrow guess is only safe if the previous section is already built, so that refusal comes back as an outcome you can see rather than as an empty result.

4. A transcript that survives the run

There is a difference worth being pedantic about. What a run says about itself is testimony. What the run actually did, the commands, the outputs, the diffs, the refusals, is evidence. You want the evidence, so the process can be scrutinised and challenged.

The cost: isolation says the box is disposable and should be thrown away the moment the task ends. Evidence says nothing readable inside that box may be lost. So you pay for the wall, and then you pay a second time to lift everything worth reading back over it before it goes.

5. Room to run more than one job at once

Running many jobs at once is not just parallelising the flow described up to here. It means creating admissions, claims and guard checks.

Admissions: something has to decide whether a job starts now, because capacity is finite: model quota, containers, machines, money. With no admission decision the answer is that everything starts at once, and everything starting at once behaves remarkably like nothing starting at all.

Claims: two workers must not build the same thing, so a piece of work has to be claimed before it is started, the claim has to be visible to every other worker, and it has to be released when the work ends or expire on its own if the worker disappears. Get this wrong and the second worker's output is not merely wasted. It conflicts with the first, and now somebody is reconciling two attempts at one job.

Guard checks: something watching if the system has stopped moving, checking nothing crashes when work is added or stops flowing.

What running many at once actually needs

Nothing throws an error when the queue stops draining. You have to go and ask.

Guard checks is the cheapest thing on this entire list and the easiest to skip. Work waiting and nothing running is one query away, it needs no new component. Write the query, run it on a schedule, treat the answer as an alarm.

Parallel work is also where the harness starts carrying weight. With one run, watched, you can read the transcript yourself and tell if work has been done by looking at it. With several, unwatched jobs, judgement can drift and the harness can become the bottleneck if it is not set up properly.

What I have not solved

The initial refinement: how can we tell if the system understands contextually what we have in mind? This is a solved problem for narrow scope and small tasks, but when the work becomes broader, quality takes a hit and the jobs start to drift. My next attempt would be to introduce cadence and work review in agile fashion, so the system is always honest and actual value is delivered and measured.

This is the second post in what I want to make a short series on building an agentic harness. Let me know if you have any more features that you think would be beneficial in a harness. Happy coding.

Originally published on juliantellez.com.

Agentic Harness: Why an Agent Needs a Harness

Julian Tellez — Fri, 23 Oct 2026 21:53:56 +0000

A model that can write anything still cannot open a file, run a command, or know when to stop pretending it did. This is the first post in a series on what actually turns a language model into something you can safely put to work.

I once asked a model to fix a failing test, expecting code back. What came back instead was a paragraph explaining, correctly, what was wrong. Nothing else happened. The bug was still there. The model thought it had done its job. At this point in time nobody had built the part that takes a correct paragraph and turns it into a saved file, a passing test, a done thing, a valuable output.

That gap has a name now. People call it an agentic harness: whatever sits between a model and the actual world, turning what the model says into things that happen, and reporting back what happened so the model can decide what to say next.

A language model on its own is a very good guesser of the next right word. It has no hands. It cannot open a file unless something opens the file for it, run a command unless something runs the command, or remember what it did five minutes ago unless something wrote that down and hands it back. Everything that makes an agent look like it is doing work, rather than describing work, is the harness. Not the model.

This has only become a real design problem recently. A model that just answers questions does not need much of a harness: a text box and a network connection will do. A model asked to write code, run it, read the failure, and try again needs somewhere to write the code, something to run it, a way to read the failure back, and a decision about how many tries it gets before a person is called in. Multiply that by more than one task running at once, or a task running while nobody is watching, and the harness stops being plumbing. It becomes the product.

There is not one shape this takes. So far there are at least five patterns at this point in time, and each is a different bet about who is watching and how much can go wrong before someone notices.

The simplest is the one most people already have: a terminal, an editor open beside it, an agent that reads and writes files and runs commands with your own permissions. It is the fastest thing to build and the fastest thing to use, because nothing stands between the model deciding to do something and the something happening. That speed is also the whole risk. The agent can do anything you can do, including the things you did not mean to ask for, and there is no wall between one task and the next. A mistake in one session has a blast radius in your actual home directory.

Local process

Fast because nothing stands in the way. Risky for the same reason.

Put a wall up and you get the next shape: each task, or each session, gets its own container or virtual machine, thrown away when the task ends. A mistake stays inside the box it happened in. You can run many of these at once, including ones nobody is watching in real time, because the damage any single one can do is small on purpose. The cost is that somebody has to build and run the wall. A fresh environment takes time the local version never spends. Credentials have to be handed into a place you cannot see directly. And the box needs its own way of telling you what happened inside it, because you are no longer sitting next to it while it works.

Sandboxed containers

A mistake stays inside the box it happened in.

A third shape hands the wall to somebody else entirely: a managed platform that runs the sandbox, the model calls, and the wiring between them, and gives you an interface to start a task and collect a result. You stop operating infrastructure, which is real time back. You also stop being able to see very much of what happened in between, which matters the first time a result comes back wrong and you want to know why.

Managed platform

You get a result back. You do not get to see how it got there.

A fourth shape is declarative: a graph of steps, some of them a model call and some of them a tool call, wired together ahead of time so the same automation runs the same way every time it fires. This is the shape that scales best for something you want running on a schedule or in response to an event, because nobody has to sit and prompt it each time. It is also the shape most likely to mark a step successful because nothing threw an error, not because anything was actually achieved, unless somebody deliberately built a check that looks past the model's own account of how it went.

Workflow graph

Nothing threw an error is not the same as something got done.

A fifth shape lives inside the tool you already write code in, a panel next to the file you are editing, one person and one task at a time. Nothing beats it for the tight loop of asking, watching, correcting. It was never built for the version of this where the task runs after you have gone home.

Editor embedded

Built for someone watching. Not built for after you have gone home.

None of the five are wrong. Each is a reasonable answer to a different question about who is around when something breaks. What I want out of any of them, after watching each fail in its own particular way, is a shorter list than you would think.

Real isolation, first. A mistake in one task should not be able to reach another task, or reach you. If a harness cannot say precisely what a given task is unable to touch, it does not have isolation. It has hope.

A visible difference between done and gave up, second, and this is the one I would not trade for any of the others.

The most dangerous failure is not a crash. It is a run that finishes clean and did nothing.

A model would rather answer than admit it cannot see what it needs to see, and a harness that lets a plausible non answer read as success is worse than one that has no safety features at all, because it hides exactly the runs that needed a person to look.

Permissions built for nobody watching, third. Anything that only works when a person is at the keyboard to click approve will eventually run unattended and fail there, silently, by design. A harness meant for background work needs a way to say yes in advance, scoped tightly, rather than a wall only a human hand can get past.

Credentials that reach the sandbox, not just the settings screen, fourth. Something that shows as set in a list but never actually arrives where the work happens is worse than an honest error, because it looks like it should have worked.

A transcript that survives the run, fifth. Whatever the model actually said and did needs to stay readable after the task finishes, not filed somewhere only a direct query against a database can reach. The summary is not the evidence. The transcript is.

And room to run more than one thing at once without losing any of the above, last. The entire point of handing work to an agent is not doing it yourself while it runs. A harness that only works one task at a time, watched, has not saved anyone anything.

This is the first post in what I want to make a short series on building an agentic harness. Let me know if you have any more features that you think would be beneficial in a harness. Happy coding.

Originally published on juliantellez.com.

Agentic Harness: What Spec Driven Development Actually Is

Julian Tellez — Fri, 23 Oct 2026 21:52:54 +0000

At the AI Engineer World's Fair in 2025, Sean Grove from OpenAI described how most of us work with coding agents: "We keep the generated code, and we delete the prompt." He compared it to shredding the source code and version controlling the binary.

A harness can check a great deal about an agent's output: whether it stayed in its box, whether it finished, what it did. None of that tells you whether the agent understood what was asked. When the work is small, understanding is cheap to confirm by reading the result. When the work gets broad, the jobs drift, and every check passes on the wrong thing.

Spec driven development is the industry's answer to that problem.

What it is

Spec driven development means you write down what you want before the agent writes any code, and that document becomes the source of truth for both of you. The agent plans from it, breaks it into tasks from it, and is checked against it.

The definition is still moving. Thoughtworks put it on their Technology Radar in November 2025 in the Assess ring and said plainly that "the term's definition is still evolving". Birgitta Böckeler, a distinguished engineer at Thoughtworks, wrote the most useful map of it so far on Martin Fowler's site, and she splits it into three levels:

Spec first. You write a careful spec, the agent builds from it, and the spec is done with when the task is.
Spec anchored. The spec is kept after the task and evolved alongside the feature, so the next change starts from it.
Spec as source. The spec is the thing you maintain. Humans edit only the spec, and the code is generated from it.

Almost every tool today does the first. Few commit to the second. Only one has seriously tried the third.

Three levels of commitment

Each level asks you to trust the spec more and the code less.

Why it is needed

Agents are good at writing code and bad at guessing intent. GitHub's announcement of Spec Kit says this directly: the problem is not the agent's coding ability, it is the approach. We treat agents like search engines when they behave like "literal minded pair programmers". Give one an ambiguous sentence and it will pick an interpretation, build it well, and explain convincingly why it is right.

Vibe coding works for a prototype. It fails on anything that has to live in an existing codebase, match an existing architecture, or still make sense in six months. The prompt that carried the intent is gone, the code is all that is left, and code is a lossy copy of intent. It tells you what the system does. It rarely tells you why, or what it was never supposed to do.

A spec is the attempt to keep the intent. Written down, reviewed by a person before any code exists, and kept somewhere the next agent can read it.

Where it comes from

None of this is new. What is new is who reads the spec.

Design by contract, from Bertrand Meyer and the Eiffel language in the 1980s. Preconditions, postconditions and invariants written next to the code, checked at run time.
Model driven architecture, from the Object Management Group in 2001. Draw a precise model, generate the code from it. It promised exactly what spec as source promises today, and it mostly did not stick, because the models were rigid and the generated code was painful to live with.
Behaviour driven development, from Dan North in 2006. Scenarios written as "given, when, then" in plain language that a test runner can execute. Most spec tools today write their acceptance criteria in this shape.
The Easy Approach to Requirements Syntax, usually called EARS, from Alistair Mavin and colleagues at Rolls-Royce in 2009. They built it while analysing airworthiness rules for a jet engine control system: a handful of sentence templates that remove most of the ambiguity from a requirement. Kiro uses it today.

Two EARS requirements, written for this post:

WHEN a user submits the signup form with an email that is already registered,
THE SYSTEM SHALL reject the request and show "this email already has an account".

IF the email service does not respond within 5 seconds,
THEN THE SYSTEM SHALL queue the welcome email and complete the signup.

Each line is one trigger and one behaviour, and each one is a test waiting to be written.

The difference this time is that the reader of the spec is a model. Model driven architecture needed a strict, parseable model because the generator was a compiler. An agent reads plain language, which removes the rigidity. It also brings back the uncertainty, because the same spec can produce different code on two runs.

Who is proposing it

GitHub, through Spec Kit, announced by Den Delimarsky in September 2025.
Amazon Web Services, through Kiro, an editor built around specs that launched in July 2025.
OpenAI, less as a product and more as a position. Grove's talk argued that "the new scarce skill is writing specifications that fully capture the intent and values", and pointed at OpenAI's own Model Spec, a versioned set of markdown files where every clause has an id and a set of prompts that act as its tests.
Tessl, founded by Guy Podjarny, which made the boldest bet of all: specs, not code, as the primary artifact.
The open source community, through OpenSpec and the BMAD Method, which between them have over a hundred thousand stars on GitHub.

And on the sceptical side, the same people who mapped it. Thoughtworks and Böckeler are the clearest critics, covered further down.

How people use it

Under the different names, almost every tool runs the same loop.

The loop most spec tools share

The approval happens on the spec, before any code exists.

Set the rules once. A file of things that are always true for the project: the language, the testing rules, what is never allowed. Spec Kit calls it a constitution.
Write the spec. What you are building and why, as user stories with acceptance criteria. Usually the agent drafts it from a short description and you correct it.
Plan. You give the technical direction. The agent writes the design: data model, interfaces, how it fits what exists.
Break it into tasks. Small, ordered, each one checkable on its own.
Build one task at a time, and review a small change rather than a large one.

The review moves. Instead of reading a thousand lines of generated code and asking whether it is right, you read a page of requirements and ask whether it is what you meant. That is a cheaper question to answer, and a much cheaper one to get wrong.

The five approaches worth knowing

Ranked by how much they are used and how often they appear in independent comparisons. Star counts were read on 22 October 2026.

1. Spec Kit, from GitHub

github.com/github/spec-kit, about 142,000 stars, MIT licensed.

The reference implementation. A command line tool installs templates and slash commands into whatever agent you use: Copilot, Claude Code, Gemini and others. The flow is constitution, specify, plan, tasks, implement, with clarify and analyse steps between them to catch gaps and contradictions. It is the most thorough of the five and the heaviest. The project now describes its steps as "independent entry points, not mandatory phases", a softer position than its first release.

Best for: new features of real size where you want a documented trail.

2. Kiro, from Amazon Web Services

kiro.dev, an editor rather than a toolkit.

Every spec is three files in .kiro/specs/: requirements.md, design.md and tasks.md. Requirements are written in EARS. When you run the tasks, Kiro builds a dependency graph and runs independent tasks in parallel waves. It has separate flows for features and for bug fixes, and a quick spec mode that skips the approval gates for work you already understand.

Best for: teams happy to move into a new editor in exchange for the most integrated experience.

3. OpenSpec

github.com/Fission-AI/OpenSpec, about 72,700 stars, MIT licensed.

Built around changes rather than whole systems. Each change gets a folder with a proposal, a design, a task list and spec deltas that describe what the change adds to the system. When the change ships, it is archived and its deltas are merged into the living specs in openspec/specs/. That makes it the tool that comes closest to spec anchored without asking you to describe your entire system first. It claims support for more than thirty assistants.

Best for: existing codebases, which is most codebases.

4. BMAD Method

github.com/bmad-code-org/BMAD-METHOD, about 54,400 stars.

The Breakthrough Method for Agile AI Driven Development. Where the others are workflows, BMAD is a team: separate agents carrying product, architecture, user experience, development and testing roles, each handing structured documents to the next. Small changes go straight to build. Larger ones go through the full planning cycle first.

Best for: people who want the agile ceremony modelled explicitly, and are willing to pay for it in process.

5. Tessl

tessl.io.

The only serious attempt at spec as source. In its framework, each spec mapped to one code file, and the generated file carried a header saying "GENERATED FROM SPEC - DO NOT EDIT". Thoughtworks described it as the approach where "the specification itself becomes the maintained artifact, rather than the code". It was in private beta through 2025. Today the home page describes Tessl as "the agent enablement platform", built around reusable skills and context rather than specs as source. Tessl has not published a reason for the change, but the change itself is a data point about how hard the third level is.

Best for: watching. It is where the idea goes if it works.

What the critics say

Böckeler tried the tools on real work.

Kiro turned a small bug into four user stories with sixteen acceptance criteria. She called it "like using a sledgehammer to crack a nut". Spec Kit produced so many repetitive markdown files that she concluded "I'd rather review code than all these markdown files". Agents still ignored the spec in places, and in one run regenerated classes that the research step had already documented as existing. A detailed spec gave a feeling of control the agent did not honour.

Thoughtworks went further and asked whether the field is relearning the bitter lesson: that handcrafting detailed rules for AI "ultimately doesn't scale". And Böckeler's sharpest point is the history one. Model driven development failed on rigidity. Language models remove the rigidity and add uncertainty. Spec as source risks getting both problems at once.

A spec only helps if somebody reads it, and only protects you if something checks the code against it.

Most of the criticism is really about those two failures. Specs too long to review, and specs that nothing enforces.

What to do with it right now

Use it for features, not for fixes. A bug with a clear reproduction does not need a requirements document. A feature that touches three parts of the system does. If you cannot say what a spec would catch, skip it.
Keep the spec short enough to review in one sitting. If you would not read it, neither will anyone else, and an unread spec is just a longer prompt. Böckeler's warning about "spec" becoming a synonym for "detailed prompt" is the real risk here.
Write every acceptance criterion so it can fail. EARS or "given, when, then", it does not matter which. "The signup works" cannot fail. "When the email is already registered, the system rejects the request" can, and it is a test before it is code.
Make the spec executable. The spec is a claim. The test is the evidence. Turn each criterion into a test before the implementation exists, and let the test runner decide, not the agent.
Aim for spec anchored, not spec as source. Keep the spec and update it with each change, the way OpenSpec does with its deltas. Do not try to stop editing code. Tessl's change of direction shows how hard that is.
Start with the lightest tool that fits. For an existing codebase, OpenSpec. For a new product with real scope, Spec Kit. If you already live in an editor you like, a folder of markdown files and a clear habit get you most of the way.

The open problem is drift. A spec written today and code changed tomorrow will disagree soon enough. Some of these tools have verify or analyse steps, but a June 2026 paper still names specs that silently fall out of sync with the code as a failure the existing approaches do not fully solve. Until something does, a spec can report success on work that no longer matches what anybody meant.

This is the third post in a short series on building an agentic harness. If you are using any of these tools on real work, I would genuinely like to hear where they held up and where they did not. Happy coding.

References

Birgitta Böckeler, Understanding Spec Driven Development: Kiro, spec-kit, and Tessl, martinfowler.com, 15 October 2025.
Thoughtworks Technology Radar, Spec driven development, November 2025.
Den Delimarsky, Spec driven development with AI: get started with a new open source toolkit, GitHub Blog, 2 September 2025.
Sean Grove, The New Code, AI Engineer World's Fair 2025.
GitHub Spec Kit.
Kiro specs documentation.
OpenSpec.
BMAD Method.
Tessl.
Easy Approach to Requirements Syntax, Alistair Mavin and colleagues, IEEE Requirements Engineering Conference, 2009.

Join the community

I run a free community for people moving from vibe coding to production: spec driven development with Claude Code, Cursor and agents, and the habits that make the output something you can ship. If this post was useful, the conversation carries on there.

Join Vibe Coding to Production on Skool

Originally published on juliantellez.com.

Handling complexity in lambda functions

Julian Tellez — Sun, 10 Feb 2019 01:32:54 +0000

See the original published version here.

TLDR;

Middlewares can handle the complexity of your lambdas while isolating business logic and cross-cutting concerns in reusable components that can be modelled by event cycles.

I have had the luck and pleasure of working with lambda functions recently and boy I am having fun! The idea of having an event-driven execution environment is both daring and exciting.

If you are new to the FaaS world, don't worry, the community has already prepared a curated list of reads for you here and here. Have a browse, drink some coffee. ☕

As you can see, (because you checked at least one of the links, didn't you?) A huge weight has been lifted. We can now develop, run and manage applications without the intricacy of building and maintaining infrastructure. But with new beginnings, new challenges arise. One of such problems and the reason for this blog is function complexity.

How complex is too complex?

Generally speaking, when writing lambda functions you would organise them into a series of dedicated modules following the single responsibility principle. If you would rather do otherwise, please read this excellent blog comparing monolithic vs single purposed functions. Isolating application from business logic reduces code complexity, simplifies reasoning and will help you debug your code more efficiently. Your colleagues and your future self will thank you.

Take for example some sort of CRUD API, we would like to make sure that a payload is properly formatted before handling any data. The payload validation could emit an error that would communicate its status to the consumer. If the validation was successful it would then move onto executing some business logic. For both scenarios, we would also want to set up some percentile tracing and error logging. While all of these sound great, we have essentially mapped out a request lifecycle with various cross-cutting concerns:

Data validation.
Error detection and correction.
Logging.
Monitoring.

I know what you are thinking… (gif from giphy)

Following the serverless mantra, we should outsource anything that isn't related to the core business, so wouldn't it be great if we could just focus on the business logic and outsource these application-specific problems too?
Of course, this isn't a new concept and we have been using it in web frameworks like hapijs and express for quite some time now.

Lambda Middleware

A lambda middleware is essentially a function that contributes to managing cross-cutting concerns in a consistent manner. It provides a clear separation of application and business logic and outlines an easy to reason configuration cycle. It is the glue that when adopted by everyone in a team becomes the lingua franca that supports and creates features. But do we really need a middleware? Well, look at the current state of your lambdas and ask yourself a few questions:

What does the code look like?
Can you envision anyone supporting it in at least 2 years?
How much business logic expertise do you need? (if specialised then separation is imperative)
Is logic intertwined and hard to reason about?
Does the project have a clear path or do you see it going in different directions?
How will the project look like the day the expert leaves?
Are best-practices enforced and maintained?

Jack Nicholson approves. (gif from giphy)

Enter Lambcycle

Lambcycle 🐑 🛵 is a declarative middleware for lambda functions. It defines a configurable event cycle and allows you to focus on your application's logic. It has a "Feature as Plugin" approach, so you can easily create your own plugins or reuse your favourite packages with very little effort.

In practice there is no such thing as an un-opinionated framework, every idea is an opinion and a way of working with your application. Lambcycle favours configuration over the code and believes that business logic should be as isolated as possible.

Labmcycle flow chart.

Lambcycle enhances lambda functions with a few extension points (see graph), each of which can be used to interact with the event in a decomposed manner.

The first extension point is Request which occurs immediately after the lambda is called. You can use this step for parsing, validation, etc...
Note: If you are thinking of auth, please consider a lambda authoriser instead.
The Pre Handler extension comes in handy when you need to adapt data to fit an interface. It is also a great place for fetching secrets.
The Handler, where your beautiful business logic lives.
Next up is the Post Handler, use this extension to validate and/or cache the output of your business logic.
Error is an implicit extension for logging and tracing.
And finally Pre Response, your chance to format a response to the consumer (which could be data or an error).

Using the middleware

import Joi from 'joi'
import lambcycle from 'lambcycle'
import joiPlugin from 'lambcycle/dist/plugin-joi'
import bodyParser from 'lambcycle/dist/plugin-body-parser'
import realTimeEventTracker from './bugFreeBizLogic'

const businessLogic = async(event, context) => {
    const {error, data} = await realTimeEventTracker(event.data)

    if(error) {
        throw error
    }

    return data;
};

const schema = Joi.object().keys({
    assetId: Joi.string().required(),
    competition: Joi.string().required(),
    sport: Joi.string(),
    tournamentCalendar: Joi.date(),
}).required()

const handler = lambcycle(businessLogic)
.register([
    bodyParser({type: 'json'}),
    joiPlugin(schema)
])

export default handler

The example above leverages express' body parser and hapi's joi validation. We have configured both plugins with a parsing type and a validation schema respectively. By abstracting these two we have rid of complexity and shift the focus to the business logic. Note that a more robust lambda would include error tracing, and response plugins.

Crafting a Plugin

The possibilities are endless when it comes to plugins! Do you have something in mind? Contributions are more than welcome ❤️

import * as Sentry from '@sentry/node';
import MyAwesomeIntegration from './MyAwesomeIntegration'

const sentryPlugin = (config) => {
    Sentry.init({
        dsn: `https://${config.key}@sentry.io/${config.project}`,
        integrations: [new MyAwesomeIntegration()]
    });

    return {
        config,
        plugin: {
            onPreResponse: async (handlerWrapper, config) => {
                Sentry.captureMessage('some percentile log perhaps?')
            },
            onError: async (handlerWrapper, config) => {
                Sentry.captureException(handlerWrapper.error);
            }
        }
    }
}

export default sentryPlugin;

In your lambda ....

import lambcycle from 'lambcycle'
import sentryPlugin from './sentryPlugin'

const myApplicationLogic = async (event, context) => {
    return await someLogic()
}

const handler = lambcycle(myApplicationLogic)
.register([
    sentryPlugin({
        key: process.env.SENTRY_KEY,
        project: process.env.SENTRY_PROJECT,
    })
]);

export default handler;

DX

Lambcycle has been built with developer experience "DX" in mind and ships with type definitions, for consistency and auto-completion 🚀 (VScode only).

Lambcycle DX.

Conclusion

It is a brave new world and serverless is here to stay! Promoting reusable components and consistent error handling will help you and your team create and support features in an more controlled and organised fashion. Lambcycle's main goals are to shift the conversation around a predictable unidirectional cycle and encouraging component reusability across features. Its worth mentioning that for more complex scenarios you should be looking at Step functions. Let me know if you would like to know more about them in the comments, in the meantime check out these use cases!

Special thanks to the legendary burning monk for the in-depth review of this article.