Six days of hand-built core, then the software factory took over, and now 40% of its PRs getting one-shotted

Lezgin Zilan — Mon, 26 Oct 2026 07:54:56 +0000

Have you ever looked at a backlog full of small fixes that everyone agrees should be done and nobody has time for? We do, and now the software factory is taking a bite on them

We're a Stockholm fintech with ~40 engineers powering recurring payments across Europe, running on a AWS stack using Kubernetes, serverless, and Python (alongside some legacy PHP). Our developers spend their time on the deliveries that matter, like new markets, complex features and regulatory work. The low-hanging fruit kept lying around in the backlog, and every week someone asked when it would get picked up and by next week there is even more getting added.

So, we built a software factory. Software what? Well, you describe a piece of work in a Jira ticket or in Slack, a coding agent does the work in our codebase in a cloud environment, and you get a pull request.

The part we built by hand

The first version was a chicken-and-egg problem. Someone had to write the thing that turns a ticket into an agent run, and the environment the agent runs in, before there was anything to hand work to. Two engineers did that in about six days, starting cloud agents by hand until the factory could start them itself.

The core is two repositories. An orchestrator receives webhooks from Jira and Slack, compiles the prompt and hands out short-lived credentials. A cloud agent environment holds a catalog of every Billogram repository, our build tooling and a set of skills describing how we work. Each run follows the repository's AGENTS.md.

Then it wrote most of the rest

When we say it builds itself, a human still outlines the goals and refines the ticket, but our factory's retro pass also generates its own follow-up tickets to keep improving. The factory writes the code and opens the PR. A person reviews and merges. But, after each run it does a reflection / retro on the delivery to understand what went well and less well and also creates improvement tickets.

Measured on the lines still in the code today, 89% of the orchestrator and 97% of the agent environment are factory-written. Of 639 merged PRs across the two repositories, 596 were opened by the factory.

The pattern we ended up with is that a person lays down the skeleton of a new capability, and then the factory takes over and iterates.

Jira intake was built entirely by the factory, in a single ticket, about 6,000 lines. We started on Linear on purpose, so nobody would accidentally kick off agents from a real Jira board while we were learning. After the switch, the factory removed Linear from both repositories itself.

Mission Control, the web page listing every run, its source and its PRs, had a human first version. The factory shipped about 30 PRs on top of it: an activity log of each agent's tool calls, starting agents from a prompt box, repo filters, mobile layout and dark mode.

Claude Code (self hosted runners) now run alongside Cursor Cloud agents. Each session gets its own on-demand Lambda microVM in our AWS account, and the person who started the task can follow and steer it live from the Claude app if needed.

Auto-docs write down what we teach. When a PR merges, an agent turns the reviewers' comments into guidance in the repository's AGENTS.md, so the next run doesn't repeat the mistake.

It even handles its own support. Questions and error reports about the factory go into its own Slack channel. Most of them end up as tickets handed to an agent, so by now the factory handles a fair share of its own support.

The promo video below is one-shotted. One prompt with a few Slack screenshots went into a coding agent, and the agent made the video by itself.

The numbers

From 31 August to 22 October, the factory opened about 1,100 PRs. Of those, about 40% were one-shotted, meaning the factory was spot on. About 50% needed some rework, but were good enough to build on. The reviewer tagged the factory in the PR with feedback, and it pushed the fix itself. The remaining 10% were to far off and closed without merging, most often because of a poor use case or too little prework on the ticket.

Other fun figures:

First days: about 30 tickets per day, mostly the factory's own backlog.
Busiest week so far, starting 28 September: 245 sessions and 219 merged PRs.
No single vendor: ~53% sessions ran on Cursor Cloud, ~47% on Claude Code, even though Claude Code was only almost three weeks late into the game.

Example use case - One upgrade, three versions, one missing cent

One of our oldest services and part of our core needed to move to a new major version of PHP. That kind of upgrade is complex, and in a system that moves money it has to be done carefully and with a lot of testing.

We wanted to see how much of that care the factory could take on by itself. So, we gave our software factory one sentence: upgrade the service to the latest language version and a current OS base image, and fix whatever needs fixing. It ran for a few hours.

So what did it actually do? The software factory kicked off a bunch of agents and pulled down the C source code of the different language versions, then it lined that up against our codebase to understand what functionality we use and what could potentially impact our functionality. the implementation of rounding had changed in the newest release.

Then it proved it with a test. It rounded 126.045 to two decimals on every version. Ours returned 126.05, and so did every newer version except the latest, which returned 126.04

One cent sounds like nothing until you multiply it by millions of monthly transactions and 5B SEK a year, all of it other people's money. We were completely surprised by the route the agent took. None of us would likely have thought to read the interpreter's C source! The agent did it unprompted, and found the one cent that mattered.

Example use case - 85 days in the backlog, ten minutes to two PRs

This one had sat untouched in the backlog for 85 days. It was a well-defined improvement with a good ticket description, just never urgent enough to get picked up. We handed it to the factory, and ten minutes later two PRs were up, one in the backend and one in the frontend service: 440 lines including tests. Part of the logic didn't exist in the backend at all, so the agent wrote it from scratch.

Human time came to about three hours in total. That covered verifying the behaviour before and after the change, a review round with two findings that the agent fixed in 12 and 7 minutes, and the release. Our estimate for the same work is one to two days without AI, four to eight hours for a developer with AI in the IDE, and two to three hours with the factory. Validation, review and deploy take at least two hours whichever way you go. The factory removes everything else.

What didn't work

Review is the hard part and let’s look at another example, the agent built a monitoring dashboard for the factory as four PRs across different repositories, and reviewing 2,000 lines properly can take about as long as writing them yourself.

Every PR carries a Decisions and assumptions section, so the reviewer sees what the agent chose on its own. The agent also runs a cleanup pass on its own diff before it opens the PR. We haven't solved this yet, but it's getting better.

When it comes to cases where we actively decide for the agents to take bigger tasks and the PRs would be big regardless, we use Githubs stacked PRs to break them down to smaller dependent PRs that can be reviewed independently.

Agent speed is not the same as lead time. The 85-day ticket above still took 22 calendar days to reach production. We ran it as a trial without the owning team directly involved: developer from outside the team picked it up. The way of working wasn't clear, but the expectation was still that the team would review it, and quickly. The team's developers were focused on their own commitments, not on a PR from outside the team, and the concept was new. It was one of the first well-specced tickets we tried. The lesson is that the factory changes the way of working too. A developer within the domain must own the outcome from start to end, and the team has to reserve capacity for review.

What this means if you're a developer

It follows our normal processes but encourages a shift from "vibe coding" to spec-driven development. It only ever makes changes through a pull request. It opens drafts, gets reviewed by another ai agent and by the reporter, answers review comments and pushes fixes. It never merges on its own. Infrastructure changes are PRs to our infrastructure repository that the infra team plans and applies.

What changes is who can start the work, but not who owns it. Non-technical colleagues have already passed questions straight to the agent in Slack, with a developer reviewing what came back. For product work, a PM refines the ticket together with an engineer before the agent picks it up. We involve developers early because we want their system knowledge, technical direction and read on feasibility, not just their sign-off at the end. The pattern is to align up front on the spec and approach, so the review afterwards only checks the result against what was agreed. A developer shouldn't get a finished PR out of nowhere and have to work out what it is and whether it's right. A small, early time investment saves the expensive review later. That's how the small fixes the business has been waiting for get picked up without pulling developers out of the delivery they're focused on.

Where we're heading

We want to automate a bigger portion of the flow than we do today. To be realistic about where we are, the factory is not a drastic change yet. A developer with a well-specced ticket could already paste it into Cursor or Claude Code and get a PR back. What the factory changes is the way of working. It doesn't let anyone shortcut the spec, and developers can expect someone else to write it instead of having to drive that part themselves.

So, next up is giving the agent access to QA environments so it can verify against a deployed service and read the signals from how it behaves under test. After that it follows our normal policies and processes for review and release, since we are in a regulated industry with multiple audits.

The dream state is spec-driven development, where the ticket is the spec and what we write down decides what comes back. Stakeholders would iterate on ideas and implementations themselves, and well-defined, low-risk changes would be handled by them end to end, with high confidence with guardrails. We're not there yet, and we would rather be realistic than live in a dream.

If you're building something similar preferably in a regulated industry, we like to hear how you but also practical solutions for reviewing large agent-authored PRs, since that is the problem we're still working on, reach out.

And to be clear, we are not replacing our developers with AI. The factory takes the low-hanging fruit off their plate, and it needs good developers to spec, steer, review but also code by hand. We're hiring, so if you want to be part of the journey of building a platform that moves millions of monthly transactions and 5B SEK a year, like to build systems and know python or can easily learn a new language, take a look at career.billogram.com.

DEV Community: Billogram