DEV Community: Daily Context

Warp's Ben Holmes on Building the Software Factory

Kara Silverman — Wed, 14 Oct 2026 12:13:55 +0000

Ben Holmes is the DevEx Engineering Lead at Warp. The company has spent the year shifting from the terminal that made its name to Warp Factories – open infrastructure for teams building software factories. He's also known for his whiteboard explainers, which turn complicated systems into short, drawn diagrams.

Ahead of his AIE New York talk, "Your Agents Need a Skill to Improve Their Skills," he sat down with The Daily Context to talk about factory engineering, why Warp built its pivot in public, and how product development teams still aren't ready to hand it to an agent.

This interview has been edited for length and clarity.

The Daily Context: Tell us a little about yourself and your role at Warp.

Ben Holmes: I'm the DevEx engineering lead at Warp, so I work on everything that helps people work with the product. That means working with both our open source community and our growing community of teams using Warp Factories. We've made a pretty big shift. The terminal is what defined Warp up until now, but Warp Factories is what defines Warp today. Warp Factories is open, flexible infrastructure we've set up to help teams build software factories. We've built it as much in the open as we can, sharing our learnings along the way.

People are trying to automate development. They're trying to take these interactive agents that they have to hand-hold through every PR and ask: can we automatically trigger an agent when errors happen? Can we work multiplayer in Slack and communicate that way? Can we trigger it from task boards? In all these places, we want to give agents more control over creating everything.

We also want to create the system that actually lets us deploy all these agents. That system should give us tools to measure whether those agents are performing well, and give the agents tools to review their own work and suggest ways to improve the system. That's why our CEO, Zach Lloyd, uses the term "factory engineering." It means doing high-level engineering on the system the agents are working inside of, and less "prompt, prompt, prompt." Let's look at the skill makeup. Let's look at how we define each agent if we're orchestrating multiple agents together. Let's set up the automation chain: where do we trust agents to just do something without us prompting them? That's the high-level work engineers are really starting to focus on now.

The Daily Context: What is your personal relationship to AI?

Ben Holmes: Right now, I see AI as a super capable partner that can proactively figure out how to do a task, and more importantly, what tasks need to be done. That second part has been really interesting to me, maybe more so than just asking agents to go do things. Working in a fast-moving startup, it's hard to tell: did this thing fall off my calendar? Is this something we should be focusing on right now?

In software development, we're slowly getting there. We have agents that look at Sentry alerts and Grafana alerts and figure out which bugs you should actually focus on. That's one example. There are also agents that watch my conversations and say, "This conversation you had in the past went really poorly. You should probably update your skills to make it better next time." The agents that are watching how everything is going are maybe the more interesting ones to me.

So, back to the relationship question: I see them as highly capable tools. More and more, though, I see them as something that can be proactive and help me figure out what I should be focusing on and doing. Of course, I've been trying out crazy scheduled agents that review my Gmail and do all the personal assistant type stuff. We're also trying to figure out what that means for a whole team. We have a virtual assistant bot right now. We have a thread where we do our standup and share what we did yesterday and what we're doing today. The bot reviews all of it and puts it in a Notion document so we're all on the same page. It can also be proactive and ask, "Did you get this done? Did you get that done?" They're starting to become proactive tools that you don't have to poke anymore. They're just constantly working for you, and that feels like an experience I want to craft.

My last talk at AI Engineer was actually about second brains: something that can look through a bunch of scattered dots and figure out which of them are the gems worth a second look.

The Daily Context: Your bio notes your whiteboard explanations. As teams adopt something like Warp Factories, how do you think about helping people learn new skills in the AI era?

Ben Holmes: I think if you can distill something down into a really short video or a really small diagram, then you really understand it. If you can't do that, you don't understand it super well yet. So I almost take it as a challenge to make sure I really understand something, and that usually leads to a better explanation.

I've done whiteboard shorts for a long time, and those aren't long-form videos. For a while, YouTube Shorts had a one-minute limit, so I had to figure out how to explain a front-end JavaScript framework, which is what the shorts were about at first, in one minute. The whiteboard was the perfect medium for that. I set the limit that I couldn't use overlays on the screen. It had to be a drawn diagram. I could use some stop motion to explain it, but I forced the medium as a way to be more creative and to make sure I was explaining something super clearly.

Now I've been using virtual whiteboards a lot more, like tldraw and Excalidraw, which have become very popular throughout engineering. It's also how I give all my talks now. I just open tldraw, show a diagram, and slowly walk toward the full explanation. That feels like a better way to tell a story than narrating your way through everything, especially now that we're thinking in loops and deployment patterns. Those concepts naturally need a loop or a graph drawn out before people can understand them.

The Daily Context: Warp made a big pivot this year and did a lot of it in public. Why make that process visible and open source?

Ben Holmes: The mission has stayed the same for us: help developers ship better software. The terminal was a vessel for that, and it made a lot of sense. Then, by following the community and seeing what people were working on, we saw that these automations around agents were clearly the next step in that journey.

Honestly, we didn't know what that would look like. We were watching how everyone was using these tools. As an experiment, we first open sourced the terminal. Second, we started building a system to maintain it, because open source is especially tough now with the amount of contributions coming in. If automation is really where we're heading, with agents proactively working on things, we need to build a system around that, even if we don't yet have a product we can help people with. We just needed to figure it out.

So we started building that system to manage the Warp terminal open source project, and before that, to manage our own work. On DevEx, I put out some resources on it and supported Zach Lloyd's articles and explainers, which showed early ideas using GitHub Actions. You can have an agent that triages every issue, an agent that does automatic code review passes, and an agent that implements with computer use. We were stitching those patterns together in an ad hoc way. Then we realized we had a system we could reuse and show to teams over and over again. We could distill it into something where you just sign up and get the triage agent and the review agent we found success with, plus a cloud runner with computer use, because the agent needs to verify its work. That became so obvious to us.

So we were dogfooding our own approach and sharing it as much as possible. Once we saw people really responding to it, we distilled it into something that helps other people do the same.

The Daily Context: Has the community used it in ways you didn't expect?

Ben Holmes: It's pretty early. Our actual factory setup has been around for a month and a half, and we have some early partners working on it. I don't think there have been any surprises yet, maybe just some early success stories. If there's one early surprise, it's that one company is using it for in-depth Android development. Seeing the amount of compute involved, we realized we really need to get a handle on driving down the cost per PR now that other people are using it. That's the first surprise to address. A lot of work still needs to happen, both on researching agent makeup and on improving our platform, so that if you want to run an agent automatically, it runs at a cost and speed that makes sense for teams. Maybe that means open-weight models. Maybe it means simplifying how many subagents you have. We're doing all sorts of research on this right now. That's been the main thing: we're stress testing it.

The Daily Context: What is your AIE talk about?

Ben Holmes: It's about self-improvement, which I mentioned earlier. It's titled "Your Agents Need a Skill to Improve Their Skills."

The Daily Context: Can you give an example of how agents improving themselves has played out in your own work?

Ben Holmes: First, I can explain the system briefly. The idea is scoring them on criteria that you design. It's kind of LLM-as-a-judge. Efficiency during a conversation is one example. Beyond skill improvements, we've been trying to expand this approach to also suggest architecture improvements, so you can have an even better codebase setup.

The Daily Context: How are you making self-improving agents easier for teams to see and work with?

Ben Holmes: We're coming up with a few ideas to make self-improvement more accessible and make it feel more like something you're optimizing. We have graphs that show all the scores and how they track over time, so you can see something like "your efficiency has gone up 10%." We're also playing with a blueprint idea, where you define all the agents currently deployed in your company and make that a little more visual. One of those agents might be the self-improvement agent, watching over the implementation agent. Making it visual makes it clearer that you're building a system and that the agents can all talk to each other. It's not a direct learning from Skill Doctor, but it's an idea I'm bringing in. We could make this more visual and make it feel like something you're actively building, instead of just looking at YAML files or a list of open PRs.

The Daily Context: Warp Factories lets teams run agent workflows from triage to verification. Is there a step teams don't want to hand over to an agent?

Ben Holmes: That's a good question, and you might expect what I'd say. I don't think any of them are really steps people don't want to give over. Triage is kind of a half answer, and I think it's something we need to do better. The idea of triage is that an agent goes off and researches something automatically when you suggest a feature idea, either in a task board ticket or in a chat. We're still figuring out whether that's a necessary step. Spec writing is another one. We have a spec writing agent, but we don't know if it should be separate, and we don't know how useful it is to have that speed bump of "Do you want to work on a spec together?" We may be at a point where it's more natural for a human to ask that. So we're thinking about removing, or at least optimizing, these speed bumps we put early in the design process.

Maybe the better question is about the types of tasks people don't want to hand over, which is kind of the same thing. I think we're still trying to figure out how to close the loop on product development: figuring out the next thing to build. That's still mostly humans saying, "Here's the next feature we should put in the product." This is kind of a blind spot for us too. It would be a nice world if an agent looked at usage data and figured out what your next feature or A/B test might be. That area feels underexplored. It also feels like something people don't really want to hand away yet, because that's product vision and product decisions, and that's kind of the last stand of human touch. We don't want to fully give it away, although if you point an agent at customer data, it might figure out some smart features worth trying. We haven't experimented with it yet. It just feels like an inevitable thing we'll need to reckon with. Closing the loop is something we talk about a lot, and this feels like an important piece. We've only closed the loop with bug fixes. We haven't figured it out for feature building.

A Data Company in an AI World

Kara Silverman — Wed, 14 Oct 2026 12:05:44 +0000

Anekha Sokhal spent nearly a decade trading energy derivatives at JP Morgan, then came to the US on a Fulbright to study data science at Rice University. Today she's the founder and CEO of Moshi, a Houston-based, VC-backed real-time quant platform for institutional finance. Ahead of her talk at AIE New York, we talked about why she calls Moshi a data company, what it's like to build in Houston, and the growing gap between how easy it is to build with AI and how hard it is to run what you've built.

Moshi, Anekha, and building in Houston

Tell me about Moshi.

Moshi is a real-time quant for institutional finance. We provide real-time analytics for financial traders. Every time you open Moshi, you're going to learn something new because of the volume and quality of data that we watch.

A fun fact about Moshi: it takes a human four seconds to read a sentence. In that same amount of time, Moshi has learned 300 new things. We watch more than 2.5 billion data points, and we grow our internal database every minute.

The value we add is that you're going to learn something new about your market every time you look at Moshi, and you're going to be so much more efficient. The problem traders have is that they're trying to keep up with everything, and everything is changing constantly. When you're a financial investor or trader, what you knew this morning is no longer true by lunchtime. The news is changing, Twitter is changing, Instagram is changing. People are flooded with data. And the problem with that is: What is true? What is relevant to me? What can I trust? That's where we come in. We are that one-source platform.

Typically, you would hire a team of analysts to do that for you. I used to be a trader at JP Morgan, and I could never even hire an intern, so I was doing all of this myself. So I built the platform I wish I had when I was trading. Because I'm also a data scientist, a lot of machine learning and statistics has gone into it. We've built all of our own models.

And tell me about yourself.

I'm Anekha, the founder and CEO of Moshi. I've grown Moshi myself. We're a VC-backed startup based in Houston, and the reason we're based in Houston is that I used to be a trader at JP Morgan. I was there for nearly a decade, trading energy derivatives.

I really just love data. I'm obsessed with data. I got a Fulbright scholarship (England has one Fulbright scholarship for data analytics) and came to Rice University to do my master's in data science. I got really into machine learning there and was published at conferences like ICLR.

When I was interviewing for jobs, a bunch of hedge funds reached out, and they all had this same problem of data synthesis. Instead of building it for one hedge fund and doing very similar work to what I did before my master's, I thought: I'm in the US. There's a huge venture industry. It's such an entrepreneurial country. That's why I came to the US. I really wanted to be part of that ecosystem.

I did an accelerator program while I was at university, then happened to meet a venture capital fund that loved the idea of building Moshi for everyone. And that's where we are today. My story is very intertwined with the company's story.

That sounds like a wild ride. What is it like to build in Houston?

The straightforward answer is that you get to be a big fish in a small pond. But I do feel like you have to keep your connections with other cities alive.

We're backed by one of the biggest venture capital funds in Texas. If someone is coming to Houston and wants to know which AI companies or finance companies are there, we're top of mind for everyone, which is great.

However, we can't ignore that, at least in our situation, AI and finance have other cities that are very important. San Francisco is one. New York is one. London is huge in finance, and so is Singapore. Houston is a very good city, especially for energy. I used to be an energy trader, and a lot of energy clients are here, so it's great in that way. But you also need to be connected to the other cities, because the hype is happening there too. San Francisco for the venture environment and the people. Austin has a great ecosystem. London has a fantastic finance community. And New York, I feel like, is coming up a little more in startups now, and it's obviously huge in finance.

So what is it like building here? There's a fantastic community. Some of the nicest people I've ever met in my life are here, so many helpful people. You can really make a difference here. But there aren't always going to be as many events, and it's not as easy to meet people because it's a very spread-out city. You don't have those shoulder-to-shoulder conversations you'd have in a walking city.

A data company in an AI world

So, what is your personal relationship to AI?

For my undergrad, which was a very long time ago now, I studied MORS (mathematics, operational research, statistics and economics) at the University of Warwick, and it was not a sexy degree. If I told anyone I was studying MORS at Warwick, they were just like, "Oh my God."

Because I loved statistics and data, I went on to trading. It was a very natural transition. Trading is just data and machine learning. And when I was doing it, no one cared. No one thought what I was doing was interesting or sexy.

I remember once meeting a family friend with my best friend, who was studying law. I'd gotten my internship at JP Morgan at the time. The family friend asked me, "What about you? Do you want to be a tutor or a teacher?" Then I looked at my friend and said, "Wow, you're incredible. You're going to be a lawyer."

It's so different now. I graduated in 2014, and now if you say "machine learning" to people, you're blowing their minds.

So my connection is that I was in it before it was cool to be in it. Now we have this separation between machine learning and generative AI. I call Moshi a data company because I love data. We have data scientists. I'm obsessed with data. One thing we do that's very different from most companies is that we do so much data engineering and machine learning, and then generative AI sits on top of all of that. So my relationship is: I want to understand the data, and generative AI is how we communicate it to people in a more human way.

The bar for calling a company an AI company has gotten way higher. It's not a differentiator anymore.

I finished work super late last night, and to decompress I went on Instagram, because it's the most mindless scrolling. Embarrassingly, I was on it for 40 minutes, until I got the timer notification. This was at 1 a.m. In that time, I got 25 ads for AI.

I screenshotted every single one and sent them to Rick, my head of growth, and said, "Rick, we cannot advertise on Instagram." Twenty-five companies in 40 minutes calling themselves AI, AI for finance. It's so easy to generate. How can I trust them? I think that says everything about what it's like to build today.

I'll never forget driving in from the San Francisco airport earlier this year. There was nothing but AI billboards.

There was a time when our investor told us we needed a .ai domain. I don't really believe in all of that, but we got the .ai domain, and now I don't even call us an AI company. I say we're providing a solution, and part of it does have generative AI, as you'd expect. That's an expectation now. If you have an LLM, it's not cool. It's the expectation.

It's the same in media relations. Five years ago AI was unique. Now the best way to get a company coverage is to not talk about its AI.

It's definitely different. Before Moshi, while I was at university, I tried to start another company, which I closed down, called Jewel Vision. My parents are in the jewelry business, and I wanted to give people the ability to try on jewelry in 3D without even needing models.

It was too early. The technology didn't exist. It was 3D computer vision. To this day, I have recruiters asking, "Can you come to Apple's recruiting events? Can you come to Tesla's?" because it's not a skill people have. I tried to build a company doing it for a year.

That was super cool AI. That is pure 3D computer vision. But if you have an LLM agent, to me, that is not AI. There's a difference.

Prompt fatigue and where humans come in

Where are you seeing prompt fatigue show up?

I want to explore how our work is changing and where humans actually play a part. One thing I've found is that people are now just copy-pasting Claude output to each other. So where does that stop?

If I think about the whole workflow of where we are today: people want solutions that make them more efficient, make them better decision makers and integrate their workflows, and they're using generative tools to do that. The questions that come up a lot for us are: How can I trust the output? How can I connect everything? And what about the data? Where are you getting the data from?

The biggest thing we've realized is that the foundational companies are making it feel like it's so easy to build. And it is easy to build. You can get a $200 subscription to any of them and build a dashboard for yourself, and it will be fantastic. I do it for my wedding planning. I have my little dashboard. Now try to make that production-ready. Try to make it run for more than one person. If I want to send that dashboard to my mom, who doesn't have a Claude account, that's the problem I want to talk about.

What happens when you try to take that into an enterprise?

When you try to implement that in an enterprise, you actually need to run the system. And when you need to run the system, you need DevOps, you need error handling, you need to pay other people to run it. That's the bit no LLM is going to tell you upfront.

So yes, you can spend weeks building a dashboard on Claude. Now try to make that dashboard available to the public. Now you need DDoS protection when you host it on a website. Now you have to make sure your API keys are hidden and not in the code. You have to make sure you're not showing your code on the website. You have to get a database. You can't just host it locally. Those are the things that end up wasting people's time.

Instead of managing it yourself, giving that job to a human, or even doing it the old-fashioned way, you've built it yourself on Claude. But now you have to run it. So you're dealing with prompt management, because every time you want to make a change, you wanted to change one thing and you've gone and changed 50 other things. That's what I wanted to talk about: the prompt fatigue, and the realization that I can build really easily, but I cannot run a system as easily.

So where do people come into all of that?

I think companies are still figuring that out. CEOs and CTOs are constantly posting about how, oh, let's fire everyone, because now we only need two or three people to do that job. And I'm sure you've seen the AWS outages and all these other outages. This never used to happen.

What we're seeing now is a confidence everyone has been given by these AI tools: I can build anything I want. But the thing people are forgetting is that every single person has access to the same tool. Now you're going to tell me, "But I've got a different skills.md file, or a different Claude connector, or a different MCP." Yes, I agree, those are different. But you are still pulling and pushing the same data. It's still the same model. If my engineer asks Claude a question, they're going to get the exact same answer, even if the prompt changes a little.

So where do humans come into this? It's the judgment piece. Claude can't build Moshi. Why? Because I had to take 10 years of derivatives trading, and I'm a data scientist as well, so I knew how to build the system. I had people sitting with me when I was 19 or 20 years old at JP Morgan, spending three hours explaining systems to me. I took all of that, and that's what I'm bringing to my agents. A lot of people are not doing that.

I think one thing we're going to come across is that things are going to break more easily, because of the trust people have in these systems. One tool says, give it all your bank details, your credit card details and your email details, and it will go and run everything for you. We don't do any of that internally, and we have engineers from Microsoft and Meta. No one on our team does that. You can ask them to, and they won't. It's similar to the CEOs of all these tech companies who won't let their kids use social media.

I think there's going to be a sort of awakening. We have big enterprise clients who come to us and say, "Why do I need you? I've got an Anthropic subscription." And I say, try to run that same thing for more than one person. You're going to have security issues. You're not going to want to manage it. You're supposed to be trading and watching the markets. You're not supposed to be telling Claude that FOMC is today, not tomorrow.

There's going to be friction for a while. It's like a purge. People will do all the AI, and then they're going to realize: actually, I want to see a spreadsheet that I have to use my brain to check. I want to remember someone's phone number. I want to see a PDF that isn't perfect. I think you just have to let it happen.

The interesting thing for us is that enterprise software is changing. We have clients sending us their prompts. We had a client today who said, "Hey, can you just run this for us?" And I said, that's not what we do. And they said, "Yeah, but I don't want to run it anymore. You guys do a better job. You do all the trust, you do all the things." People are interacting with us like that now.

Did you do it?

We had to build that, because we didn't know we needed to before.

For us, having been a trader for so long, the human brain is so much more advanced than machine learning. The number of connections we can make in real time, the power of the human brain: there is no data analytics that powerful. We're trying to get closer and closer to that human way of thinking, but machines are not there yet. So why would you want a machine to make such important decisions?

Claude is an incredible tool, but it's incredible at summarizing what you tell it to do and explaining the prompt. What it's not great at is making those second- and third-order connections, which is what humans are great at. It's seeing around the corner, being able to say, "This is what I think I should do," or "This is what I don't see." Those gut feelings, those human interactions.

Can you give an example?

A simple one is hiring. So many times LinkedIn will ask, "Do you want us to AI-interview this person?" And I'm like, no. I want to interview this person.

When I was hiring a couple of weeks ago, I ran a simple experiment. I ran every résumé through Claude, because I wanted to see who Claude would hire versus who I would hire. It was so different. Claude came up with a scoring system and said: based on these people's answers, how they use AI and everything else, these are the numbers one, two and three you should hire. I thought, okay, that's really interesting.

Then I brought everyone to the office. One thing I do is spend three to four hours with each person, because I want to see how you think. Each candidate met me, our team members and our investor. Every person agreed on the ranking, and it was not the same as Claude's. The person Claude ranked first wasn't even in our top five.

There's a company that raised a ridiculous amount of money, something like $30 million, and they're a one-person company. Their whole thing is, "We are a one-person company." On social media they're doing incredibly. But you look at the reviews, and every single person says, "I just want to speak to a human. There's 99% I can do myself, but for this 1%, I just want to speak to someone." You can't replace that.

That's what we're trying not to do at Moshi. We give you the data and the information. We do all the data engineering you wouldn't know how to do, because most people are not trained data scientists, data engineers and quants. But ultimately, you have to decide what trade you want to put on and how you want to manage it. We can give you the conviction and the data. That's what we're selling: you are going to be so much more confident in your trades, because you have blind spots. Why do you have blind spots? Because we watch way more data than is humanly possible. We'll tell you the connections, but ultimately you're the one who's going to put on the trade and sleep with that risk.

I think it's a really interesting time. Social media makes people see things a certain way, but the reality is that AI needs to be a tool for certain things, not a tool to replace everything.

So much of this resonates with me. I can't tell you how many times I've gone to a brainstorm or a workshop and people have shown up with the same ideas, because they all used the same system. It's like everyone shopped at the same stores and showed up in the same gray top and jeans.

It really is. And one funny thing about what you just said, everyone coming up with the same thing: we gave about 20 people a hiring assessment, and we told them to use whatever AI tools they wanted. We asked them to come up with a list of 10 target people we should speak to. Real people at real companies, from LinkedIn.

Bear in mind we interviewed people across the US, and LinkedIn has hundreds of millions of people. Can you believe that 90% came up with the same 10 companies?

I'm not putting anyone down, but we were hiring salespeople and a head of growth. If you're doing outreach to clients, think about that. Yes, I agree AI can help you. But think about what your difference is.

There's an idea I keep coming back to: the iPhone moment that Major League Hacking (MLH) CEO Swift has written about. Suddenly everyone thought they could be a photographer, but you still wouldn't let someone with an iPhone shoot your wedding, you know?

There's a lot we can do with AI, and there's a lot we don't have to do with it. It's simple: if I can get Claude to do something, then I will get Claude to do it. I'll get Claude to do the 90%. But that 10% is exactly where you don't want Claude.

You don't want AI there, because at the end of the day, if something goes wrong, you want to go talk to someone. You want to be able to say, "Hey, let's figure this out." You don't want to just talk to your chat model.

The AI Data Paradox: The Safer You Make the Data, the Worse the AI Gets

Tom Fairey — Wed, 14 Oct 2026 12:05:35 +0000

AI engineers spend enormous amounts of time tweaking small things to achieve bigger gains. Energy is poured into analysing model performance, rewriting prompts, tuning retrieval and building elaborate evals. But in enterprise settings, model performance is often most affected by what happens before that work even begins, by what goes into the pipeline first.

Data isn’t something that a system solely consumes; it is part of the architecture itself. It is as structural as the model or prompt — the blueprint for the structure. In most enterprises, where data is so sensitive, this blueprint arrives with sections missing.

Enterprises have the most delicate pain point. The data is richest here because it is proprietary, intrinsically tied to each organisation. And this richness of context requires the most cleaning. Compliance and security teams, working to either internal or external policy, run data through protection tools that detect sensitive information and then redact, mask, or tokenise it. Their job is to make sure nothing sensitive is exposed or leaked, not to check whether the data is still useful.

A paradox is created: an enterprise’s dataset is valuable because of its specificity and its representation of a company’s customers, successes, failures. Yet the data carrying these signals is the first to go, leaving a mottled blueprint.

If data is a blueprint, then redaction is a black marker taken to the plans. The room layout survives, but the dimensions, load-bearing walls and materials are incomplete or damaged. Even with the best builder, or developer, the building will still stand on shaky foundations. You can’t expect it to be the most beautiful, sturdiest building on the block if it was built from partial plans.

The richer the data, the more aggressively it is stripped. This produces technically safer but sterile data that is rarely useful for generating insight. This cost is measurable: a recent study from the University of Bonn and Fraunhofer IAIS found that a model scored 0.73 on a reasoning benchmark using the original text, but this dropped to 0.52 once personal details were redacted. Replacing them with consistent pseudonyms brought it back to 0.70.¹

Starting a pipeline with incomplete records puts a burden on developers and engineers to improve other aspects and fix a problem that started before them.

Every option to preserve safety gives something up. Self-hosted models keep data in-house but demand infrastructure and talent many teams lack or cannot budget for. Aggregated data smooths away the edge cases that are often the most insightful, and doesn’t work when single records need to be reasoned with.

An adaptive approach would not look at “what is sensitive?” but “what needs to be kept for the model to perform best?” Identity and context might seem inseparable, and in many ways they are. But providing relevant or equivalent context while protecting individuals behind data is one way models can be given a more complete blueprint.

Models, prompts, and retrieval are already seen as things to design. Privacy measures should be seen like this too. A requirement or restriction that has to be adhered to can be an opportunity for innovation and creativity.

The combination of a safety requirement and the need to stay attuned to an organisation’s specific data produces fertile ground for new approaches — and this shouldn’t fall only on engineers. Look at data as part of the system architecture, and its treatment becomes a crucial first step in achieving high performance.

Tobias Deußer et al, ‘On the Impact of Anonymisation on Large Language Models’, September 2026, https://arxiv.org/html/2609.11335. ↩

An Agent’s Plan Isn’t Permission

dev.to staff — Wed, 14 Oct 2026 12:05:28 +0000

When a coding agent investigates a production bug, it might need to read a repository, query logs, and open a pull request. If it finds something suspicious in another service, reading that service’s logs could be a reasonable next step. Whether it should have access depends on the investigation, but its credentials often grant the same permissions regardless of the task.

API keys and tokens usually say little about what the agent was asked to do, who assigned it, or how the task reached the agent. Intent-based authorization uses that purpose and context to determine access for each task. It evaluates the requested access against the permissions of the person or system that assigned the task, the agent’s assigned purpose, and organizational policy. A request to retrieve HR records, for example, should remain outside a customer-support agent’s remit even if its tools could reach them.

Access follows the task

Consider an alert reporting errors in a payments service. An orchestrator proposes an investigation, and an authorization service evaluates it using the authenticated trigger, the agent’s permitted role, and policy established outside its control. It then grants an incident agent read access to that service’s logs for the previous hour.

As the investigation progresses, the agent finds evidence pointing to an authentication service and requests those logs too. Policy may permit that additional access automatically or require human approval, but the runtime blocks the request until access is granted. The agent can follow a new lead without deciding for itself which permissions come with it.

If the incident agent delegates log analysis, the receiving agent uses its own verifiable identity and gets only the access needed for its part of the task, within its own permissions. That handoff can narrow the existing grant but cannot expand it, and it preserves the task context and records who passed it along.

Throughout the investigation, authorization must distinguish trusted context from untrusted content. An authenticated alert can contain a log line written by an attacker, so trusting the alert’s source does not mean instructions inside it can authorize access to another system.

Running a software factory

In a software factory, agents increasingly propose tasks as well as execute them. An orchestrator can watch incidents and scanner findings, pick up bugs without a ticket, and dispatch agents to investigate and propose fixes. Its ability to identify useful tasks doesn’t give it permission to assign whatever access those tasks might require. The factory needs an independent authorization decision, with limits enforced as the agents run.

Those limits extend beyond access because an agent can pursue a legitimate task badly. It might make thousands of unnecessary requests or produce a change that passes review but fails in production. Budgets help limit consumption, while checks and tests help determine which changes can advance. Audit records outside the agent’s control preserve what happened, and revocable access gives operators a way to stop activity when something goes wrong.

The Software Factory reference architecture, developed by Keycard, Docker, Barclays, Socket, and Aaron Stanley, shows how these controls fit together. Built on Agent Baseline, it defines four boundaries: identity and access, supply chain, sandbox, and network. These boundaries hold as autonomy increases, combining task-scoped access with controls on the software agents use, where they execute, and which systems they can reach.

A factory can let agents decide what to investigate next while keeping decisions about access under independent control. That separation allows it to take on more responsibility and autonomy without giving every agent broader standing permissions.

Read the reference architecture at agentbaseline.org.

Inside the $30-a-Week Self-Improving Agents Eliminating Enterprise Back-Office Friction

Kara Silverman — Wed, 14 Oct 2026 12:05:21 +0000

In the high-stakes world of enterprise AI engineering, the default assumption is that reliability costs money. Scaling accuracy usually demands larger models, endless retry loops, expensive visual processing, and human-in-the-loop fallback teams.

At Brainbase Labs, Engineering Lead Milind Shah is quietly proving the opposite.

Ahead of his Wednesday afternoon session at AI Engineer NYC (“Hill Climbing Back Office Agents”), Shah sat down with The Daily Context to unpack a case study with Domino’s Pizza that flips the standard AI economics curve on its head. Facing a costly, manual process to track competitor pricing and promotional discounts across local fast-food chains, Domino’s deployed autonomous browser agents managed by Brainbase.

Over nine iterative versions, the system’s task accuracy soared from 41% to 91%. Simultaneously, operational costs plummeted from thousands of dollars per month for an external consulting team down to just $20 to $30 per week—with inference costs dropping to $1.16 per source.

The story behind that 50-point accuracy jump offers a masterclass in pragmatic AI engineering for the 1,000+ technical leaders gathered this week at the Sheraton Times Square.

Phase 1: Defining “Correct” Before Writing Code

Before Brainbase touched a single prompt or model parameter, Shah’s team had to solve a foundational problem: defining what success actually looked like.

Domino’s had previously relied on human analysts to manually inspect competitor websites, collect local coupon codes and promo pricing, and construct structured spreadsheets for executive decision-making. To replicate this with software, Shah sat down directly with non-technical business teams at Domino’s to decode their reference spreadsheets.

“They already knew what a good spreadsheet looked like,” Shah explained. “We used that to create an internal benchmark dataset of questions and expected outputs. Without an internal benchmark, you can’t prove accuracy numbers or tie improvements to cost reductions.”

This benchmark became the foundational ground truth against which every prompt tweak, model swap, and agent instruction was scored.

Phase 2: The Coordinate Hack & Model Optimization

Once the benchmark was established, Brainbase systematically tackled the agent’s execution bottlenecks. Surprisingly, the single largest jump in accuracy didn’t come from a bigger LLM—it came from stripping away visual inference.

In early iterations, the browser agent relied on visual processing to locate buttons and navigation links on competitor websites. This approach was slow, expensive, and prone to misclicks. Shah realized that fast-food ordering interfaces have static UI layouts. Instead of making the model parse image tokens to find a “View Deals” button on every page load, Brainbase hardcoded fixed X/Y screen coordinates for recurring UI elements.

“Instead of having the agent figure out where the button is by reading the image, we told it the exact X/Y coordinates,” Shah said. “That eliminated expensive image-reading inference and single-handedly produced our biggest accuracy gain.”

Concurrently, Shah experimented with model selection. Rather than defaulting to Anthropic’s flagship Claude Opus 4.6, the team discovered that tightly specified, highly structured prompts paired with Sonnet 5 delivered superior accuracy at a fraction of the token cost.

Phase 3: The Unsolved Frontier of Self-Improvement

Brainbase’s platform now powers more than 10,000 internal agents across enterprise back-office workflows—ranging from invoice exception clearing to inventory transfers. While Brainbase’s automated “Hillclimbing” engine can automatically detect and remediate failure modes like wrong tool calls or hallucinations, Shah remains candid about the limits of current automation.

Complex browser navigation traces still require human engineering eyes. “Automated improvement works for tool calls and hallucinations,” Shah noted. “But browser navigation is still too intricate for auto-improvement agents today. Engineers still have to manually review agent traces to find where browser actions break down.”

For teams working with tight evaluation budgets, Shah offers straightforward advice: don’t overcomplicate your initial evals. “Deterministic eval criteria go a long way at low inference cost—checking for hallucinations, tool-use correctness, and adherence to knowledge bases.”

The Engineer Behind the Agents

Shah’s path to leading engineering at Brainbase spans software engineering roles at DoorDash and Adobe, scaling the code-data business at Merkle from zero to eight figures, and co-founding the evals platform Pipelines.

Despite building autonomous systems that handle complex enterprise workflows, Shah maintains a grounded, reflective view of the AI revolution.

“I’m optimistic overall, but I think about skill atrophy a lot,” Shah admitted. “If I’m using AI to write code today, and tomorrow I’m using it to write everything else, you risk losing those core skills. A good AI engineer still needs strong engineering fundamentals first.”

Milind Shah speaks on “Hill Climbing Back Office Agents” on Wednesday, October 14 at 2:50 PM ET at AI Engineer NYC.

Building Better Agents through End-to-End Evals

Ryan Stevens — Wed, 14 Oct 2026 12:05:13 +0000

Deeply understand the task at hand

The biggest mistake teams make with agents is automating a process they don’t understand. Before writing one line of code, you should be able to describe exactly how a human does the job.

To build Ramp Stack, we wanted to automate bookkeeping tasks. Before thinking about code, we thought about the jobs to be done. There are four primary categories of work: reconciliation, data entry, variance analysis, and schedules and accruals. Categories keep the eval balanced across the real job and later show you where the agent is weak.

Next, we break each task into its steps. Take an accounts payable reconciliation: an accountant pulls unpaid bills and the AP ledger, lines them up in a spreadsheet, and investigates anything that doesn’t match. If the bills total $3,500 and the ledger shows $3,000, the job is to find the missing $500 bill and record the correction. Each step is something the agent must do and something your eval must check.

Data is the bottleneck

An end-to-end eval has the same shape at every stage: business data populates a database, the agent works with that state through tools, and you grade what it produces. Before launch, you have no data to work with, so you must simulate it.

Our answer was synthetic businesses, which we call worlds. One is a DTC home goods brand selling through Shopify, Amazon, and wholesale, with a full set of books, realistic transactions, and messy Excel files. The mess matters; too often, teams build data that is too clean. This means things look good offline, but do not survive once confronted with real users.

Anatomy of an eval

Each eval case combines four parts:

Task: the request, such as “Reconcile our AP payables and bills for Q4 2025.”
World: the synthetic ERP and business data the agent can access.
Trajectory: the steps the agent takes, like reading bills, matching payments, and investigating differences.
Score: a rubric of accountant-written criteria, graded by an LLM judge on the final output.

The rubric is what makes the eval useful. Pass/fail hides partial progress; six criteria such as “Q4 AP balance correct” and “duplicates flagged” tell you the agent scored 4 of 6 and exactly which two it missed. Have domain experts write the criteria, since they know what a reviewer would actually reject.

Using the eval to make changes

An eval earns its keep when it changes decisions. The first decision was model choice, but the biggest alpha is tuning your harness: optimizing skills, tools, and infrastructure to produce good results at low cost and latency. Several popular ideas didn’t hold up:

Auto-tuned skills (GEPA-style autoresearch): training gains didn’t generalize. We have yet to see good evidence that automated prompt optimization works.
High reasoning effort: same accuracy, higher cost.

What did work was treating the agent like an ML model and its context like features:

Skill ablation: we benchmarked one skill at a time against a no-skill baseline, then trimmed eight skills to four, gaining 2.8 points of mean score across six worlds. Several skills actively hurt, and none was best everywhere.
Skill compression: cutting our largest skill, a spreadsheet skill, by 61% left quality essentially unchanged and saved about $30k a month.

The pattern: measure every change against a fixed baseline, across many worlds, and keep only what moves the score.

Test what others skip: memory

Memory is rarely evaluated because it requires time to pass. We built roll-forward worlds: the same business, advanced one month. The agent completes this month’s close, writes down what it learned, and is graded on next month’s tasks.

The rule we learned: remember how the business works, but use fresh numbers. Memory distilled from prior trajectories, with period-specific amounts removed, improved mean score by 5.7 points over skills alone. A memory that captured a reusable workflow took one task from 0% to 100%. One that smuggled in last month’s specific recipe dropped another from 75% to 25%.

The takeaway

Agent development needs the same discipline as ML development. Understand the process, build a dataset hard enough to be meaningful and stable enough to compare runs, and let it, not intuition, decide what ships.

Your Financial Agent Passed Every Eval. Can You Defend Its Decision?

Anirudh Veeraragavan — Wed, 14 Oct 2026 12:04:58 +0000

In audit, the opinion is the product, and an opinion is only as good as its basis. Being right isn’t enough; you have to show the decision-making process behind it. This raises the question: what does agentic decision making look like?

Let’s Build an Agent

We’ll take a financial audit use case and focus on a specific procedure: testing a company’s bank reconciliation. Companies compare the cash balance on their books to what their bank reports, and any difference should be explained by things like deposits in transit or uncleared checks. Testing this is tedious, high-volume work, which makes it a good fit for an agent.

How might we do this? We’ll probably feed the agent the client’s books, reconciliation, and bank evidence. We can even include the audit methodology in the prompt and use prior-year audits as examples. Finally, we build an eval from previously reviewed audits and ensure a human signs off before anything is finalized. The metrics look great, users are happy, and we ship it.

Then, the audit gets selected for review, and someone asks: why was that sufficient? They don’t care what the evals say, and they don’t even care what the users say. For this specific audit, in this specific instance, why was this decision made?

Four Controls for Defensibility

In light of this, what does it mean to arrive at a “defensible” answer? Audit standards have a good definition for inspiration, paraphrased as “an experienced auditor, new to the work, can follow it.” We can translate this into “can follow the agent” and enforce this through four controls: authorization, execution control, evidence provenance, and evaluation integrity. Additionally, there is always the line where an agent stops and a human makes the final call.

Authorization. An auditor flags a $62,500 check to a payee missing from the vendor master, and a reviewer signs off. A few days later, new statements arrive, which triggers the agent, and it rewrites that row to “no exception.” The signature now vouches for an answer nobody reviewed. Every write was permitted for the agent’s identity, but it should not have been allowed on a signed row. Authorization isn’t just “can this identity call this tool”; it’s whether this action is allowed on this object, in this state, right now.

Execution Control. The agent says a $96,400 check cleared. An auditor reruns the job, and today’s run reads $69,400. Logs tell you the job ran, but they don’t let you reexamine one decision. Instead, break the run into steps (fetch, extract, match, conclude), and record each step’s inputs, outputs, and model version under a stable ID. Then rerun only the contested step on its exact inputs. It isn’t about whether the old agent or the new agent is correct; it’s about knowing why the result is what it is.

Evidence Provenance. The agent cites “January bank statement” for a $61,900 deposit. This ended up being wrong, but where did the citation come from? Many documents could share the same name, or the same document could have been modified over time. Bind every citation to a stable identity (document ID, version, hash), not the model’s text. Then enforce that the evidence attributes the claim depends on, such as relevance, actually match.

Evaluation Integrity. Can we trust the eval? It’s built on last year’s conclusions, not ground truth. There could be misses where the agent was actually right, but the label was wrong. There could also be hits where the agent was actually wrong, but the eval wasn’t updated to reflect changes in methodology. Tag every gold label with the methodology and period it’s valid for, audit both the misses and the hits, and report results by slice.

Human Review. Each failure above had a moment when the agent should have handed off to a human. The agent should stop when it’s about to change work a human signed, when a rerun disagrees with the record, when evidence doesn’t match the item, or when a required rule fails. Additionally, escalations should be easy to verify, not necessarily easy to approve. When one of these fires, ask a specific question, show the conflicting evidence side by side, and record the human’s answer.

Going Beyond Audit

None of this is specific to bank reconciliations or even financial audit. A declined loan, a closed AML alert, a price determination: the same four questions apply every time. Was the action allowed? Can you show what ran? Is each claim bound to the right source? Can you trust the benchmark?

Wherever an agent’s financial decision can be challenged, defensibility, not just correctness, is the bar.

Anirudh is Lead Security Engineer and the founding security hire at Fieldguide, the leading transformation partner for agentic audit and advisory.

Memory Isn’t Learning: Why AI Agents Keep Making the Same Mistakes

Sparsh Jain — Wed, 14 Oct 2026 12:04:49 +0000

Have you ever told your AI agent something important, only to realize a few conversations later that it had forgotten everything? So you tell it again. And again.

A few months ago, I was speaking with a Director at a hedge fund who described a problem that stuck with me.

“Sparsh, the problem isn’t that we can’t train an agent. The problem is that as soon as someone else on my team starts using it, everything resets to zero.”

Think about how humans work. A senior analyst doesn’t start from scratch every morning. Over years of working through difficult cases, making mistakes, and observing outcomes, they develop something much more valuable than information: judgment.

They know which signals matter, which approaches tend to fail, and when an apparently reasonable decision deserves a second look. Our agents, however, can execute thousands of tasks and still make the same mistake tomorrow. And that raises a fundamental question:

Are we building agents that remember more, or agents that actually get better?

1. The Memory Illusion: More Context, Better Intelligence?

If you’ve been working in agentic AI recently, you’ve probably heard the same answer everywhere: memory.

Give agents longer context windows. Connect vector databases. Store conversations, tool calls, execution histories, and previous decisions. The assumption seems reasonable. If the model has access to everything that happened before, shouldn’t it make better decisions?

We initially believed so too. So we started feeding our agents more context: historical cases, previous executions, retrieved documents, and accumulated knowledge. Then we ran independent evaluations.

More context wasn’t consistently producing better performance. In some configurations, performance actually deteriorated.

We tried manually refining the context, restructuring instructions, and carefully selecting what the agent could see. Some configurations that looked more complete performed worse than simpler ones. And as we started removing information, we discovered something counterintuitive:

Sometimes, the best way to make an agent smarter is to give it less to remember.

2. Facts Are Not Experiences

This led us to an important distinction:

Facts and learning are fundamentally different.

Facts describe the world. A customer’s contract expires next month. An API requires authentication. A particular repository contains the billing service. These belong in databases, knowledge graphs, or document stores.

Learnings, however, describe how to act. For example:

When investigating a production incident, check recent deployment changes before debugging unrelated components.
When two sources conflict, prefer the recently verified source rather than the most semantically similar document.

These are not merely facts. These are refined judgments derived from experience. Experience requires understanding what action was taken, its outcome, why it occurred, and whether the lesson should influence future decisions. An execution log tells you what an agent did.

An experiential memory tells it what and why to do differently next time.

3. Memory Must Be Built as a Learning Loop

A useful memory system needs a complete lifecycle:

Capture → Evaluate → Consolidate → Retrieve → Update

Capture: Record meaningful experiences, not every interaction.
Evaluate: Determine whether the strategy worked and whether the outcome was independently verified.
Consolidate: Transform repeated experiences into reusable lessons. Ten similar failures shouldn’t become ten separate memories.
Retrieve: Surface the right lesson when the agent encounters a relevant situation.
Update: Revise or delete memories as environments change.

A strategy that worked three months ago may become dangerous after an API or policy changes.

Even accurate memories can become counterproductive if surfaced at the wrong time. That’s why we started thinking about progressive disclosure: giving an agent the minimum relevant information for its current decision, revealing additional context only when needed.

The objective isn’t maximum recall. It’s better decision-making with the right information at the right moment. Progressive disclosure works like wonders in these scenarios.

Final verdict: From Agents That Remember to Agents That Improve

The industry is rapidly converging on memory as an essential component of agentic systems. I agree. But memory by itself isn’t the destination. The real challenge is building systems where experience compounds.

An agent executes a task, observes an outcome, extracts a lesson, validates it, and applies that lesson to a future decision. If the lesson proves useful, it becomes stronger. If it becomes outdated or harmful, the system revises or removes it.

Over time, the agent doesn’t just accumulate more information. It develops more reliable decision-making patterns. That’s the direction we’ve been building toward through Loop Engineering: treating evaluation, memory, and feedback as a continuous system rather than isolated components.

Because the goal isn’t to build an agent that remembers every mistake it has ever made.

The goal is to build an agent that doesn’t need to make the same mistake twice. A system where experience compounds.

Nobody’s winning the AI race yet

Ryan Swift — Wed, 14 Oct 2026 12:04:39 +0000

By Ryan Swift

Yesterday, Google finally released its Gemini 4 Argon model for public use. It’s Google’s first flagship release since Gemini 3.1 Pro in February. Whether or not real-world usage lives up to the promising evals, reports of Gemini’s death have been greatly exaggerated. We’re still incredibly early in AI’s development, and while the pace keeps accelerating, there’s plenty of room for comebacks and setbacks in this race.

Over the summer, Google shipped new Flash models at a steady clip, and I found each one was noticeably better than the last. You wouldn’t have known it from the online commentary, which treated Google’s entire model family as unusable. At the AI Engineer World’s Fair conference in July, I wrote that most tasks don’t need a frontier model. All summer, Gemini Flash and Flash-Lite quietly handled the vast majority of my real work. Now that Argon is out, I’d bet many of those same voices will flip-flop over the next few weeks and declare Google destined to win. That’s just as wrong.

It’s still far too early to declare winners and losers of the AI model race. First-mover advantage doesn’t seem to carry the weight it once did. Google was the first major lab in the always-on assistant space (Gemini Spark), but it has since been leapfrogged by Grok Bot, Meta Muse, and OpenAI’s Dots. There’s no clear winner here either. All of these products (and their underlying models) are still in the earliest stages of adoption. Until my mom asks me which assistant to use, they haven’t hit the mainstream.

If anyone’s telling you who’ll win the race, they’re just guessing. Stay flexible. Try the latest releases. Build a small set of your own evals and tasks, because your benchmark matters more than the leaderboard. Avoid locking yourself into any one model. And give Gemini 4 a shot. I bet it’ll surprise you.

Operational intelligence is sociotechnical

Spencer Pitman — Wed, 14 Oct 2026 12:04:31 +0000

A scan of the Slack channels for a recent private equity deployment of Clonal (our operational intelligence platform) might confuse you at first. There is far less talk of dbt models, evals, or Monte Carlo simulations than you would expect, and far more about “human factors,” “emotional resonance,” and endless conversations about how to facilitate alignment. The human kind, not the AI model kind.

Creating software is increasingly the easy part of our job. The harder part is building systems that people trust, use, and improve as they work with them. In financial services that matters a great deal. On the day we deploy Clonal, it is inert. Even once data is flowing, connecting sources and generating summaries does not create value on its own. Value shows up when better context changes front-line work: how analysts prepare, how ICs decide, how BD coordinates, and how the firm acts on an investment.

Our PE work shows this. Neither our clients nor we want another place to go look things up. We want the firm’s own history available to the person making a call, when they are making it. Part of that is an engineering problem. The rest is social. It takes careful system design and iteration, and it takes what we call human motions: the recurring things people do around the system, like walkthroughs, reviews, and deciding who owns what.

Workflow even over feature

Consider how a deal lead prepares for a banker conversation meant to generate dealflow. Their questions are simple: What have we discussed before? Which companies are coming to market? What did we pass on, and why? What should we ask next?

A technically impressive platform can make that preparation harder if the answers are scattered across overlapping screens, abstract scores, and unfamiliar categories.

Early on, we built these as fully realized features, with visuals and styling. John Cutler, our product lead, noticed that this collapsed several mental models into one. The client was reacting to the data, the logic behind it, and the presentation all at once, so we could not tell which of those their feedback was about, and we rarely got a clear read on the actual state of the graph data. Now we start by understanding the question and writing sample queries, then sit with the client and walk through what the data surfaces. It’s a bit like paper-prototyping the insight.

The operating model lives inside the architecture

In ontology design (we call it “topology”), small distinctions carry large consequences.

A company is not a deal. One company can have multiple transaction processes over time. An internal decision to pass may not be the same as communicating that decision externally. A fund passing does not mean the company is no longer in market. A missing interaction record does not prove a relationship has gone cold. We hit these constantly. They are the everyday semantics of the business.

If the platform collapses those distinctions, it can present a clean interface that misrepresents reality. Forward-deployed engineers and product managers work through system design with the client so the distinctions show up in the data model, workflow ownership, permissions, and visible action states. Without that regular contact, we build things that function but don’t work.

Closing workflow loops and learning loops

Forward-deployed work keeps paying off after implementation.

Engineers and product leaders working alongside users see the gap between a requested capability and a usable routine. They watch someone navigate a real deal, investigate an unexpected comparable, or struggle to find the next action. That proximity exposes problems a requirements document rarely captures. Cheap code lets us compress those cycles a lot, but design and interaction with real people doing real work is the most important kind of HITL we do.

We are borrowing heavily from DDD and service design to make these sessions useful: short walkthroughs, focused wargaming, testing against real records, visible feedback tracking. All of it helps clarify what “good” means. Sometimes we can gather that from the environment. When we step into judgment-heavy work like deal evaluation, we have to find ways to turn intuition into evaluation.

Trust is designed through restraint

In investment work, an attractive black-box score is usually less useful than a dated note with a clear source. A label like “cooling relationship” sounds intelligent, but it backfires when the professional knows they have had several recent conversations. “No recorded interaction since this date” claims less, and the reader can act on it.

The same applies to automation. An elapsed deadline can prompt a review, but it should not declare a process dead on its own. A queued update should not appear completed. Unknown data should stay unknown instead of becoming a confident-looking zero. None of this is hard to express in software, even hand-written code. It is easy to overlook in design.

The deliverable is a repeatable capability

Our goal is for clients not to need our deployment teams. We aim to leave behind an organization with clearer ownership, reusable context, reliable decision pathways, and the ability to keep improving its own system.

The software we build and configure provides the machinery, and the human motions around it make it operational.

Are You Smarter Than a Fruit Fly?

Nick Nisi — Wed, 14 Oct 2026 12:04:21 +0000

Google mapped the complete male fruit fly brain, and my first question to an agent was “Can this run on a laptop?” The simulation has 176,422 neurons and runs on a laptop CPU.

My second question was “Is there something ridiculous that we can do with it, though? That’s engineering-focused: make it do a code review or clear context or something.”

The fly can’t read code, so Jev, a TypeSafe model, answers thirty yes/no questions about each change and the answers become a smell. I tried to train the fly on the codebase by using mutation testing to identify good and bad parts of the code and then train the fly to have memories based on that. On 301 changes it had never seen, it agreed with the tests 290 times. A logistic regression on the same thirty answers got 298.

If a fruit fly can do code review moderately well, then why are humans still doing it?

Reviewing code is the bottleneck. Humans aren’t involved in writing it anymore, and the functionality is what matters. Models are getting good, which means the code quality is getting good and we’re at the precipice of starting to let go of that control.

It’s a big trust issue for humans. Feels like giving up a lot of control but I think that humans are just as capable of slop as agents are. Agents are able to be persistent and review, given the right set of tools and the right context on the outcome.

I want evidence and a receipt that it tested the results and that what it’s proposing to ship actually solves the problem or adds the feature or addresses the issue in the way that’s intended. I want:

Videos
Screenshots
Test results
Command output
Smoke tests
An easy way for me to replicate it myself

On the last agent PR I approved, I looked at the most important bits. The part that made it most important was just where the core business logic was. Agents can verify that it’s logically correct, but we need to make sure that it makes sense for the use case or the application.

Humans should understand why code is changed and why the way that it’s changed matters, but at a higher level, not necessarily line for line. They should just have an understanding of what the code is doing and why, and specifically why not other methods.

The Model Is the Easy Part

Steven Latre — Wed, 14 Oct 2026 12:04:00 +0000

We spent the past few months on a question that sounds simple: what does it actually take to move a coding workload off a frontier API and onto open-weight models you run yourself?

We picked coding agents deliberately. They are long-horizon, they are messy, and they are the workload most organisations would move first, because that is where the token bill is growing fastest. We ran a fixed set of real software engineering tasks across a range of open-weight models, each paired with hardware capable of serving it properly, and varied the agent harness on top. Then we measured what came out: tasks resolved, time taken, users served in parallel.

Three things surprised us.

Capability is priced in steps, it’s not linear

The intuition going in was that better models cost proportionally more to run. They don’t. Quality improves gradually across the open-weight field, but the hardware underneath moves in jumps, because what matters is whether the weights fit in the memory of the box you already have. Cross that line and you are not paying for a few billion more parameters, you are paying for the next machine.

So some models are badly priced for what they give you and others are a bargain, and which is which has little to do with where they sit on a leaderboard. One mid-sized model in our set cost 4 times more to serve than a smaller one while barely improving the application performance with 2%. A much larger one cost only marginally more than the tier below while solving substantially more. The useful question is not which model is best, but which model lands just inside a hardware tier, measured per resolved task rather than per token.

Concurrency is the variable nobody budgets for

Behind an API, concurrency is someone else’s problem. You get a reasonably predictable response time and you stop thinking about it.

If you manage your own stack, it becomes the binding constraint. Every user brings a context, every context has to fit in memory, and latency stops being acceptable long before the hardware stops working. How many developers a box can comfortably carry varied enormously across our setups, and not in the direction the spec sheets suggest. The largest, most capable configuration served the fewest people at once.

This is where most self-hosting business cases quietly fall apart. The break-even utilisation for the weaker configurations we tested sits far above what enterprises actually achieve. Reported utilisation in real deployments is low, and well-run estates are not dramatically better. If your case needs near-continuous saturation, it does not work.

The harness changes the bill and only marginally influences application performance

We compared three widely used coding harnesses on identical hardware with identical inference settings, expecting the orchestration layer to matter a great deal. For resolve rate, it barely moved the needle. The spread across harnesses was a couple of tasks either way.

For cost, it mattered a lot. Same model, same hardware, same tasks resolved, and wall time differing by a factor of two depending on the pairing. When you are renting GPUs, wall time is what really defines your cost.

And when you settle on a harness, evals are key as things can easily break. In our experiments, one harness was the clear winner, until the next software release, made our performance collapse. The culprit was a small change in how prefix caching was handling. Suddenly, the caching mechanism broke entirely and the GPU rental cost skyrocketed with a factor 5. The key takeaway is: pick the harness your developers want to use, then instrument it, because a point release can move your costs more than a model swap.

What we would tell you to do with this

The question is not whether self-hosting is cheaper. Asked that way it has no answer, because it depends on your workload, your peaks and your tolerance for slower agents.

The better question is what your real utilisation looks like, and what you cannot put on someone else’s API at any price. Data that is not allowed to leave the building, a stack nobody can rate-limit, prices that do not move under you mid-quarter. Those reasons survive contact with the spreadsheet. Cost alone usually does not.