Ben Holmes is the DevEx Engineering Lead at Warp. The company has spent the year shifting from the terminal that made its name to Warp Factories – open infrastructure for teams building software factories. He's also known for his whiteboard explainers, which turn complicated systems into short, drawn diagrams.
Ahead of his AIE New York talk, "Your Agents Need a Skill to Improve Their Skills," he sat down with The Daily Context to talk about factory engineering, why Warp built its pivot in public, and how product development teams still aren't ready to hand it to an agent.
This interview has been edited for length and clarity.
The Daily Context: Tell us a little about yourself and your role at Warp.
Ben Holmes: I'm the DevEx engineering lead at Warp, so I work on everything that helps people work with the product. That means working with both our open source community and our growing community of teams using Warp Factories. We've made a pretty big shift. The terminal is what defined Warp up until now, but Warp Factories is what defines Warp today. Warp Factories is open, flexible infrastructure we've set up to help teams build software factories. We've built it as much in the open as we can, sharing our learnings along the way.
People are trying to automate development. They're trying to take these interactive agents that they have to hand-hold through every PR and ask: can we automatically trigger an agent when errors happen? Can we work multiplayer in Slack and communicate that way? Can we trigger it from task boards? In all these places, we want to give agents more control over creating everything.
We also want to create the system that actually lets us deploy all these agents. That system should give us tools to measure whether those agents are performing well, and give the agents tools to review their own work and suggest ways to improve the system. That's why our CEO, Zach Lloyd, uses the term "factory engineering." It means doing high-level engineering on the system the agents are working inside of, and less "prompt, prompt, prompt." Let's look at the skill makeup. Let's look at how we define each agent if we're orchestrating multiple agents together. Let's set up the automation chain: where do we trust agents to just do something without us prompting them? That's the high-level work engineers are really starting to focus on now.
The Daily Context: What is your personal relationship to AI?
Ben Holmes: Right now, I see AI as a super capable partner that can proactively figure out how to do a task, and more importantly, what tasks need to be done. That second part has been really interesting to me, maybe more so than just asking agents to go do things. Working in a fast-moving startup, it's hard to tell: did this thing fall off my calendar? Is this something we should be focusing on right now?
In software development, we're slowly getting there. We have agents that look at Sentry alerts and Grafana alerts and figure out which bugs you should actually focus on. That's one example. There are also agents that watch my conversations and say, "This conversation you had in the past went really poorly. You should probably update your skills to make it better next time." The agents that are watching how everything is going are maybe the more interesting ones to me.
So, back to the relationship question: I see them as highly capable tools. More and more, though, I see them as something that can be proactive and help me figure out what I should be focusing on and doing. Of course, I've been trying out crazy scheduled agents that review my Gmail and do all the personal assistant type stuff. We're also trying to figure out what that means for a whole team. We have a virtual assistant bot right now. We have a thread where we do our standup and share what we did yesterday and what we're doing today. The bot reviews all of it and puts it in a Notion document so we're all on the same page. It can also be proactive and ask, "Did you get this done? Did you get that done?" They're starting to become proactive tools that you don't have to poke anymore. They're just constantly working for you, and that feels like an experience I want to craft.
My last talk at AI Engineer was actually about second brains: something that can look through a bunch of scattered dots and figure out which of them are the gems worth a second look.
The Daily Context: Your bio notes your whiteboard explanations. As teams adopt something like Warp Factories, how do you think about helping people learn new skills in the AI era?
Ben Holmes: I think if you can distill something down into a really short video or a really small diagram, then you really understand it. If you can't do that, you don't understand it super well yet. So I almost take it as a challenge to make sure I really understand something, and that usually leads to a better explanation.
I've done whiteboard shorts for a long time, and those aren't long-form videos. For a while, YouTube Shorts had a one-minute limit, so I had to figure out how to explain a front-end JavaScript framework, which is what the shorts were about at first, in one minute. The whiteboard was the perfect medium for that. I set the limit that I couldn't use overlays on the screen. It had to be a drawn diagram. I could use some stop motion to explain it, but I forced the medium as a way to be more creative and to make sure I was explaining something super clearly.
Now I've been using virtual whiteboards a lot more, like tldraw and Excalidraw, which have become very popular throughout engineering. It's also how I give all my talks now. I just open tldraw, show a diagram, and slowly walk toward the full explanation. That feels like a better way to tell a story than narrating your way through everything, especially now that we're thinking in loops and deployment patterns. Those concepts naturally need a loop or a graph drawn out before people can understand them.
The Daily Context: Warp made a big pivot this year and did a lot of it in public. Why make that process visible and open source?
Ben Holmes: The mission has stayed the same for us: help developers ship better software. The terminal was a vessel for that, and it made a lot of sense. Then, by following the community and seeing what people were working on, we saw that these automations around agents were clearly the next step in that journey.
Honestly, we didn't know what that would look like. We were watching how everyone was using these tools. As an experiment, we first open sourced the terminal. Second, we started building a system to maintain it, because open source is especially tough now with the amount of contributions coming in. If automation is really where we're heading, with agents proactively working on things, we need to build a system around that, even if we don't yet have a product we can help people with. We just needed to figure it out.
So we started building that system to manage the Warp terminal open source project, and before that, to manage our own work. On DevEx, I put out some resources on it and supported Zach Lloyd's articles and explainers, which showed early ideas using GitHub Actions. You can have an agent that triages every issue, an agent that does automatic code review passes, and an agent that implements with computer use. We were stitching those patterns together in an ad hoc way. Then we realized we had a system we could reuse and show to teams over and over again. We could distill it into something where you just sign up and get the triage agent and the review agent we found success with, plus a cloud runner with computer use, because the agent needs to verify its work. That became so obvious to us.
So we were dogfooding our own approach and sharing it as much as possible. Once we saw people really responding to it, we distilled it into something that helps other people do the same.
The Daily Context: Has the community used it in ways you didn't expect?
Ben Holmes: It's pretty early. Our actual factory setup has been around for a month and a half, and we have some early partners working on it. I don't think there have been any surprises yet, maybe just some early success stories. If there's one early surprise, it's that one company is using it for in-depth Android development. Seeing the amount of compute involved, we realized we really need to get a handle on driving down the cost per PR now that other people are using it. That's the first surprise to address. A lot of work still needs to happen, both on researching agent makeup and on improving our platform, so that if you want to run an agent automatically, it runs at a cost and speed that makes sense for teams. Maybe that means open-weight models. Maybe it means simplifying how many subagents you have. We're doing all sorts of research on this right now. That's been the main thing: we're stress testing it.
The Daily Context: What is your AIE talk about?
Ben Holmes: It's about self-improvement, which I mentioned earlier. It's titled "Your Agents Need a Skill to Improve Their Skills."
The Daily Context: Can you give an example of how agents improving themselves has played out in your own work?
Ben Holmes: First, I can explain the system briefly. The idea is scoring them on criteria that you design. It's kind of LLM-as-a-judge. Efficiency during a conversation is one example. Beyond skill improvements, we've been trying to expand this approach to also suggest architecture improvements, so you can have an even better codebase setup.
The Daily Context: How are you making self-improving agents easier for teams to see and work with?
Ben Holmes: We're coming up with a few ideas to make self-improvement more accessible and make it feel more like something you're optimizing. We have graphs that show all the scores and how they track over time, so you can see something like "your efficiency has gone up 10%." We're also playing with a blueprint idea, where you define all the agents currently deployed in your company and make that a little more visual. One of those agents might be the self-improvement agent, watching over the implementation agent. Making it visual makes it clearer that you're building a system and that the agents can all talk to each other. It's not a direct learning from Skill Doctor, but it's an idea I'm bringing in. We could make this more visual and make it feel like something you're actively building, instead of just looking at YAML files or a list of open PRs.
The Daily Context: Warp Factories lets teams run agent workflows from triage to verification. Is there a step teams don't want to hand over to an agent?
Ben Holmes: That's a good question, and you might expect what I'd say. I don't think any of them are really steps people don't want to give over. Triage is kind of a half answer, and I think it's something we need to do better. The idea of triage is that an agent goes off and researches something automatically when you suggest a feature idea, either in a task board ticket or in a chat. We're still figuring out whether that's a necessary step. Spec writing is another one. We have a spec writing agent, but we don't know if it should be separate, and we don't know how useful it is to have that speed bump of "Do you want to work on a spec together?" We may be at a point where it's more natural for a human to ask that. So we're thinking about removing, or at least optimizing, these speed bumps we put early in the design process.
Maybe the better question is about the types of tasks people don't want to hand over, which is kind of the same thing. I think we're still trying to figure out how to close the loop on product development: figuring out the next thing to build. That's still mostly humans saying, "Here's the next feature we should put in the product." This is kind of a blind spot for us too. It would be a nice world if an agent looked at usage data and figured out what your next feature or A/B test might be. That area feels underexplored. It also feels like something people don't really want to hand away yet, because that's product vision and product decisions, and that's kind of the last stand of human touch. We don't want to fully give it away, although if you point an agent at customer data, it might figure out some smart features worth trying. We haven't experimented with it yet. It just feels like an inevitable thing we'll need to reckon with. Closing the loop is something we talk about a lot, and this feels like an important piece. We've only closed the loop with bug fixes. We haven't figured it out for feature building.
Top comments (1)
The distinction between closing the loop on bug fixes and on product decisions is important. A practical intermediate step might be to let an agent produce ranked product hypotheses with the evidence trail attached—support themes, usage segments, and the counterexamples it considered—while a human still owns the decision. Then evaluate the system on calibration: which proposed opportunities were later validated, and which were confidently wrong? That makes the handoff auditable without pretending that a scorecard can replace product judgment.