DEV Community: Kara Silverman

Warp's Ben Holmes on Building the Software Factory

Kara Silverman — Wed, 14 Oct 2026 12:13:55 +0000

Ben Holmes is the DevEx Engineering Lead at Warp. The company has spent the year shifting from the terminal that made its name to Warp Factories – open infrastructure for teams building software factories. He's also known for his whiteboard explainers, which turn complicated systems into short, drawn diagrams.

Ahead of his AIE New York talk, "Your Agents Need a Skill to Improve Their Skills," he sat down with The Daily Context to talk about factory engineering, why Warp built its pivot in public, and how product development teams still aren't ready to hand it to an agent.

This interview has been edited for length and clarity.

The Daily Context: Tell us a little about yourself and your role at Warp.

Ben Holmes: I'm the DevEx engineering lead at Warp, so I work on everything that helps people work with the product. That means working with both our open source community and our growing community of teams using Warp Factories. We've made a pretty big shift. The terminal is what defined Warp up until now, but Warp Factories is what defines Warp today. Warp Factories is open, flexible infrastructure we've set up to help teams build software factories. We've built it as much in the open as we can, sharing our learnings along the way.

People are trying to automate development. They're trying to take these interactive agents that they have to hand-hold through every PR and ask: can we automatically trigger an agent when errors happen? Can we work multiplayer in Slack and communicate that way? Can we trigger it from task boards? In all these places, we want to give agents more control over creating everything.

We also want to create the system that actually lets us deploy all these agents. That system should give us tools to measure whether those agents are performing well, and give the agents tools to review their own work and suggest ways to improve the system. That's why our CEO, Zach Lloyd, uses the term "factory engineering." It means doing high-level engineering on the system the agents are working inside of, and less "prompt, prompt, prompt." Let's look at the skill makeup. Let's look at how we define each agent if we're orchestrating multiple agents together. Let's set up the automation chain: where do we trust agents to just do something without us prompting them? That's the high-level work engineers are really starting to focus on now.

The Daily Context: What is your personal relationship to AI?

Ben Holmes: Right now, I see AI as a super capable partner that can proactively figure out how to do a task, and more importantly, what tasks need to be done. That second part has been really interesting to me, maybe more so than just asking agents to go do things. Working in a fast-moving startup, it's hard to tell: did this thing fall off my calendar? Is this something we should be focusing on right now?

In software development, we're slowly getting there. We have agents that look at Sentry alerts and Grafana alerts and figure out which bugs you should actually focus on. That's one example. There are also agents that watch my conversations and say, "This conversation you had in the past went really poorly. You should probably update your skills to make it better next time." The agents that are watching how everything is going are maybe the more interesting ones to me.

So, back to the relationship question: I see them as highly capable tools. More and more, though, I see them as something that can be proactive and help me figure out what I should be focusing on and doing. Of course, I've been trying out crazy scheduled agents that review my Gmail and do all the personal assistant type stuff. We're also trying to figure out what that means for a whole team. We have a virtual assistant bot right now. We have a thread where we do our standup and share what we did yesterday and what we're doing today. The bot reviews all of it and puts it in a Notion document so we're all on the same page. It can also be proactive and ask, "Did you get this done? Did you get that done?" They're starting to become proactive tools that you don't have to poke anymore. They're just constantly working for you, and that feels like an experience I want to craft.

My last talk at AI Engineer was actually about second brains: something that can look through a bunch of scattered dots and figure out which of them are the gems worth a second look.

The Daily Context: Your bio notes your whiteboard explanations. As teams adopt something like Warp Factories, how do you think about helping people learn new skills in the AI era?

Ben Holmes: I think if you can distill something down into a really short video or a really small diagram, then you really understand it. If you can't do that, you don't understand it super well yet. So I almost take it as a challenge to make sure I really understand something, and that usually leads to a better explanation.

I've done whiteboard shorts for a long time, and those aren't long-form videos. For a while, YouTube Shorts had a one-minute limit, so I had to figure out how to explain a front-end JavaScript framework, which is what the shorts were about at first, in one minute. The whiteboard was the perfect medium for that. I set the limit that I couldn't use overlays on the screen. It had to be a drawn diagram. I could use some stop motion to explain it, but I forced the medium as a way to be more creative and to make sure I was explaining something super clearly.

Now I've been using virtual whiteboards a lot more, like tldraw and Excalidraw, which have become very popular throughout engineering. It's also how I give all my talks now. I just open tldraw, show a diagram, and slowly walk toward the full explanation. That feels like a better way to tell a story than narrating your way through everything, especially now that we're thinking in loops and deployment patterns. Those concepts naturally need a loop or a graph drawn out before people can understand them.

The Daily Context: Warp made a big pivot this year and did a lot of it in public. Why make that process visible and open source?

Ben Holmes: The mission has stayed the same for us: help developers ship better software. The terminal was a vessel for that, and it made a lot of sense. Then, by following the community and seeing what people were working on, we saw that these automations around agents were clearly the next step in that journey.

Honestly, we didn't know what that would look like. We were watching how everyone was using these tools. As an experiment, we first open sourced the terminal. Second, we started building a system to maintain it, because open source is especially tough now with the amount of contributions coming in. If automation is really where we're heading, with agents proactively working on things, we need to build a system around that, even if we don't yet have a product we can help people with. We just needed to figure it out.

So we started building that system to manage the Warp terminal open source project, and before that, to manage our own work. On DevEx, I put out some resources on it and supported Zach Lloyd's articles and explainers, which showed early ideas using GitHub Actions. You can have an agent that triages every issue, an agent that does automatic code review passes, and an agent that implements with computer use. We were stitching those patterns together in an ad hoc way. Then we realized we had a system we could reuse and show to teams over and over again. We could distill it into something where you just sign up and get the triage agent and the review agent we found success with, plus a cloud runner with computer use, because the agent needs to verify its work. That became so obvious to us.

So we were dogfooding our own approach and sharing it as much as possible. Once we saw people really responding to it, we distilled it into something that helps other people do the same.

The Daily Context: Has the community used it in ways you didn't expect?

Ben Holmes: It's pretty early. Our actual factory setup has been around for a month and a half, and we have some early partners working on it. I don't think there have been any surprises yet, maybe just some early success stories. If there's one early surprise, it's that one company is using it for in-depth Android development. Seeing the amount of compute involved, we realized we really need to get a handle on driving down the cost per PR now that other people are using it. That's the first surprise to address. A lot of work still needs to happen, both on researching agent makeup and on improving our platform, so that if you want to run an agent automatically, it runs at a cost and speed that makes sense for teams. Maybe that means open-weight models. Maybe it means simplifying how many subagents you have. We're doing all sorts of research on this right now. That's been the main thing: we're stress testing it.

The Daily Context: What is your AIE talk about?

Ben Holmes: It's about self-improvement, which I mentioned earlier. It's titled "Your Agents Need a Skill to Improve Their Skills."

The Daily Context: Can you give an example of how agents improving themselves has played out in your own work?

Ben Holmes: First, I can explain the system briefly. The idea is scoring them on criteria that you design. It's kind of LLM-as-a-judge. Efficiency during a conversation is one example. Beyond skill improvements, we've been trying to expand this approach to also suggest architecture improvements, so you can have an even better codebase setup.

The Daily Context: How are you making self-improving agents easier for teams to see and work with?

Ben Holmes: We're coming up with a few ideas to make self-improvement more accessible and make it feel more like something you're optimizing. We have graphs that show all the scores and how they track over time, so you can see something like "your efficiency has gone up 10%." We're also playing with a blueprint idea, where you define all the agents currently deployed in your company and make that a little more visual. One of those agents might be the self-improvement agent, watching over the implementation agent. Making it visual makes it clearer that you're building a system and that the agents can all talk to each other. It's not a direct learning from Skill Doctor, but it's an idea I'm bringing in. We could make this more visual and make it feel like something you're actively building, instead of just looking at YAML files or a list of open PRs.

The Daily Context: Warp Factories lets teams run agent workflows from triage to verification. Is there a step teams don't want to hand over to an agent?

Ben Holmes: That's a good question, and you might expect what I'd say. I don't think any of them are really steps people don't want to give over. Triage is kind of a half answer, and I think it's something we need to do better. The idea of triage is that an agent goes off and researches something automatically when you suggest a feature idea, either in a task board ticket or in a chat. We're still figuring out whether that's a necessary step. Spec writing is another one. We have a spec writing agent, but we don't know if it should be separate, and we don't know how useful it is to have that speed bump of "Do you want to work on a spec together?" We may be at a point where it's more natural for a human to ask that. So we're thinking about removing, or at least optimizing, these speed bumps we put early in the design process.

Maybe the better question is about the types of tasks people don't want to hand over, which is kind of the same thing. I think we're still trying to figure out how to close the loop on product development: figuring out the next thing to build. That's still mostly humans saying, "Here's the next feature we should put in the product." This is kind of a blind spot for us too. It would be a nice world if an agent looked at usage data and figured out what your next feature or A/B test might be. That area feels underexplored. It also feels like something people don't really want to hand away yet, because that's product vision and product decisions, and that's kind of the last stand of human touch. We don't want to fully give it away, although if you point an agent at customer data, it might figure out some smart features worth trying. We haven't experimented with it yet. It just feels like an inevitable thing we'll need to reckon with. Closing the loop is something we talk about a lot, and this feels like an important piece. We've only closed the loop with bug fixes. We haven't figured it out for feature building.

A Data Company in an AI World

Kara Silverman — Wed, 14 Oct 2026 12:05:44 +0000

Anekha Sokhal spent nearly a decade trading energy derivatives at JP Morgan, then came to the US on a Fulbright to study data science at Rice University. Today she's the founder and CEO of Moshi, a Houston-based, VC-backed real-time quant platform for institutional finance. Ahead of her talk at AIE New York, we talked about why she calls Moshi a data company, what it's like to build in Houston, and the growing gap between how easy it is to build with AI and how hard it is to run what you've built.

Moshi, Anekha, and building in Houston

Tell me about Moshi.

Moshi is a real-time quant for institutional finance. We provide real-time analytics for financial traders. Every time you open Moshi, you're going to learn something new because of the volume and quality of data that we watch.

A fun fact about Moshi: it takes a human four seconds to read a sentence. In that same amount of time, Moshi has learned 300 new things. We watch more than 2.5 billion data points, and we grow our internal database every minute.

The value we add is that you're going to learn something new about your market every time you look at Moshi, and you're going to be so much more efficient. The problem traders have is that they're trying to keep up with everything, and everything is changing constantly. When you're a financial investor or trader, what you knew this morning is no longer true by lunchtime. The news is changing, Twitter is changing, Instagram is changing. People are flooded with data. And the problem with that is: What is true? What is relevant to me? What can I trust? That's where we come in. We are that one-source platform.

Typically, you would hire a team of analysts to do that for you. I used to be a trader at JP Morgan, and I could never even hire an intern, so I was doing all of this myself. So I built the platform I wish I had when I was trading. Because I'm also a data scientist, a lot of machine learning and statistics has gone into it. We've built all of our own models.

And tell me about yourself.

I'm Anekha, the founder and CEO of Moshi. I've grown Moshi myself. We're a VC-backed startup based in Houston, and the reason we're based in Houston is that I used to be a trader at JP Morgan. I was there for nearly a decade, trading energy derivatives.

I really just love data. I'm obsessed with data. I got a Fulbright scholarship (England has one Fulbright scholarship for data analytics) and came to Rice University to do my master's in data science. I got really into machine learning there and was published at conferences like ICLR.

When I was interviewing for jobs, a bunch of hedge funds reached out, and they all had this same problem of data synthesis. Instead of building it for one hedge fund and doing very similar work to what I did before my master's, I thought: I'm in the US. There's a huge venture industry. It's such an entrepreneurial country. That's why I came to the US. I really wanted to be part of that ecosystem.

I did an accelerator program while I was at university, then happened to meet a venture capital fund that loved the idea of building Moshi for everyone. And that's where we are today. My story is very intertwined with the company's story.

That sounds like a wild ride. What is it like to build in Houston?

The straightforward answer is that you get to be a big fish in a small pond. But I do feel like you have to keep your connections with other cities alive.

We're backed by one of the biggest venture capital funds in Texas. If someone is coming to Houston and wants to know which AI companies or finance companies are there, we're top of mind for everyone, which is great.

However, we can't ignore that, at least in our situation, AI and finance have other cities that are very important. San Francisco is one. New York is one. London is huge in finance, and so is Singapore. Houston is a very good city, especially for energy. I used to be an energy trader, and a lot of energy clients are here, so it's great in that way. But you also need to be connected to the other cities, because the hype is happening there too. San Francisco for the venture environment and the people. Austin has a great ecosystem. London has a fantastic finance community. And New York, I feel like, is coming up a little more in startups now, and it's obviously huge in finance.

So what is it like building here? There's a fantastic community. Some of the nicest people I've ever met in my life are here, so many helpful people. You can really make a difference here. But there aren't always going to be as many events, and it's not as easy to meet people because it's a very spread-out city. You don't have those shoulder-to-shoulder conversations you'd have in a walking city.

A data company in an AI world

So, what is your personal relationship to AI?

For my undergrad, which was a very long time ago now, I studied MORS (mathematics, operational research, statistics and economics) at the University of Warwick, and it was not a sexy degree. If I told anyone I was studying MORS at Warwick, they were just like, "Oh my God."

Because I loved statistics and data, I went on to trading. It was a very natural transition. Trading is just data and machine learning. And when I was doing it, no one cared. No one thought what I was doing was interesting or sexy.

I remember once meeting a family friend with my best friend, who was studying law. I'd gotten my internship at JP Morgan at the time. The family friend asked me, "What about you? Do you want to be a tutor or a teacher?" Then I looked at my friend and said, "Wow, you're incredible. You're going to be a lawyer."

It's so different now. I graduated in 2014, and now if you say "machine learning" to people, you're blowing their minds.

So my connection is that I was in it before it was cool to be in it. Now we have this separation between machine learning and generative AI. I call Moshi a data company because I love data. We have data scientists. I'm obsessed with data. One thing we do that's very different from most companies is that we do so much data engineering and machine learning, and then generative AI sits on top of all of that. So my relationship is: I want to understand the data, and generative AI is how we communicate it to people in a more human way.

The bar for calling a company an AI company has gotten way higher. It's not a differentiator anymore.

I finished work super late last night, and to decompress I went on Instagram, because it's the most mindless scrolling. Embarrassingly, I was on it for 40 minutes, until I got the timer notification. This was at 1 a.m. In that time, I got 25 ads for AI.

I screenshotted every single one and sent them to Rick, my head of growth, and said, "Rick, we cannot advertise on Instagram." Twenty-five companies in 40 minutes calling themselves AI, AI for finance. It's so easy to generate. How can I trust them? I think that says everything about what it's like to build today.

I'll never forget driving in from the San Francisco airport earlier this year. There was nothing but AI billboards.

There was a time when our investor told us we needed a .ai domain. I don't really believe in all of that, but we got the .ai domain, and now I don't even call us an AI company. I say we're providing a solution, and part of it does have generative AI, as you'd expect. That's an expectation now. If you have an LLM, it's not cool. It's the expectation.

It's the same in media relations. Five years ago AI was unique. Now the best way to get a company coverage is to not talk about its AI.

It's definitely different. Before Moshi, while I was at university, I tried to start another company, which I closed down, called Jewel Vision. My parents are in the jewelry business, and I wanted to give people the ability to try on jewelry in 3D without even needing models.

It was too early. The technology didn't exist. It was 3D computer vision. To this day, I have recruiters asking, "Can you come to Apple's recruiting events? Can you come to Tesla's?" because it's not a skill people have. I tried to build a company doing it for a year.

That was super cool AI. That is pure 3D computer vision. But if you have an LLM agent, to me, that is not AI. There's a difference.

Prompt fatigue and where humans come in

Where are you seeing prompt fatigue show up?

I want to explore how our work is changing and where humans actually play a part. One thing I've found is that people are now just copy-pasting Claude output to each other. So where does that stop?

If I think about the whole workflow of where we are today: people want solutions that make them more efficient, make them better decision makers and integrate their workflows, and they're using generative tools to do that. The questions that come up a lot for us are: How can I trust the output? How can I connect everything? And what about the data? Where are you getting the data from?

The biggest thing we've realized is that the foundational companies are making it feel like it's so easy to build. And it is easy to build. You can get a $200 subscription to any of them and build a dashboard for yourself, and it will be fantastic. I do it for my wedding planning. I have my little dashboard. Now try to make that production-ready. Try to make it run for more than one person. If I want to send that dashboard to my mom, who doesn't have a Claude account, that's the problem I want to talk about.

What happens when you try to take that into an enterprise?

When you try to implement that in an enterprise, you actually need to run the system. And when you need to run the system, you need DevOps, you need error handling, you need to pay other people to run it. That's the bit no LLM is going to tell you upfront.

So yes, you can spend weeks building a dashboard on Claude. Now try to make that dashboard available to the public. Now you need DDoS protection when you host it on a website. Now you have to make sure your API keys are hidden and not in the code. You have to make sure you're not showing your code on the website. You have to get a database. You can't just host it locally. Those are the things that end up wasting people's time.

Instead of managing it yourself, giving that job to a human, or even doing it the old-fashioned way, you've built it yourself on Claude. But now you have to run it. So you're dealing with prompt management, because every time you want to make a change, you wanted to change one thing and you've gone and changed 50 other things. That's what I wanted to talk about: the prompt fatigue, and the realization that I can build really easily, but I cannot run a system as easily.

So where do people come into all of that?

I think companies are still figuring that out. CEOs and CTOs are constantly posting about how, oh, let's fire everyone, because now we only need two or three people to do that job. And I'm sure you've seen the AWS outages and all these other outages. This never used to happen.

What we're seeing now is a confidence everyone has been given by these AI tools: I can build anything I want. But the thing people are forgetting is that every single person has access to the same tool. Now you're going to tell me, "But I've got a different skills.md file, or a different Claude connector, or a different MCP." Yes, I agree, those are different. But you are still pulling and pushing the same data. It's still the same model. If my engineer asks Claude a question, they're going to get the exact same answer, even if the prompt changes a little.

So where do humans come into this? It's the judgment piece. Claude can't build Moshi. Why? Because I had to take 10 years of derivatives trading, and I'm a data scientist as well, so I knew how to build the system. I had people sitting with me when I was 19 or 20 years old at JP Morgan, spending three hours explaining systems to me. I took all of that, and that's what I'm bringing to my agents. A lot of people are not doing that.

I think one thing we're going to come across is that things are going to break more easily, because of the trust people have in these systems. One tool says, give it all your bank details, your credit card details and your email details, and it will go and run everything for you. We don't do any of that internally, and we have engineers from Microsoft and Meta. No one on our team does that. You can ask them to, and they won't. It's similar to the CEOs of all these tech companies who won't let their kids use social media.

I think there's going to be a sort of awakening. We have big enterprise clients who come to us and say, "Why do I need you? I've got an Anthropic subscription." And I say, try to run that same thing for more than one person. You're going to have security issues. You're not going to want to manage it. You're supposed to be trading and watching the markets. You're not supposed to be telling Claude that FOMC is today, not tomorrow.

There's going to be friction for a while. It's like a purge. People will do all the AI, and then they're going to realize: actually, I want to see a spreadsheet that I have to use my brain to check. I want to remember someone's phone number. I want to see a PDF that isn't perfect. I think you just have to let it happen.

The interesting thing for us is that enterprise software is changing. We have clients sending us their prompts. We had a client today who said, "Hey, can you just run this for us?" And I said, that's not what we do. And they said, "Yeah, but I don't want to run it anymore. You guys do a better job. You do all the trust, you do all the things." People are interacting with us like that now.

Did you do it?

We had to build that, because we didn't know we needed to before.

For us, having been a trader for so long, the human brain is so much more advanced than machine learning. The number of connections we can make in real time, the power of the human brain: there is no data analytics that powerful. We're trying to get closer and closer to that human way of thinking, but machines are not there yet. So why would you want a machine to make such important decisions?

Claude is an incredible tool, but it's incredible at summarizing what you tell it to do and explaining the prompt. What it's not great at is making those second- and third-order connections, which is what humans are great at. It's seeing around the corner, being able to say, "This is what I think I should do," or "This is what I don't see." Those gut feelings, those human interactions.

Can you give an example?

A simple one is hiring. So many times LinkedIn will ask, "Do you want us to AI-interview this person?" And I'm like, no. I want to interview this person.

When I was hiring a couple of weeks ago, I ran a simple experiment. I ran every résumé through Claude, because I wanted to see who Claude would hire versus who I would hire. It was so different. Claude came up with a scoring system and said: based on these people's answers, how they use AI and everything else, these are the numbers one, two and three you should hire. I thought, okay, that's really interesting.

Then I brought everyone to the office. One thing I do is spend three to four hours with each person, because I want to see how you think. Each candidate met me, our team members and our investor. Every person agreed on the ranking, and it was not the same as Claude's. The person Claude ranked first wasn't even in our top five.

There's a company that raised a ridiculous amount of money, something like $30 million, and they're a one-person company. Their whole thing is, "We are a one-person company." On social media they're doing incredibly. But you look at the reviews, and every single person says, "I just want to speak to a human. There's 99% I can do myself, but for this 1%, I just want to speak to someone." You can't replace that.

That's what we're trying not to do at Moshi. We give you the data and the information. We do all the data engineering you wouldn't know how to do, because most people are not trained data scientists, data engineers and quants. But ultimately, you have to decide what trade you want to put on and how you want to manage it. We can give you the conviction and the data. That's what we're selling: you are going to be so much more confident in your trades, because you have blind spots. Why do you have blind spots? Because we watch way more data than is humanly possible. We'll tell you the connections, but ultimately you're the one who's going to put on the trade and sleep with that risk.

I think it's a really interesting time. Social media makes people see things a certain way, but the reality is that AI needs to be a tool for certain things, not a tool to replace everything.

So much of this resonates with me. I can't tell you how many times I've gone to a brainstorm or a workshop and people have shown up with the same ideas, because they all used the same system. It's like everyone shopped at the same stores and showed up in the same gray top and jeans.

It really is. And one funny thing about what you just said, everyone coming up with the same thing: we gave about 20 people a hiring assessment, and we told them to use whatever AI tools they wanted. We asked them to come up with a list of 10 target people we should speak to. Real people at real companies, from LinkedIn.

Bear in mind we interviewed people across the US, and LinkedIn has hundreds of millions of people. Can you believe that 90% came up with the same 10 companies?

I'm not putting anyone down, but we were hiring salespeople and a head of growth. If you're doing outreach to clients, think about that. Yes, I agree AI can help you. But think about what your difference is.

There's an idea I keep coming back to: the iPhone moment that Major League Hacking (MLH) CEO Swift has written about. Suddenly everyone thought they could be a photographer, but you still wouldn't let someone with an iPhone shoot your wedding, you know?

There's a lot we can do with AI, and there's a lot we don't have to do with it. It's simple: if I can get Claude to do something, then I will get Claude to do it. I'll get Claude to do the 90%. But that 10% is exactly where you don't want Claude.

You don't want AI there, because at the end of the day, if something goes wrong, you want to go talk to someone. You want to be able to say, "Hey, let's figure this out." You don't want to just talk to your chat model.

Inside the $30-a-Week Self-Improving Agents Eliminating Enterprise Back-Office Friction

Kara Silverman — Wed, 14 Oct 2026 12:05:21 +0000

In the high-stakes world of enterprise AI engineering, the default assumption is that reliability costs money. Scaling accuracy usually demands larger models, endless retry loops, expensive visual processing, and human-in-the-loop fallback teams.

At Brainbase Labs, Engineering Lead Milind Shah is quietly proving the opposite.

Ahead of his Wednesday afternoon session at AI Engineer NYC (“Hill Climbing Back Office Agents”), Shah sat down with The Daily Context to unpack a case study with Domino’s Pizza that flips the standard AI economics curve on its head. Facing a costly, manual process to track competitor pricing and promotional discounts across local fast-food chains, Domino’s deployed autonomous browser agents managed by Brainbase.

Over nine iterative versions, the system’s task accuracy soared from 41% to 91%. Simultaneously, operational costs plummeted from thousands of dollars per month for an external consulting team down to just $20 to $30 per week—with inference costs dropping to $1.16 per source.

The story behind that 50-point accuracy jump offers a masterclass in pragmatic AI engineering for the 1,000+ technical leaders gathered this week at the Sheraton Times Square.

Phase 1: Defining “Correct” Before Writing Code

Before Brainbase touched a single prompt or model parameter, Shah’s team had to solve a foundational problem: defining what success actually looked like.

Domino’s had previously relied on human analysts to manually inspect competitor websites, collect local coupon codes and promo pricing, and construct structured spreadsheets for executive decision-making. To replicate this with software, Shah sat down directly with non-technical business teams at Domino’s to decode their reference spreadsheets.

“They already knew what a good spreadsheet looked like,” Shah explained. “We used that to create an internal benchmark dataset of questions and expected outputs. Without an internal benchmark, you can’t prove accuracy numbers or tie improvements to cost reductions.”

This benchmark became the foundational ground truth against which every prompt tweak, model swap, and agent instruction was scored.

Phase 2: The Coordinate Hack & Model Optimization

Once the benchmark was established, Brainbase systematically tackled the agent’s execution bottlenecks. Surprisingly, the single largest jump in accuracy didn’t come from a bigger LLM—it came from stripping away visual inference.

In early iterations, the browser agent relied on visual processing to locate buttons and navigation links on competitor websites. This approach was slow, expensive, and prone to misclicks. Shah realized that fast-food ordering interfaces have static UI layouts. Instead of making the model parse image tokens to find a “View Deals” button on every page load, Brainbase hardcoded fixed X/Y screen coordinates for recurring UI elements.

“Instead of having the agent figure out where the button is by reading the image, we told it the exact X/Y coordinates,” Shah said. “That eliminated expensive image-reading inference and single-handedly produced our biggest accuracy gain.”

Concurrently, Shah experimented with model selection. Rather than defaulting to Anthropic’s flagship Claude Opus 4.6, the team discovered that tightly specified, highly structured prompts paired with Sonnet 5 delivered superior accuracy at a fraction of the token cost.

Phase 3: The Unsolved Frontier of Self-Improvement

Brainbase’s platform now powers more than 10,000 internal agents across enterprise back-office workflows—ranging from invoice exception clearing to inventory transfers. While Brainbase’s automated “Hillclimbing” engine can automatically detect and remediate failure modes like wrong tool calls or hallucinations, Shah remains candid about the limits of current automation.

Complex browser navigation traces still require human engineering eyes. “Automated improvement works for tool calls and hallucinations,” Shah noted. “But browser navigation is still too intricate for auto-improvement agents today. Engineers still have to manually review agent traces to find where browser actions break down.”

For teams working with tight evaluation budgets, Shah offers straightforward advice: don’t overcomplicate your initial evals. “Deterministic eval criteria go a long way at low inference cost—checking for hallucinations, tool-use correctness, and adherence to knowledge bases.”

The Engineer Behind the Agents

Shah’s path to leading engineering at Brainbase spans software engineering roles at DoorDash and Adobe, scaling the code-data business at Merkle from zero to eight figures, and co-founding the evals platform Pipelines.

Despite building autonomous systems that handle complex enterprise workflows, Shah maintains a grounded, reflective view of the AI revolution.

“I’m optimistic overall, but I think about skill atrophy a lot,” Shah admitted. “If I’m using AI to write code today, and tomorrow I’m using it to write everything else, you risk losing those core skills. A good AI engineer still needs strong engineering fundamentals first.”

Milind Shah speaks on “Hill Climbing Back Office Agents” on Wednesday, October 14 at 2:50 PM ET at AI Engineer NYC.

Kepler's Vinoo Ganesh on Building AI You Can Trust

Kara Silverman — Mon, 12 Oct 2026 21:04:02 +0000

Vinoo Ganesh is the CEO of Kepler, which builds AI for industries where accuracy matters, starting with finance. Before Kepler, he spent seven years at Palantir, built and sold an alt data company, and led business engineering at Citadel.

Ahead of his AIE New York sessions, including a fireside chat with Jack Woodroffe on Tuesday at 2:55 PM, he sat down with The Daily Context to talk about building AI people can trust, why Kepler's model never writes a number, and why he doesn't believe a bigger model is the answer.

This interview has been edited for length and clarity.

The team, the talk, and New York

The Daily Context: Tell us a little about your background and about Kepler.

Vinoo Ganesh: I'm Vinoo, Kepler's CEO. I spent seven years at Palantir leading Spark and the compute side there. Then I built an alt data company, grew it, and it got acquired. After that, I was Head of Business Engineering at Citadel and really focused on how we build software products and data products for portfolio managers, which, ironically, is very similar to what Kepler is doing now.

It's a heavy ex-Palantir team. Sarah was one of the engineering leads for the Palantir Ontology. Chris was at Palantir for 15 years, doing a lot of optimization. We have folks from Meta, Reflection AI, Datavant, Goldman, Oddity, you name it. So, it's a team that has spent a lot of time building products in environments where the accuracy and fidelity of information and data matter.

The Daily Context: That's an incredible pedigree, but it's also hard to convince people to walk away from those companies. How do you approach recruiting?

Vinoo Ganesh: We've been very lucky in that there are people who trust us, who've worked with us before, and who feel very positive about joining us again. But honestly, I make Eddie's job very hard at times, because I'm always saying, "Let's recruit these insanely talented people."

Ultimately, it comes down to a few things. First, the problems we solve are super technically deep. In a world where there are things that look like ChatGPT wrappers, a very deep set of technical problems is super appealing to engineers who are motivated by that.

Second, we're really lucky that our problem space has a huge element of importance to humanity. Yes, we started in finance, but AI you can trust, or enterprise intelligence you can codify, is kind of the last remaining alpha-generating milestone. Kepler is really powerful in that sense: if everyone has the same access to the same tools, what alpha or uniqueness remains? It turns out that's an existential question for a lot of folks.

Third, talent compounds. We have a really strong pedigree team, and people like working with very strong engineers. We have an insanely high bar for talent; you can ask Adil about that. We try to be very fair in our interview process, but we do have people who are intimidated by the backgrounds. We've gotten that feedback: "Do I want to work for an ex-Palantir, ex-Citadel team? Sounds scary." I think we're approachable and reasonable.

The Daily Context: What is your personal relationship to AI?

Vinoo Ganesh: I think "cautious consumers" is probably the best way to describe us. AI is really helpful for vast synthesis of information, especially in non-deterministic ways. If I have a bunch of queries I just want to understand at a high level, or get a vibe for, AI has been very powerful for that.

The challenges come in when people use AI in ways it shouldn't be used, like deterministic work, or work where accuracy is materially important. AI is very positive in a number of ways, but it also agrees with the vast majority of things you say. It doesn't follow steps, even when you ask it to. And it's a moving target. When models are being released at a pace of roughly one a day, keeping up with that rapidly changing environment is super hard. So it's a powerful tool and accelerant when it's deployed in the right way.

The Daily Context: What is your AIE New York talk about?

Vinoo Ganesh: We see the promise of AI so clearly because we're deep in it every day. The simple reality, though, is that deep exposure to and deployment of AI is still in its infancy at most companies. Even the ones that think of themselves as super AI-forward are normally just throwing Copilot on a few queries, and that's it.

Our question is always: why? What's the blocker to getting to the level that we, our team, and our peers use AI at? It comes back to the question of trust. So the talk is about how you build trust in an AI system, and our view on it: to really trust an AI system, you can't just use AI by itself. You need infrastructure around it. That infrastructure uses AI for the right scenarios, uses other tools for the right scenarios, shows proof, and, finally, is deeply embedded in the systems you already work in. How we go from early adopters to people using it in depth, every day, in 95% of cases, is the exciting space we'll talk about and that we're working on in detail.

The Daily Context: Why New York over San Francisco?

Vinoo Ganesh: As a forward-deployed engineer, I've learned that being close to your customers, geographically and emotionally, matters a lot. Our initial customers happen to be based here in New York. Finance is predominantly here, but I also think the urgency and importance of this problem space are uniquely captured in New York, where the speed of development, the speed of technology adoption, being first, and being smarter all matter a lot. SF is really powerful in a number of ways, more on the research side, but at Kepler we're very focused on the applied side, even though we do a good chunk of research ourselves.

How Kepler works

The Daily Context: My research suggested Kepler checks citations against sources in real time, while the output is streaming. How does that work?

Vinoo Ganesh: We actually don't do it while streaming or in real time. The way we operate is that there are certain things deterministic tools will always be better at. So we run massive preprocessing steps that extract what we call facts. Those can be numbers, entities, people, places, and things. We annotate those facts with metadata that describes each unique fact. That can be relationships, or, for a number, it can be magnitude, scale, whether it's dollars or pounds, and so on.

The second piece is that we persist all of this data with provenance records. As data flows through the Kepler platform, the platform and agents are architected so they're consistently writing to a provenance ledger that, at any given moment, holds effectively every previous step taken. So we can see every twist, every turn, every determination an LLM makes, and we can even see deterministic changes. If two numbers get added together, we persist that output. If two numbers are multiplied, we persist that product. And we can show you the chain of transformations the number went through.

So a lot of it comes down to very detailed preprocessing and very detailed engineering up front, which then compounds because we can switch between the deterministic and non-deterministic pieces.

The Daily Context: Walk me through Kepler's first public equities ontology. Why was it important to build?

Vinoo Ganesh: Great question. Humans don't think in terms of rows and columns in a database. If you think about your Facebook friends, you probably think in terms of relationships: I am this person's friend; I believe this person is this many years old. We think in these complex, relationship-driven ways.

What's interesting for us, especially in the finance ecosystem, is that being able to codify and entity-resolve even something as simple as company names is a really hard problem. For example, Palantir and PLTR obviously refer to the same thing, but most systems can't easily entity-resolve them. That problem compounds when it's a person saying something. If I'm the CFO of a company and I mention Palantir, and it's transcribed incorrectly, how do I do any kind of canonicalization or entity resolution?

What we realized is that LLMs are really good at thinking through how to solve things, but they're not necessarily as good at doing the solving themselves. Developing the ontology allowed us to do a few things. First, it let us give LLMs tooling to interact with that ontology, so the LLM can do what it does best and the tools can do what they do best.

It also lets us be very mindful from a cost perspective. We're in an era post token-maxing, where using as many tokens as possible is no longer the cool new thing to do. Getting the best ROI out of each dollar you spend on tokens becomes really important. Kepler is predicated on using the right tool for the job, and sometimes a deterministic system, like knowledge graph traversal over an ontology, is a much better solution than a potentially multi-trillion-parameter model trying to do anything and everything. So the ontology lets us represent the preprocessed data in a way that both LLMs and Kepler's own execution engine can traverse pretty easily.

The Daily Context: The applications to finance are super clear. What other industries are a good fit for Kepler?

Vinoo Ganesh: Any industry that's regulated, where material accuracy matters. I've jokingly called these the "non-vibes industries." Think of something like insurance claims adjustment. Insurance is a very interesting potential area of expansion for us. Manufacturing, too. Even legal. Legal is interesting because if you think about the data, like case briefs, there's an inherent structure to it. There's precedent; there are cases like A v. B; and there's a time-based nature to it: this happened before that, so this precedent overrides that one.

So any industry where the accuracy of data matters is a potential opportunity for Kepler. That's why we've positioned Kepler as more of an engine than a particular vertical SaaS play. We now have pull across a bunch of different industries. Defense has been an interesting, more recent area, and fund of funds as well. Any of those spaces.

The Daily Context: Part of Kepler's promise is that the model never writes a number. How does that work? And Kepler has a connector inside Claude. How do you make sure that promise holds when Claude is delivering the last mile?

Vinoo Ganesh: This is probably the best question I have ever gotten, because we are desperate to talk about it, and no one lets us.

A couple of things. Kepler having an MCP was actually a request from Anthropic themselves. It wasn't a deliberate strategic positioning for what Kepler wants to be, because you're right: if Kepler runs through an MCP, the model can still hallucinate. It can pull the wrong number, and a bunch of other things.

Here's how we solve that problem. Kepler in Claude ships with apps, so what you're seeing is a rendered application, not Claude giving you text-based output. That rendered application has full auditability and traceability. We're passing what is effectively a blob of code or a blob of visualization back to Claude, and all Claude does is output it. It's almost like an API call, and the apps feature in Claude renders the Kepler UI. That's what makes every number auditable and traceable.

How do we ensure the model doesn't produce numbers? It's because of the invariant we've built into the platform. By preprocessing data and information, we assign identifiers to facts. What the model knows how to do is use deterministic tools that retrieve those facts. Think of it almost like a getter or a setter: I can do a lookup from a table, and the model substitutes that lookup in.

Right now in finance, the facts we care about are things like company names and numbers. But in law, we could enforce that same idea for court cases, to make sure each case actually exists before the model can return it. So right now, if the model can't resolve its output against the lookup, it rejects it. It can't actually state that fact.

The Daily Context: What does it say when that happens?

Vinoo Ganesh: It isn't included in the output at all. In practice, it looks like facts disagreeing. Say someone from Google said on an earnings call that Google's earnings were $1 last year, and the 10-K reports $100. Those are conflicting facts. Rather than trying to pick one, the model realizes that, upon resolution, it can't identify that number appropriately. So it has to present it to the user and say it's getting conflicting information.

Sometimes that's alpha, and it's right. If someone is saying something different, you could have new sources of information, and that could be alpha in its own right. Sometimes it's actually a mistake.

The state of the industry

The Daily Context: What else do you not get asked about that you really want to talk about?

Vinoo Ganesh: I have one that's more of a comment on the state of the industry. In tech as a whole, there are a lot of waves, and we've seen the first wave of companies raising a lot of capital very quickly. They're in the land grab right now. We're very much in the early innings.

A lot of people ask me who Kepler's competition is, and it's a somewhat challenging question to answer, because, at least to my knowledge, there isn't a set of people out there trying to build this harness or this product in this way. So our competition looks a lot like vertical SaaS companies that have to build the airplane while it's in flight. More importantly, we're still early enough that this is a thesis: here's how we would solve this problem in the data.

I think we'll see a lot of other approaches. There are labs trying to remove hallucination by throwing as many tokens at it as they possibly can. I inherently think that doesn't make sense. A probability machine will continue to be a probability machine, even if the hallucination rate is 0.0001%. Ultimately, we're an execution engine. We look more like a database than a vertical app company.

Other questions are around where we are in this AI cycle, and the desire to use LLMs for everything. I don't think a bigger, badder, more powerful model will result in better output, any more than driving a Hummer from your home to the grocery store gets you to the grocery store in a better way. It doesn't make sense to me.

The last thing people don't seem to care about as much is that a lot of people are very okay sending private information to the labs. That's the weirdest thing to me, because coming from Palantir, data security was the most important thing. I've had meetings with major financial institutions that said, "Well, we signed an agreement, so we're just going to send all the customer data out to the labs," which is mind-blowing to me.

So there's a big question around what the future looks like when everyone is using this technology, and when alpha is much harder to come by. Those are open questions right now. That's exactly what I want Jack and others to talk about, and that's where I think the opportunity really is.

The Daily Context: What do you recommend for companies that want to apply AI to sensitive data?

Vinoo Ganesh: Ultimately, the question is how you maintain your edge. That's the most important thing. My frustration is that a lot of these organizations are digging their own grave, and I don't understand why that's the path they want to go down. Your own data is your alpha.

The Daily Context: More tactically, is there a way for a financial company working with inherently sensitive data to use AI without sending that data to the labs?

Vinoo Ganesh: I would deploy on-prem. The crazy thing is that deploying a model seems like the most insane thing in the world. But you spend two seconds thinking about it and realize you can pretty easily run the model in your VPC and do any kind of editing of these agents on your own platform. It sounds really scary until you do it, and then you realize it's not insane in any way.

The Daily Context: If you were advising a 20- or 21-year-old recent college grad looking at engineering and AI engineering, what would you tell them?

Vinoo Ganesh: In any major technical shift, there's one big driving question. Here, it's whether AI is just another disruptive technology — not the harbinger of the end of the world, just another disruptive technology we all learn to live with. I think that framing makes people feel a lot less scared.

The Industrial Revolution was a huge technological shift. The dot-com era was a huge technological shift. There were a bunch of these material changes, and none of them ended the world. What I did was figure out how to adapt to the new way of doing things and the new set of processes.

My belief is that people in college right now have a very unique opportunity to rebuild the world through the lens of these new technologies. I wonder what the AI version of McKinsey will be. What's the AI version of Lazard? I'm not saying those companies will be gone. It's more that there's a huge opportunity to get very familiar with using a technology to fundamentally rewrite how industry works.

Personally, I'm really excited about medical use cases, if we can solve the accuracy problem. That, to me, is a huge opportunity. How many prescriptions are filled incorrectly because someone can't read a doctor's handwriting, and a patient gets 10x the dosage they should have? Those are all huge opportunities, and I think the next generation has a unique ability to build these future technologies.

Q&A with Al Chen, Software Engineering Leader at Cisco

Kara Silverman — Sun, 11 Oct 2026 19:45:03 +0000

We interviewed Al Chen for year's AI Engineer NYC event. Al is a Software Engineering Leader at Cisco, where he leads the customer engineering team for Splunk Agent Observability (formerly Galileo). He specializes in production AI infrastructure, enterprise deployments, and agent engineering.

We talked about what it really takes to deploy AI agents inside major banks, from navigating security reviews and compliance requirements to preventing data leaks and designing agents that fail gracefully.

Here's our conversation.

Tell us about your talk.

The talk is a bit of war stories: our experiences deploying our platform inside various banks, how we get through the different compliance, security, and legal gates, and some of the potential solutions on the path forward. That's the main gist, and I'll walk through a few deep-dive examples.

How did you get into this work?

My path into AI was kind of circuitous. Like many people out there, I wasn't doing this even three years ago. I come from the world of spreadsheets and productivity software. I got the AI bug in 2023, early 2024, and I was super skeptical at first. I saw ChatGPT and thought, this is great for small use cases. Then I dug in some more and got really into the weeds with LangChain and different AI frameworks. About a year ago I had the opportunity to join a small startup, Galileo, which was acquired by Cisco. We're deploying our software within the largest banks and financial institutions in the world, and that's how I ended up dealing with CVEs on an everyday basis.

You've taken agent tooling into five Fortune 100 banks. What breaks first when a stack built for SaaS hits a tier 1 bank security review?

The first thing that breaks, and I'm going to talk about this at the conference, is dealing with CVEs: Common Vulnerabilities and Exposures. It's probably the number one thing all of our banks deal with. Before we even touch one piece of hardware or software, getting through that gate is the biggest hurdle. Until you have a really buttoned-down process for remediating vulnerabilities in your software, it's a showstopper for getting anything through the front door of a bank.

My team is on the frontlines of dealing with these day in, day out. It's a full-time job. Really, it's a full team's job. The problem is each bank has its own unique process, its own policies, and its own scanning software to determine what counts as an exploit or a vulnerability.

My job is to figure out how to systematize this, because we don't want to deploy bespoke solutions for every single customer. How do you do it all as one cohesive process? That's definitely been a challenge, and I'm curious to hear how others are doing it. Even at a big company like Cisco, we're probably still figuring out how to deal with different customers.

How do you split your time day to day between execution, strategy, learning, and problem solving?

It's the typical startup phrase: you're building the plane while you're flying it. I didn't know anything about CVEs or Kubernetes clusters before this role. I'm using AI to learn about these things, but until I really grasp the concepts, I still have to do some of my own self-teaching.

Building AI Analysts for Wall Street: A Q&A with Compound Co-Founder Yijia Liang

Kara Silverman — Sun, 11 Oct 2026 19:43:48 +0000

Building AI Analysts for Wall Street: A Q&A with Compound Co-Founder Yijia Liang

Ahead of AI Engineer New York, The Daily Context spoke with Yijia Liang, co-founder of Compound, about building autonomous financial analysts, why general-purpose AI falls short in finance, and what it takes to earn the trust of high-stakes users.

The Daily Context: Tell us about yourself and Compound.

Yijia Liang: I'm EJ, one of the co-founders at Compound. We're building AI for Wall Street and finance, specifically private equity. We're building long-running autonomous analysts that behave and perform tasks the same way they would if you delegated the work to a human.

Previously, I was a vice president on a private markets team, investing in late-stage companies like Rippling, Databricks, and ClickHouse. Before that, I was an investor at Y Combinator's growth team, and I began my career at Bain in its private equity group.

I've spent a lot of time in buy-side finance, and I've found that the most interesting parts of the job aren't where people spend most of their time.

What's exciting about dealmaking is understanding a company, determining whether it's a good investment, and deciding whether to invest. But people spend most of their time building Excel models, PowerPoints, and Word memos, along with all the research and orchestration that goes into a deal.

That's what Compound is intended to help with. We're building much higher-quality outputs than you'd normally get from Claude or ChatGPT off the shelf, specifically for financial services.

The Daily Context: What will you be talking about at AI Engineer New York?

Yijia: The talk is about long-running autonomous analysts for finance and how we've approached that problem.

Recently, we've seen personal agents reach a kind of takeoff velocity because they've crossed a certain threshold of quality and productivity. But in finance, a lot of people in private equity and banking are still using AI as a chatbot. They're using ChatGPT or Claude to answer questions, or as plugins in Excel and PowerPoint.

That's very different from saying, "I'm going to delegate an entire workstream to you and come back 12 hours later."

That's what we're focused on building. The talk will walk through the technical details of how we're approaching that problem.

The Daily Context: When you set out to build an AI system that actually does financial analysis, rather than another general-purpose agent, what fundamentally broke?

Yijia: From day one, our focus was on how we could credibly produce ROI for the end user. In our eyes, that was the analyst or associate doing the grunt work. ROI means getting hours back in your day. Instead of you doing the Excel model or PowerPoint, Compound does it for you in exactly the same way you would have done it.

Initially, the models weren't good enough. We spent a lot of time figuring out how to get o3 or Sonnet 3.7 to work better with Excel. The models improved, but we realized three other things were broken.

First, the tools themselves. Excel, PowerPoint, and Word are built for humans, not agents, and certainly not cloud-based agents running hundreds of parallel processes.

Second, collaboration. So much of finance happens in small teams, with multiple people providing opinions and feedback. But AI has been so single-player. Even if you're giving an individual a few hours back, you're not solving the bigger problem of how to give hours back to the team or firm.

Third, context. Even with skills, memory, and dreaming, off-the-shelf models aren't perfect for finance because of the sheer number of unstructured files involved. Code is largely characters and Markdown files. Finance involves PDFs, Excel spreadsheets, and PowerPoints, which are much richer in information.

The Daily Context: What does good actually look like? Is it getting to the right number, showing the steps, or something more?

Yijia: There are two ways I think about good.

The first is reaching this almost magical threshold where someone looks at your output and says, "Yeah, that's what I would've done," and gives it a checkmark.

We're already seeing that in coding. Basically every software engineer I know is using AI to drive their coding. They look at the result, decide it looks good, and check it off. If we can produce financial models and PowerPoints that reach that same bar, that's great.

The second is whether AI passes the sniff test for judgment and critical thinking. That's where I don't think we're there yet.

Evaluating deals requires creativity, judgment, and self-iteration. When you work with an incredible human analyst, they often have next steps and follow-ups ready, even without being asked. They'll proactively address issues before you raise them.

AI isn't there yet. It produces something, and the implicit understanding is that there isn't necessarily a follow-up or next step.

When you work with a really good human analyst, they've usually already anticipated the three comments you're going to have. That's the level of quality we aspire to.

The Daily Context: Fast-forward five years. Will AI remain embedded in Excel, or will the entire interface for financial analysis change?

Yijia: Excel has survived for 40 years for two reasons. It's an incredible piece of software, and it functions as both a calculator and a canvas for financial firms to express ideas.

Some firms have 40 years of financial history stored in Excel. Unless you figure out how to map that information into a structured database, it's going to be difficult to move away from it.

Five years from now, I think Excel will still exist. But most firms will have moved toward having a data warehouse and will be migrating firm-level information out of Excel.

I think we'll see people relying more on HTML or another type of artifact that isn't Excel.

But people will still want auditability and traceability. How do you step into calculations and fully trust that a number is correct?

That's what Excel is great at. You can see the math, and the engine is deterministic. Whatever replaces it needs to offer that same trust.

The Daily Context: Do you think vertical AI is the future, or will general-purpose models eventually serve specialized industries like finance?

Yijia: I think about AI in terms of how it augments what differentiates one firm from another. For some firms, that's their human capital, the history of deals they've done, their relationships, or their platform.

You can accomplish that by building your own AI, using general-purpose tools, or using domain-specific solutions.

I strongly believe there will be vertical-specific solutions in finance, and multiple winners, because the category is so massive.

The reason is the unique nature of the data. Look at the most successful financial businesses before AI: Bloomberg, Capital IQ, and FactSet. They're built around supplying data to customers.

But what's becoming more interesting isn't just the data you provide, which will become increasingly commoditized. It's how you help firms harness their own data.

I think every firm will eventually have a data warehouse, a data lakehouse, and an ontological layer. I call it a "company brain."

The big winner will be someone who can operate that brain and become a real partner as firms deploy AI across their businesses.

The Daily Context: Financial services companies handle enormous amounts of sensitive information. How should they think about sharing data with AI models?

Yijia: Zero data retention needs to be table stakes. Model companies should not train on your data. That's our stance.

We have zero data retention agreements with every one of our model providers. We've chosen not to roll out new models if they don't have that protection available.

The next question is how you build valuable systems on top of that data. Firms will want to own those systems themselves.

You have to start with the most stringent version of security. Start from least privilege rather than most privilege.

The Daily Context: What's your advice for AI engineers building tools for high-stakes users?

Yijia: Two things.

First, you get one or two shots on goal before finance users form an opinion of your product. And that opinion is basically set in stone for a long time.

When Claude came out with its plugins and Opus 4.5, the narrative on Wall Street became that Anthropic had won the enterprise. Even when newer models came out from competitors, that narrative didn't change much.

Finance users treat AI products almost the same way they treat human analysts. They have incredibly high expectations. If you can't deliver the first or second time, they're going to write you off.

Second, you have to constantly reinvent. There's creative destruction happening in product development because AI is moving so fast.

Every week or month, we reevaluate what we've built. Is there an opportunity to rewrite it, make it better, make it faster, or even start from scratch because of how much AI has improved?

The Daily Context: What's something you wish more people would ask you about?

Yijia: I don't think people talk enough about eval construction in finance.

We're focused on the outputs that move the needle for customers. Our evals focus on entire workflows, from information ingestion and research gathering to writing and grading the outputs.

A lot of benchmarks evaluate whether the answer is correct, rather than evaluating the entire body of work.

But in finance, the body of work is actually the most important thing people evaluate.

We focus on high-quality outputs because that's what drives ROI for customers. How realistic our evals are compared with others in the market is a real differentiator for us.

The Daily Context: Is there anything else you want to highlight?

Yijia: We talked earlier about multiple people working with AI at the same time. I think that's going to be a big part of the future, especially for deal teams.

With our new product version, we've built that in natively. It's almost like a shared Slack channel where multiple people can talk with AI together.

Our belief is that you have to unify where people collaborate with the outputs they're working on.

That might mean multiple people talking in a channel about a financial model or leaving comments on a PowerPoint for the agent to incorporate.

It's a much more natural way of working. From an engineering perspective, it's also difficult to build a system that works well when taking feedback from five different people at once.

I think multiplayer and collaborative AI will be a major next frontier.

Your Docs Are Doing Your Marketing Now (Whether You Like It Or Not)

Kara Silverman — Wed, 15 Jul 2026 16:15:36 +0000

TL;DR -

Nobody has one AI visibility number. You have six (one per model), and they disagree — a cloud infra company we audited scored 33% on ChatGPT/Gemini but 16% on Perplexity.
For dev tools, documentation is your top citation source — not the blog, not press. Our best performer pulled 17% of its category citations from its own domain vs. a 4.5% median.
One honest comparison page that includes competitors beat years of blog content: it drove a 33% owned-citation rate, the highest we measured.
A low Perplexity score means your live site isn't producing citable pages; a low Copilot score is a blind spot right where enterprise buyers research vendors.
Rebrands break AI visibility. One company's retired brand was 10x more visible in AI answers than its live one — redirects fix HTTP, not language models.
Your citations are leaking to Medium (37%), Reddit (32%), and YouTube (18%) — pages that could point at your domain instead.

Over the past six months, my agency Althea Labs has run brand visibility audits on 20+ companies measuring how often AI models (ChatGPT, Gemini, Perplexity, Copilot, Google AI Overviews/AI Mode) actually mention and cite them when developers ask questions in their category.

About half were developer tools companies: Observability, auth, cloud infra, databases, AI hardware. Which means we now have a decent dataset on how LLMs decide which dev tools to recommend.

Some of what we found confirmed what you'd expect, but if you work on a devtools product or you're the developer who somehow inherited "AI visibility" as a side quest there are three key learnings worth knowing.

1. Your documentation is your marketing department now

The best-performing devtools company in our dataset gets mentioned in 41% of AI answers in its category, with remarkably consistent performance across every model. When we broke down WHERE its citations come from, it wasn't the blog. It wasn't press coverage. It was:

Core documentation
SDK hubs
Protocol-level explainers (the "what is SAML vs OIDC" tier of content)

That company earns 17% of all its category citations from its own domain. The median across our whole portfolio is 4.5%.

Once you see it, the mechanism is pretty clear: When a developer asks an LLM "how do I implement X," the model reaches for content that is structured, specific, and extractable. Documentation is accidentally perfect for this. It has clear headings, canonical answers, code blocks, and no marketing language or fluff. A docs page that says exactly what a thing does, in what versions, with what limitations, is one of the most citable objects on the internet.

Another company in the Althea Labs dataset proved this with a sample size of one page. Their owned citation rate is 33%, the highest we've measured, and it's driven almost entirely by a single honest comparison page ("best X databases compared," including competitors) on their own domain. One well-structured, genuinely useful page outperforms years of blog content.

So, what can you actually do with this info? Treat documentation as a discovery surface, not just a support surface. Un-gate your white papers. Add comparison pages that include your competitors and tell the truth. Use spec tables and decision criteria instead of adjectives. If your docs answer the question better than a Medium listicle, the models will eventually notice because right now, across the categories we measured, Medium (37% citation rate), Reddit (32%), YouTube (18%), and yes, our very own DEV.to (16.5%) are eating citations that could point at your domain.

2. Your visibility varies wildly by model, and the gap tells you what's broken

Nobody has one AI visibility number. Everyone has six, and they sometimes disagree.

I want to show you a quick breakdown of a cloud infrastructure company Althea audited. It had 33% visibility on ChatGPT and Gemini, 16% on Perplexity, 18% on Copilot. For an edtech platform, we saw 56% on Gemini and just 9.6% on Perplexity. Across the 12 companies where we captured model-level splits, Perplexity came in below the other models for 10 of them.

The models sit on a spectrum from "training-data-driven" to "search-driven." ChatGPT and Gemini lean on what they learned in training on years of internet history, brand priors, and accumulated reputation. Perplexity barely does; it searches the live web and cites what it finds today.

So the Perplexity gap is diagnostic:

High everywhere, low on Perplexity → You're coasting on brand equity. The models "remember" you, but your current site isn't producing citable pages. That equity erodes every time a model retrains on a web where you're not the source.
Low everywhere, alive on Perplexity → Your content works but you have no entity footprint. The most extreme case in our data: one company registered 0% on ChatGPT, Gemini, Copilot, and both Google AI surfaces — and only existed on Perplexity, because it had a handful of genuinely citable pages and nothing else. That's a single point of failure. One change to Perplexity's retrieval behavior, and their AI visibility goes to literal zero.
Low on Copilot specifically → Pay attention if you sell to enterprises. One database company in our set showed 0% on Copilot and 0.5% on ChatGPT while doing fine on Google surfaces. Copilot is embedded in the exact environment where enterprise buyers do vendor research.

So, what can you actually do with this info? Measure per-model, not aggregate. Then fix the side you're weak on: Search-driven models want citable pages (structure, schema, llms.txt, clean sitemaps so crawlers actually reach your content); training-driven models want entity signals (Wikipedia/Wikidata, Crunchbase, consistent third-party coverage that repeats who you are and what category you're in).

3. If you rebrand without an entity migration, LLMs will keep recommending your old brand

One company in our dataset rebranded in the middle of last year. New name, new domain, redirects done properly, all the traditional SEO boxes checked.

Their old brand name ranks #3 in their category in AI answers and is mentioned 36% of the time. Their new brand name ranks #28 and is mentioned 3.7% of the time.

The retired brand is still 10x more visible than the live one.

Training cutoffs are the reasons for this issue. A model trained before the rebrand has never heard of the new company. It has years of docs, Stack Overflow threads, conference talks, and blog posts about the old name and messaging. Redirects fix HTTP; nothing about a 301 teaches a language model that two entities are the same thing.

And it decays slowly from there. Every quarter until the major models retrain on a web that consistently connects the old name to new one, the recommendations, the "what should I use for X" answers, and the tutorial mentions keep flowing to a brand that no longer exists.

So, what can you actually do if a rename/rebrand is on your roadmap? Treat the entity migration as a launch-critical workstream, not a comms afterthought. Update Wikipedia, Wikidata, Crunchbase, PitchBook, G2, and GitHub org metadata on day one. Get third-party coverage that explicitly states "X, formerly known as Y" so the models learn the equivalence from repetition in authoritative sources. Add Organization schema on your domain declaring the former name. Then monitor model-by-model until the new entity actually takes.

The wide angle view

The most encouraging stat in our whole dataset: The #1 most-visible brand in one category we measured appears in only 17% of AI answers about its own space. The "winner" is absent from 4 out of 5 answers. In dev tools, where documentation culture already produces the exact kind of content LLMs want to cite, that's not a threat. It's the most open distribution channel since early Google and the teams that treat their docs, comparison pages, and community content as citation infrastructure are going to quietly own it.

The Vertical Turn

Kara Silverman — Wed, 01 Jul 2026 14:41:07 +0000

Look at tomorrow's track list and count the rooms that have absolutely nothing to do with engineering. Go ahead, I'll wait.

Thursday’s schedule is running 12 parallel tracks. The AI in Healthcare, AI in Finance, and AI in GTM are brand new this year. On top of that, there are entirely dedicated audiences for founders, researchers, and data buyers. Trust me, that is not a scheduling decision. That is a deliberate segmentation decision that reflects where this industry is headed.

For three years, this conference spoke to one very specific audience: the AI engineer. The person writing the agent, wiring the retrieval pipeline, running the eval suite. That audience built a discipline from scratch and gave it a name. But this year, the organizers looked at who was actually hitting the registration page. People from companies like Vanguard, CVS Health, Intuitive Surgical, Two Sigma, Capital One, and more. It’s a marked expansion and a recognition of AI engineers existing not just at tech companies but in other areas of work as well.

Tomorrow's Healthcare track is running sessions on diagnostics, drug discovery, clinical workflows, and regulatory constraints. Finance covers trading, risk assessment, fraud detection, and compliance. GTM sits right next to both of them. These aren’t fluff panels about "the future of AI.” They are actual, practical working sessions for people who need AI to clear a compliance review, reduce false positives in a fraud model, or shorten a sales cycle. These are people who do not care what framework you used.

This is what real adoption looks like when it stops being theoretical. It’s not a bigger model or flashier demo. It’s a room full of clinicians, traders, and go-to-market operators who showed up because the tool finally got specific enough to matter to them.

AI Isn't Ready to Build Complex Software

Kara Silverman — Wed, 01 Jul 2026 13:19:40 +0000

In an industry currently swept up in the hype of AI coding agents, Micah Wylde, principal engineer at Cloudflare, offers a sobering reality check. Wylde is leading the development of a new suite of data platform products and has a front-row seat to both the power and the limitations of generative AI. Despite having his own "aha moment" last year when models became capable of generating significant amounts of code without heavy oversight, his hottest industry take right now is remarkably clear: AI models are still not good enough for complex software development.

For Wylde, who builds intricate database systems, the stakes are simply too high for AI to fly solo. Software at this level is nonnegotiable in its requirements, and it needs to produce completely correct results, ensure data is never lost, and remain highly maintainable over time. While an AI tool might successfully generate a massive block of code that satisfies your immediate prompt, it falls short on the bigger picture. As Wylde notes, AI "has no consciousness of what the lifetime of this project is." It solves the micro-problem in the moment while completely ignoring the macro-architecture, creating a massive gap for systems that demand rigorous, long-term stability.

So, how should developers navigate this gap? By doubling down on the fundamentals.

Wylde warns that while AI agents undeniably allow engineers to move much faster, they do not replace foundational computer science skills. To successfully orchestrate these agents and build complex systems today, engineers still must deeply understand how code actually works and how computers operate. Until AI models evolve to grasp the long-term maintainability of a codebase, the human engineer's true value isn't in generating raw code; rather, it's in architecting software that is actually meant to last.

The Evolution & Role of Context Engineering in AI Today

Kara Silverman — Tue, 30 Jun 2026 15:33:14 +0000

I was taking a break from the AIE Workshops on Monday and stepped out by the food stands to check out the crepes. That's when I saw a line literally wrapping the entire length of the Moscone West windows looking out onto Fourth Street. I couldn't imagine what a several-hundred-person line was for, and when I went to ask, they told me it was for the Context Engineering Workshop. That sent me down a rabbit hole exploring and understanding and learning. So now, for you, I will share what I got.

For the past couple of years, the AI world was obsessed with prompt engineering, aka the art of speaking to a machine. But as developers move from simple chatbots to complex autonomous agents, a new discipline has taken center stage: context engineering.

Mike Swift (@theycallmeswift), CEO of Major League Hacking (@mlhacks) , gave me some critical background. He pointed me to Dex Horthy of HumanLayer (who is actually speaking later this week), who basically coined the term at the first AI Engineer World's Fair. Dex's core thesis, said Swift, is that "agents get bad after about 100,000 tokens," which represents roughly 10% of their total available context window.

So context engineering is essentially managing an AI's working memory. Context engineering, Swift noted, is "managing how many times the loop goes around to how much you have to remember every time you do it." It is a counterintuitive concept for humans; the more we talk about a subject, the deeper our shared understanding becomes. But models work the opposite way. They lose focus as their context window fills up.

For many developers, meticulously curating this working memory is a practical necessity. I sat down with Ben Halpern (@ben), founder in residence at MLH and co-founder of DEV, who told me that context engineering is the "latest frontier of the optimization point" where developers can leverage their expertise. Beyond just keeping models coherent, Ben pointed out that developers who are doing product work manage context for "latency and cost effectiveness."

During our chat, I tested a somewhat spicy thesis I'd been developing: that context engineering feels like a proxy for control, perhaps even our "last real attempt at managing control" before models outpace us. Ben agreed that there is a massive opportunity for human impact here, while not fully buying into my thinking. He noted that because "model progress is going to be slower than the tooling progress," context engineering allows developers to "reduce the feeling of models as a black box." Instead of waiting for the next foundational model to magically fix memory limitations, developers can actively add value to workflows today.

Next, I talked with Dominik Kundel, who works on developer experience for Codex. Dominik argued that while strict context management is valuable, for example, in narrow, customer-facing support bots, it actively hinders general-purpose agents.

When developers meticulously craft context windows, Dominik said, "What you're doing is shrinking and limiting the possible range of tasks that it can solve." Instead of filtering what the model sees, Dominik advocates for progressive discovery. "I don't filter anything," he said, noting that Codex connects directly to vast, unstructured sources like Slack, Gmail, and Notion so that it can learn to navigate the noise on its own.

This unstructured approach enables what he calls "vague prompting." Because the agent has access to everything, you can talk to it like a colleague and ask it to "fix the thing that Karen asked me about yesterday." The agent will independently figure out which Karen is relevant, locate the Slack thread, and pull the necessary context. When I brought up my thesis about context engineering acting as a proxy for human control, Dominik acknowledged that strict management definitely makes outputs more predictable. However, he warned that this "determinism results in reduction capabilities" and a severe loss of flexibility.

Ultimately, walking away from Moscone West last night, it was clear that context engineering is far from a one-size-fits-all discipline. Whether a developer chooses to heavily filter an agent's working memory to maintain a human thumbprint or embrace progressive discovery to unlock vague prompting, the era of treating an AI's context window as an endless dumping ground is definitively over.

Reconciling the Distributed System: How the AI Engineer World's Fair Engineered Human Connection

Kara Silverman — Tue, 30 Jun 2026 15:08:52 +0000

On Sunday night, the sold-out 2026 AI Engineer World's Fair kicked off its orientation at Moscone West. The sheer scale of the event was staggering, featuring an estimated 7,000 attendees — a massive leap from the 1,500 who gathered at the Marriott down the street just last year. Amid this sprawling crowd, the core irony of the event was palpable: every technical talk could easily have been watched from home on YouTube, but instead the room was filled with people looking to be part of the experience IRL. One thing many first-time conference attendees — engineer, marketer, or creative — struggle with is talking to new people. Luckily, that was this gathering's true purpose: creating real human relationships.

Conference organizer @swyx opened the event with a relatable, if surprising, confession to the crowd: as a programmer and a writer, he is not a natural networker. Recognizing this common trait among developers, he introduced "NEO," or New Engineer Orientation, an initiative designed to bridge this social gap. NEO was created to give attendees a structured way to spark genuine connection and build relationships that would stay with them long after the technical sessions ended.

As the industry transitions from cloud computing to AI, the most valuable skills might not be strictly technical. The uniquely human abilities — to smile, laugh, and shake hands; to find community and process shared emotion — are still part of the equation. The AI Engineer World's Fair orientation proved that even in a sold-out crowd of brilliant engineers, people occasionally need a little analog help to break the ice.

Appreciate this very much, especially as a relatively new software creator myself.

Kara Silverman — Mon, 08 Jun 2026 14:19:38 +0000

MLH Team for Major League Hacking (MLH)

Jun 5

GitHub, Demystified

#github #beginners #opensource #software

9 min read