I went to the AI Engineer World's Fair expecting a lot of demos and left with a shorter list of things I actually want to try. These are the notes I kept going back to.
Evals are still the bottleneck
The most useful talks weren't about models, they were about evaluation. One team described keeping a spreadsheet of forty hand-written cases with expected outputs, run on every change, and catching regressions that dashboards missed for weeks. It's the least glamorous takeaway and the one I've already started copying.
The pattern I heard repeated: small, specific, in-repo eval sets beat big generic benchmarks for the thing you actually shipped.
Agents are mostly about state and retries
Every agent talk eventually turned into a talk about state machines. Who owns the state, what happens when a tool call times out, how a retry avoids double-applying an action. The teams that looked calm about production had explicitly modeled that; the ones that looked nervous were still relying on the model to remember.
I left with a concrete note: idempotency keys for every side-effecting tool call, and a place to store the result of each step. That's boring plumbing, and it's most of the difference between a demo and a product.
The talks that changed my mind
One session argued that for internal tools, a small fine-tuned model on in-domain data often beats a larger general model, mostly because latency and cost matter more than the last few percent of quality. We had been defaulting to the biggest model available, and I'm now going to test the small one first for the next internal feature.
What I'm still skeptical about
Multi-agent orchestration for its own sake. Every demo with five agents talking to each other could usually be one agent with a loop and a clearer prompt. The interesting problems are still state, evaluation, and failure handling, and adding more agents mostly multiplies those problems.
What's next for us
A small eval set in the repo, idempotency on the tool layer, and a latency budget for every user-facing call. Nothing from the conference was about a new model, which I think is a sign of where the actual work is.
Top comments (2)
The eval-set advice is the one I've actually implemented since reading this. Forty cases on every change catches more than I expected.
Agree on multi-agent orchestration. We had a five-agent demo that became one agent with a loop and better logging.