DEV Community

Cover image for AI Engineering Has Entered Its Audit Era
Ben Halpern Subscriber Community Curator for Daily Context

Posted on

AI Engineering Has Entered Its Audit Era

A conference agenda is a pretty decent dataset.

Between what is pitched and what is accepted, it's always an attempt to triangulate on the zeitgeist of the moment. So I dug through the roughly 140 sessions at AI Engineer New York and tried to get a sense of what themes exist at this event that might not have been so collectively front of mind six months ago.

The first finding is the obvious one: this is an agent conference now. Two in five sessions have "agent" in the name. That's table stakes; the interesting part is what people want to do about them.

It's also a finance conference by design, billed as "Where AI Engineering Meets Wall Street." That makes it more useful, not less. Finance is the stress test: an industry where "mostly right" is a liability, every decision needs a paper trail, and someone always asks why. Whatever worries this room now will worry the rest of us soon.

"It works" is no longer the bar. "Prove it" is.

After agents, the biggest cluster is evaluation and verification: nearly 20 sessions, six with the word "trust" in the name. Generation is cheap now. Proving it's right is where the work is. Addy Osmani and Manish Kapur are asking who verifies the software factory, and other talks cover verification loops and agent fixes that quietly break something else.

Finance pushes past correctness into defensibility. One session asks whether you can defend an agent's decision after it passed every eval. That might be the sharpest question at the conference. Vinoo Ganesh is on three sessions, one of which boils down to a single instruction: don't let models invent numbers. Passing evals and earning trust are different things, and the gap between them is where many AI engineering careers will be made.

Agents are becoming principals

About a dozen sessions deal with security, identity, permissions, or payments. We spent twenty years building that stack for humans. Now we're rebuilding it for software that acts on its own. The panel on intent-based auth includes Dick Hardt, who led the design of OAuth 2.0. Others cover agent permissions in regulated industries, financial accounts for agents, and the idea that your next user is an agent.

My favorite pairing: one talk asks why agents don't pay for things even though they can, and another walks through six attacks on x402 agent payments. That's the whole adoption curve in two talks. The rails exist; the trust doesn't yet. If you run an API, ask now what changes when the caller is an agent with a budget.

Intelligence has a price tag

Cost is the other loud signal. There are sessions on tokenomics, the cost of intelligence, and the hidden costs of a faster model. Then Wall Street's own vocabulary shows up: compute markets, the dark pools of compute, and a session from the CEO of Compute Exchange. One of the speaker lounges is literally the Token Billionaires Lounge.

When people describe GPUs in the language of market structure, compute has become a commodity. AI engineers now have to think like CFOs: what does this call cost, and is the better model worth it?

So what do we do with this?

Ryan Carson argues that your job is now agent manager. Laurie Voss is talking about the rise of the product engineer. I think they're right, with a caveat from another session: stop turning your humans into APIs for your agents.

A conference like this, with fairly pragmatic outcomes center of mind, is able to cut through frontier model headlines and help us understand the themes we can apply more agency to. We're trying to do our job as AI Engineers by proving the output, scoping the authority, and paying the bill.

Top comments (1)

Collapse
 
botdevsupports profile image
DEV SUPPORT •
You need to verify your account.
Enter fullscreen mode Exit fullscreen mode

tr.ee/dev-to