DEV Community

Cover image for Every AI Agency Has a Demo. Ask What Changed After Launch.
Israel Vásquez for Monogram

Posted on with Isaac Harper

Every AI Agency Has a Demo. Ask What Changed After Launch.

The launch version of the multi-agent solution generator we built for Google Maps Platform used about 30,000 tokens per request. Production logging then showed that most requests touched only 2 to 3 APIs from a catalog of dozens. Once we injected only the matching examples, usage dropped to about 8,000 tokens per request with no measured loss in output quality.

A demo can show you the first number. Only real traffic produces the second. If you're an engineering lead choosing an AI partner, that gap is what you're trying to measure.

A disclosure: we're an applied AI studio, so this is a vendor writing about how to vet vendors. The examples come from our own case studies, and we flag what they don't document.

Ask what changed after launch

A team with production experience can tell you what broke once real users arrived and what they changed. A demo team mostly talks about the model and the prompt.

Hamel Husain makes the same point about evals. He groups AI product work into evaluating quality, debugging issues, and changing the system's behavior, and he notes that focusing only on that last activity keeps products from getting "beyond a demo."

The Google Maps Platform pipeline went through 5 documented phases after launch. Dynamic instruction injection produced the token reduction above. Evaluation data then showed that evaluating and refining code didn't need the reasoning depth of writing it. Moving every agent except the code generator from Gemini 2.5 Pro to Gemini 2.5 Flash cut evaluation cycles from about 45 to 60 seconds to about 12 to 18 seconds. Later, as traffic grew, 429 quota errors became the main way requests failed. We responded with a three-layer retry strategy and a fallback to the previous valid solution, so a failed request became a slower one.

Signal: they can name the failure mode that dominated after launch and point to the logs that exposed it.

Ask how they know the output is right

"It looks good" is not an evaluation. Ask what the model's output is compared against and what the agreement rate is.

On Delta's customer journey intelligence platform, Claude Sonnet scores conversations against Delta's own quality rubrics. We compared its scores with conversations Delta's analysts had already scored, then refined the rubrics wherever the two diverged. Exact agreement on that calibration set is about 85%. It rises above 90% when scores within 1 point on the 1-to-10 scales count as matches.

The number comes with a named comparison set and a definition of a match. The rubric also lives in a prompt Delta controls, so changing criteria means editing text, not retraining a model.

In the Google Maps Platform pipeline, a separate evaluator reviews generated code, so the generator never grades its own work.

Signal: they give you an agreement number, the human baseline behind it, and the person on your side who owns the criteria.

Ask where they chose code over a model

Anthropic's guidance on building effective agents recommends finding the simplest solution possible, because agentic systems often trade latency and cost for better task performance.

In the Delta platform, PII redaction is deterministic code. It runs on every assembled conversation before anything reaches the model and costs zero tokens. The analysis also runs as a scheduled batch rather than in real time, because the use cases (reporting, incident investigation, trend analysis) don't need per-second latency. Batch keeps LLM costs predictable. A run covers about 2,000 conversations in a few hours.

In the Google Maps Platform pipeline, 2 of the 9 agents make no LLM calls: a schema validator and a documentation builder that maps detected APIs to URLs from a canonical registry. That removes two places where the system could hallucinate.

Signal: they can point to the steps they kept deterministic and explain why some parts aren't real time.

Ask what the model is not allowed to do

The OWASP Top 10 for LLM applications lists sensitive information disclosure and excessive agency among its risks. Production experience shows up in where a team draws those boundaries.

In the voice booking agent we built inside Delta's Amazon Connect IVR, the model creates a payment link and never touches payment data. Identity runs through Delta's own lookup system. If the agent can't resolve a request within 2 turns, it transfers the caller to a human agent along with the full conversation context and the identified account.

The same case study is also candid about limits. The response latency target was under 2 seconds, and the agent achieves 2 to 3. Transcription and text-to-speech consume 1 to 2 seconds before the model sees any input. True streaming wasn't ready, so the model streams replies into DynamoDB and Lex checks for complete sentences every few hundred milliseconds. The page reports a 92% end-to-end success rate against live APIs but no production call volume. Ask any vendor which stage their example has reached.

Signal: their numbers include the misses, and they say whether a system is a pilot or serving users at volume. The Google Maps Platform case study, for comparison, reports steady-state production at about 800 to 1,000 daily users.

Ask what your team owns afterward

The Google Maps Platform pipeline shipped with Terraform-managed dev, staging, and prod environments, CI/CD gates, BigQuery telemetry, and structured observability. On Delta's platform, the rubrics belong to Delta.

Two web projects show the same habit, though neither was an AI system build. Kasada's marketing team composes pages from a Sanity section library while developers keep control of rendering and performance. Fireworks AI's engineers build marketing pages from a web design foundation we delivered, without a new design request for each page.

Our post on shipping the AI layer with the code describes the exit test we write into acceptance: the client's team onboards a source or ships a feature with us in review only.

Signal: handoff has a written acceptance test, not just a promise of documentation.

Questions to bring to the first call

  • What did you change after launch, and which logs or evaluation data told you to?
  • What do you compare model output against, and what is the agreement rate?
  • Which steps did you keep deterministic, and why?
  • What can the model never see or do, and when does it hand off to a person?
  • Which of your numbers missed their targets?
  • What will my team be able to change without you?

A demo proves a model can do the task. Production experience is the work around the model: how its output gets measured, what it is kept away from, and what your engineers can still change after the vendor leaves.

Top comments (0)