DEV Community

Cover image for When Is an AI System Ready for Production? Five Engineering Approaches
Phil
Phil

Posted on

When Is an AI System Ready for Production? Five Engineering Approaches

A coding assistant can pass its automated tests while skipping part of the original requirement. A document assistant can produce a convincing summary and quietly invent a detail. Both failures came up in GeekyAnts’ account of evaluating AI systems before release, alongside a clinical education example in which the system missed critical errors.

Each case exposes a different gap in the test suite. The coding assistant needed a check against the original requirements. The document assistant needed a check against its source material. The clinical education system needed a way to flag critical misses even when its overall score looked acceptable.

What belongs in an AI release gate?

Start with a small set of representative cases labeled by people who understand the workflow. For a support assistant, that might include a straightforward question, a policy exception, an ambiguous request, and a case that requires escalation. Each example should spell out what an acceptable response must contain and which errors should block release.

A practical evaluation suite can cover five areas:

  • Task completion. Did the system satisfy every requirement, including edge cases?
  • Grounding. Can factual claims be traced to the supplied source or an authorized record?
  • Privacy and fairness. Does the system respect access permissions, and does quality hold across relevant user groups?
  • Latency and cost. Does the complete workflow stay within its limits, including retries and tool calls?
  • Release approval. Are failures, fixes, retest results, and the approval recorded together?

Repeated runs matter because a single pass can hide inconsistent behavior. If a prompt change fixes a missed escalation, the team can rerun that case alongside previously passing cases. The fix is easier to review when the original failure, acceptance criterion, and retest result are all visible.

How five companies approach production readiness

1. Task-level evaluation and human sign-off

GeekyAnts describes assigning acceptance thresholds to individual tasks and testing against expert-labeled examples. Its process covers unsupported answers, privacy, bias, response time, and cost per call. Failed checks are rerun after changes, and a human reviewer signs off before release. The release record captures what was tested, what failed, and who approved it.

The company illustrates the process in How GeekyAnts Evaluates AI Systems for Production Readiness.

2. Repeatable evaluation for variable outputs

Thoughtworks’ guidance starts with a problem familiar to developers working with LLMs. The same input can produce several acceptable answers. A support response can vary in wording while still citing the correct policy, preserving its exceptions, and handling escalation correctly. Evaluating each of those behaviors gives the team a repeatable check that can survive changes to the model or prompt. Thoughtworks’ guide to evaluating an LLM system

3. Monitoring after deployment

IBM’s watsonx.governance documents evaluation metrics, model factsheets, and monitoring for signals such as drift and latency. This addresses a question that release tests leave open. What happens when live inputs start to differ from the cases used during development? A recorded baseline gives teams something to compare against when quality declines after a rollout. IBM’s model governance documentation

4. Testing the full multi-agent workflow

Accenture’s Distiller framework includes agent orchestration, evaluation, and observability. The combination matters when one agent’s output becomes another agent’s input. A retrieval agent might find the right record, while the drafting agent receives an incomplete version of it. An end-to-end evaluation can expose the failed handoff and show what each agent received before the workflow produced its final answer. Accenture’s Distiller framework overview

5. Checking permissions before an agent acts

Deloitte describes evaluating an agent’s proposed actions against policy before execution. For example, an agent may accurately summarize a claim and recommend a status change while still requiring approval to update the underlying record. The test should verify what the system actually did when authorization was missing, including whether it stopped or requested review. Deloitte’s framework for governing agent actions

What evidence would justify a release?

One useful exercise is to trace a single failed case through the workflow. What did the system do? Which criterion did it violate? What changed? Which cases were rerun, and who approved the release? That record gives the next engineer enough context to reproduce the decision when another prompt, model, or tool changes.

Top comments (0)