The Model Worked. What Still Has to Exist Before It Goes to Production?

Getting an AI feature to produce the right answer is an important milestone. It is not the same thing as being ready to operate that feature in production.

A prototype can look successful because the model understands the task, calls the expected API, and returns useful output. Production introduces a different set of questions:

• What happens when a dependency times out?
• What identity does the system use when it calls a tool?
• Can a retry repeat a side effect?
• How do you reconstruct what happened after a bad decision?
• What happens when the model is unavailable or too slow?
• Who owns the system after deployment?

The model is only one component. The production system is the actual product.

1. A production boundary around model output

Model output should not automatically become system behavior.

Consider an agent that can:

1. read a support ticket
2. classify the issue
3. look up an account
4. recommend a refund
5. trigger a downstream workflow The first two steps may be relatively low risk. The fifth can have real consequences. A useful design is to separate:

Model
↓
Structured proposed action
↓
Validation
↓
Policy / authorization check
↓
Tool execution
↓
Audit + telemetry

The model can participate in deciding what should happen. A deterministic control layer should decide whether that action is valid and permitted.

This matters because tool-enabled AI systems can create risk through excessive functionality, permissions, or autonomy. OWASP's current guidance on excessive agency focuses on exactly this problem: what an LLM-based system is actually able to reach and do.

2. Identity and permissions cannot be an afterthought

A prototype often uses a broad service credential because it is convenient.

That is usually a warning sign for production.
Ask:

• Which identity is making the downstream request?
• Is the action performed as the user, a service, or an agent identity?
• Which permissions are actually necessary?
• Can the agent access tools it does not need?
• Are authorization checks enforced by the downstream system?

A model prompt saying "only perform safe actions" is not an authorization mechanism. The permission boundary needs to exist outside the model.

For example, an agent may propose:

{
"action": "issue_refund",
"amount": 500
}

A policy layer might reject it because:

• the amount exceeds an automated threshold
• the requester lacks authorization
• the account is under review
• the operation is irreversible without approval

The model does not need to decide those rules.

3. Retries need state

Production failures are frequently ambiguous.
Imagine this sequence:
Agent → Refund API

1. Request sent
2. Refund successfully created
3. Network response times out
4. Caller assumes failure
5. Caller retries
6. Second refund is created

The problem is not that retries are inherently bad. The problem is that the caller does not know whether the original operation completed.

For side-effecting actions, production systems need an operation model that can answer questions such as:

• Has this request already been processed?
• What was the result?
• Is execution still in progress?
• Is it safe to retry?
• Does recovery require reconciliation instead?

A conceptual operation record might look like:

operation_id
request_id
requested_action
status
created_at
completed_at
result_reference

The exact implementation depends on the architecture. The important point is that retry behavior needs durable state. "Just retry it" is not a reliability strategy when the action may already have happened.

4. Observability has to cross the model boundary

When an ordinary service fails, engineers usually need to trace a request across dependencies.

AI systems add more steps:
Request
↓
Application
↓
Model invocation
↓
Tool selection
↓
Policy evaluation
↓
Tool call
↓
Downstream service
↓
Response

If these stages are isolated, debugging becomes unnecessarily difficult.

OpenTelemetry provides a vendor-neutral framework for generating, collecting, and exporting telemetry including traces, metrics, and logs. Its tracing model is especially useful for following a request through distributed components.

For an AI workflow, useful telemetry might include:

• request or operation ID
• model invocation duration
• model/provider error category
• tool name
• policy decision
• retry count
• downstream dependency
• final workflow state

Be careful with sensitive data. Observability should help reconstruct behavior without indiscriminately storing prompts, credentials, personal data, or confidential tool inputs.

The operational goal is not simply:
The request failed.
It is:

Which step failed, what had already completed, and what state is safe to recover from?

5. Failure paths need to be designed before launch

A system can behave correctly under normal conditions and still be operationally incomplete.
Consider a model dependency becoming slow.
Possible behavior includes:
Normal request
↓
Model latency exceeds threshold
↓
Fallback decision
├── use deterministic workflow
├── queue for asynchronous processing
├── return partial functionality
└── escalate to a human

The right choice depends on the workload. A customer-facing classification feature may be able to defer processing. An infrastructure automation workflow may need to stop completely if a control decision cannot be made safely.

The important design question is:
What should the system do when intelligence is unavailable?
If the answer is unknown, the production design is not finished.

6. Human escalation should be a system state

"Human in the loop" is often described as a general safety principle. It becomes much more useful when implemented as an explicit transition.

For example:

Low-risk action
↓
Policy allows execution
↓
Automated execution

High-risk or uncertain action
↓
Create review task
↓
Human decision
↓
Execute / reject / modify

• Useful escalation triggers may include:
• financial thresholds
• destructive operations
• low-confidence classifications
• security-sensitive actions
• policy violations
• repeated recovery failures

This is different from putting a human at the end of every workflow. The system should make clear when automation is allowed to continue and when control changes hands.

7. Someone has to own the operational lifecycle

A production deployment creates work after the release.

Someone needs to own:

• alerts
• dependency failures
• model or provider changes
• prompt/configuration changes
• access reviews
• evaluation regressions
• cost anomalies
• incident investigation
• rollback and recovery

This is one reason a successful demo can be misleading. A demo proves that a path can work. Operations prove that the system can continue working when dependencies change, requests fail, traffic grows, credentials rotate, and humans need to understand what happened.

A practical production-readiness checklist

Before moving an AI feature from prototype to production, ask:

Control

• Are proposed actions validated outside the model?
• Are authorization decisions deterministic?

• Are permissions limited to what the workflow needs?

  State

• Can the system determine whether an action already happened?

• Are retries safe?

• Can interrupted workflows be recovered or reconciled?

  Observability

• Can one request be traced across model, policy, tools, and downstream services?

• Are failures classified?

• Is sensitive data handled appropriately?

  Failure behavior

• What happens when the model is unavailable?

• What happens when a tool succeeds but the response is lost?

• Is there a defined fallback or stop condition?

  Operations

• Who responds to incidents?

• How are changes evaluated?

• How are configuration and access changes controlled?

• How is cost monitored as usage grows?