Beyond the LLM Call: Anatomy of a Production AI Application

The first version of an AI feature is often one endpoint:

receive text → call an LLM → return output

It works until it does not.

A client retries after a timeout. A traffic burst exceeds model throughput. The model returns JSON that breaks a downstream schema. A worker crashes between inference and persistence. Someone asks why a request was slow, why it failed, or why it was invoked twice.

At that point, the problem is no longer “how do I call an LLM?”

It is:

How do I operate an AI workload reliably when the model is slow, probabilistic, rate-limited, and external to my system?

A production AI application is not an LLM call. It is a distributed system that happens to use an LLM.

This article outlines the architecture behind RunicGate, a production-oriented AI API on AWS. The goal is not to prescribe one universal AWS stack. It is to explain the engineering boundaries that make AI inference recoverable, observable, and safe to operate.

---

The deceptively simple POST /generate

The prototype architecture is familiar:

Client → API → LLM → Response

For an internal demo or low-risk experiment, that can be enough.

For a real application, it creates several questions with no good answer:

• What happens if the client disconnects after the API invokes the model?
• What happens when the client retries the request?
• What happens during model throttling?
• What happens when generation takes longer than the HTTP timeout?
• What happens when the model returns valid JSON that is invalid for your domain?
• What happens when you need to inspect one failed request across your API, workers, and model provider?
• What stops a tenant from producing an expensive burst of requests?

The direct request-response path makes one HTTP connection responsible for too much:

1. accepting work
2. validating work
3. invoking a slow external dependency
4. surviving retries
5. persisting outcomes
6. reporting errors
7. returning a response before timeout

That is not a reliable workflow boundary.

---

Turn model inference into a durable job

The central architectural shift is simple:

The API should acknowledge durable acceptance of AI work, not promise that inference will finish before an HTTP timeout.

Instead of making the API wait for a model response, the system creates a job.

The API is responsible for:

1. authenticating the caller
2. validating request shape and tenant permissions
3. establishing idempotency
4. persisting a durable job record
5. putting work on a queue
6. returning a stable job identifier

A separate worker is responsible for:

1. claiming queued work
2. invoking the model
3. validating the model output
4. persisting a result or terminal failure
5. emitting logs, traces, and metrics

That separation gives the system a recovery point between “the client asked for work” and “the model completed work.”

RunicGate: From API request to durable AI result

flowchart TD
    Client[Client application] --> Edge[CloudFront + AWS WAF]
    Edge --> Gateway[Amazon API Gateway]
    Gateway --> API[RunicGate API on ECS]

    API --> JobStore[(DynamoDB\nJobs + Idempotency)]
    API --> Queue[Amazon SQS\nGeneration Queue]

    Queue --> Worker[RunicGate Worker on ECS]
    Worker --> Model[Amazon Bedrock]
    Worker --> ResultStore[(Amazon S3\nEncrypted Results)]
    Worker --> JobStore

    JobStore --> Status[Polling API / Webhook Dispatcher]
    Status --> Client

    Queue --> DLQ[SQS Dead-Letter Queue]

    API -. traces, logs, metrics .-> Obs[CloudWatch + OpenTelemetry]
    Worker -. traces, logs, metrics .-> Obs

The diagram deliberately focuses on the workload lifecycle:

• Edge and access control: CloudFront, WAF, API Gateway
• Request/control plane: API service and job persistence
• Asynchronous processing plane: SQS and worker service
• Model provider boundary: Amazon Bedrock
• Result plane: DynamoDB job state and S3 result storage
• Observability plane: logs, metrics, traces, and alarms

The individual AWS services matter, but the durable boundaries between them matter more.

---

Why asynchronous by default?

A synchronous endpoint can be appropriate for a narrow set of AI interactions:

• short, predictable requests
• interactive UX where immediate feedback is essential
• streaming experiences with clear cancellation behavior
• bounded workloads with explicit latency targets

But it should be a deliberate exception, not the default architecture.

Model latency is variable. Throughput is limited. A request can fail after partial work has already happened. A client can retry independently from your server.

An asynchronous job API handles those realities more naturally:

POST /v1/generations
→ 202 Accepted
→ { "job_id": "gen_123", "status": "QUEUED" }

GET /v1/generations/gen_123
→ { "status": "SUCCEEDED", "result_url": "..." }

Queues also change the failure mode under load.

Without a queue, overload can become timeouts, retries, connection exhaustion, and cascading errors.

With a queue, overload becomes visible backlog. That is still a problem, but it is an operable problem. You can measure queue age, control worker concurrency, apply backoff, and alert before clients experience a complete outage.

The trade-off is client complexity. Polling is less convenient than a direct response. Webhooks add delivery concerns. But those are explicit, manageable concerns, unlike ambiguous work that may or may not have completed during an HTTP timeout.

---

Idempotency is both a correctness and cost control

Clients retry for normal reasons:

• network interruption
• DNS failure
• application restart
• load balancer timeout
• the server completed work but the response was lost

If the same logical request can trigger multiple model invocations, you have both a correctness issue and a cost issue.

A generation request should include an idempotency key:

POST /v1/generations
Idempotency-Key: 4ec6a9d0-6d58-4d08-a4f1-4b0f42e6d5aa

The server uses that key together with tenant identity:

tenant_id + idempotency_key → one logical job

A naive implementation often does this:

existing = find_job_by_idempotency_key(key)

if existing:
    return existing

return create_job()

This is race-prone.

Two identical requests can both check for a job before either creates one. Both conclude that no job exists. Both create a job. Both invoke the model.

The correct pattern is an atomic conditional write.

Note: The following is a reference implementation pattern for RunicGate, not a claim of measured production behavior.

from botocore.exceptions import ClientError

def create_idempotent_job(table, job: dict) -> dict:
    try:
        table.put_item(
            Item=job,
            ConditionExpression=(
                "attribute_not_exists(tenant_id) "
                "AND attribute_not_exists(idempotency_key)"
            ),
        )
        return job

    except ClientError as error:
        if error.response["Error"]["Code"] != "ConditionalCheckFailedException":
            raise

        existing = table.get_item(
            Key={
                "tenant_id": job["tenant_id"],
                "idempotency_key": job["idempotency_key"],
            }
        ).get("Item")

        if not existing:
            raise RuntimeError("Idempotency conflict without a retrievable job")

        return existing

The exact DynamoDB key design depends on your access patterns. The architectural point is that duplicate prevention belongs in the persistence layer, not in an in-memory if statement.

Idempotency keys also need retention rules.

If you keep them forever, storage and lookup requirements grow indefinitely. If you expire them too soon, a delayed retry might create a duplicate job. Choose a retention window based on realistic retry behavior and document it.

For example:

• short-lived interactive jobs might retain keys for 24 hours
• batch ingestion workflows may need longer windows
• high-risk side-effecting tasks may need explicit client request identifiers retained for longer

There is no universal duration. There should be an explicit decision.

---

Queue semantics: assume a message can be delivered twice

Amazon SQS Standard queues provide at-least-once delivery.

That means the same message can be delivered more than once. This is not a bug. It is a property the application must design around.

Consider this sequence:

1. Worker receives job J-123.
2. Worker invokes the model.
3. Model returns a result.
4. Worker crashes before persisting SUCCEEDED.
5. SQS visibility timeout expires.
6. Another worker receives J-123.

If the second worker blindly invokes the model, the job may be processed twice.

A worker needs a controlled state transition:

QUEUED → PROCESSING → SUCCEEDED
                    → RETRY_SCHEDULED
                    → FAILED

The worker should claim the job with a conditional update, not merely trust that receiving an SQS message means it exclusively owns the work.

def claim_job(table, job_id: str, worker_id: str) -> bool:
    try:
        table.update_item(
            Key={"job_id": job_id},
            UpdateExpression=(
                "SET #status = :processing, "
                "worker_id = :worker_id, "
                "processing_started_at = :now"
            ),
            ConditionExpression="#status = :queued",
            ExpressionAttributeNames={"#status": "status"},
            ExpressionAttributeValues={
                ":queued": "QUEUED",
                ":processing": "PROCESSING",
                ":worker_id": worker_id,
                ":now": "2026-09-09T00:00:00Z",
            },
        )
        return True

    except ClientError as error:
        if error.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

For longer-running work, add an ownership lease:

worker_id
lease_expires_at
attempt_count

A lease gives the system a way to recover if a worker dies while holding a job.

“Exactly once” is not something you get automatically from a queue, database, or cloud service. In distributed systems, it usually means carefully combining idempotency, conditional state transitions, and well-defined side-effect boundaries.

---

Model output needs a contract

LLM output is probabilistic. Even when a model returns valid JSON, it can still violate your application contract.

For example, a downstream system may require:

{
  "category": "billing",
  "priority": "high",
  "summary": "Customer reports duplicate charge"
}

The model might return:

{
  "category": "payments",
  "priority": "urgent",
  "summary": null
}

That is valid JSON. It is not valid domain data.

The worker, not the API edge, should validate the output after inference:

from pydantic import BaseModel, Field
from typing import Literal

class TicketClassification(BaseModel):
    category: Literal["billing", "technical", "account"]
    priority: Literal["low", "medium", "high"]
    summary: str = Field(min_length=1, max_length=500)


def validate_model_output(payload: dict) -> TicketClassification:
    return TicketClassification.model_validate(payload)

If validation fails, the system must decide whether the error is retryable.

A useful initial classification:

Failure class	Retry?	Example
Provider throttling	Yes	Model throughput temporarily exceeded
Temporary provider/network failure	Yes	Service unavailable or connection reset
Invalid structured output	Limited retry	Model broke the output contract
Invalid client request	No	Unsupported task type or malformed input
Authorization failure	No	Tenant lacks access to model or task
Safety-policy rejection	Usually no	Input or output violates a configured policy

Keep retries bounded. A system that retries invalid output forever is not resilient. It is expensive and noisy.

Terminal failures should preserve operational context:

• job ID
• tenant ID
• task type
• attempt count
• failure category
• provider error code where safe
• timestamp
• correlation ID

Do not automatically persist or log raw prompts, full model responses, secrets, or sensitive customer data just because debugging is difficult.

---

Why this AWS stack fits the workload

The RunicGate architecture uses AWS services because they align with specific workload needs, not because a portfolio project needs a service checklist.

SQS Standard versus FIFO

Use SQS Standard when throughput and decoupled processing matter more than strict ordering.

It is a good fit for independent generation jobs, but it requires idempotent consumers because messages can be delivered more than once.

Use SQS FIFO when ordering or deduplication guarantees are central to the domain.

The trade-off is lower throughput flexibility and more careful message-group design. For independent AI jobs, FIFO is often unnecessary.

DynamoDB versus Aurora PostgreSQL

DynamoDB fits RunicGate’s primary access patterns:

• create job
• retrieve job by ID
• retrieve an existing idempotent request
• conditionally update job state
• retrieve jobs by tenant and time range, if indexed appropriately

Aurora PostgreSQL becomes attractive when you need relational reporting, ad hoc queries, complex joins, or transactional relationships across many entities.

The trade-off is not “NoSQL is faster.” The trade-off is whether your data access patterns are stable and key-driven or relational and exploratory.

ECS Fargate versus Lambda

ECS Fargate is a good fit for workers that need:

• controlled concurrency
• longer-lived processes
• predictable connection behavior
• fine-grained worker configuration
• consistent containerized local and cloud environments

AWS Lambda is attractive for lower-volume, event-driven workloads with bursty traffic and minimal operational overhead.

The trade-off is that Lambda execution limits, cold starts, concurrency controls, and long-running workflow behavior may be less convenient for some AI worker designs.

Amazon Bedrock versus direct provider APIs

Amazon Bedrock keeps model access inside AWS IAM and can simplify model integration, security controls, and account-level governance.

Direct provider APIs may expose models or features not available in your selected Bedrock region or account configuration.

The trade-off is portability versus operational integration. Keep model calls behind an adapter so the rest of the system does not care which provider is used.

CloudWatch-only versus OpenTelemetry

CloudWatch logs and metrics are the baseline. They should answer:

• Is the queue backing up?
• Are jobs failing?
• Are workers healthy?
• Is Bedrock throttling?
• Is the DLQ growing?

OpenTelemetry tracing becomes valuable when diagnosing a single request across API ingress, job persistence, queue publication, worker processing, model invocation, and result delivery.

The trade-off is instrumentation effort and trace-volume cost. Start with correlation IDs and structured logs. Add distributed tracing where it changes your ability to debug.

---

Observability should follow the job

A job should have an identity that travels with it.

At minimum, attach:

job_id
tenant_id
correlation_id
trace_id
task_type
attempt_count
model_id
status

The API should create or accept a correlation ID. The job record should persist it. The queue message should carry it. The worker should include it in logs and trace spans.

That gives you an investigation path:

Client request
  → API validation log
  → DynamoDB job record
  → SQS publish event
  → worker receive event
  → Bedrock invocation span
  → output validation result
  → terminal job state

The metrics that matter are operational, not vanity metrics:

• end-to-end job duration
• queue depth
• age of oldest queue message
• worker concurrency
• retries by failure class
• DLQ depth
• model throttling errors
• output validation failures
• job outcomes by tenant and task type
• token usage, where your model provider exposes it

Do not publish performance claims before you have measured them.

Instead, run controlled tests and report exactly what happened in your environment.

---

What to measure during implementation

A production architecture needs evidence, not assumptions.

1. Idempotency under concurrent retries

Send the same request multiple times concurrently with the same tenant and idempotency key.

Measure:

• number of job IDs created
• number of queue messages accepted
• number of model invocations
• final job status consistency

The correctness criterion is one logical job for one logical client action.

2. Burst absorption

Submit more jobs than the configured worker concurrency can process at once.

Measure:

• queue depth
• oldest-message age
• worker concurrency
• completion latency percentiles
• provider throttling events

The question is not whether latency rises. It will. The question is whether the system degrades predictably and remains explainable.

3. Worker crash recovery

In a development environment, deliberately stop a worker after model invocation but before terminal job persistence.

Measure:

• whether duplicate processing becomes possible
• whether a lease or conditional transition detects ownership conflicts
• whether the final job state remains understandable
• whether an operator can identify what happened from logs and traces

4. Structured output validation

Run a curated, synthetic dataset through your structured task.

Measure:

• schema-valid outputs
• categories of validation failures
• retries caused by invalid output
• terminal failures after retry exhaustion

Do not present this as a general model benchmark. It is a test of your application contract and recovery behavior.

5. One trace, end to end

Take one job and follow its correlation ID through:

API log
→ queue publish
→ worker receive
→ Bedrock invocation
→ output validation
→ persisted result

If you cannot explain one job from submission to completion, you cannot confidently operate thousands.

---

The model is one dependency, not the architecture

The main lesson is not that every AI application needs every AWS service in this diagram.

The lesson is that AI inference needs the same engineering discipline as any other distributed workload:

• durable state
• explicit ownership
• bounded retries
• idempotency
• overload handling
• validation boundaries
• security controls
• traces, logs, and metrics
• a clear story for failure

Model quality is only one reliability dimension.

An AI application becomes production-ready when the system can answer:

• Did this job run?
• Did it run more than once?
• Who requested it?
• What state is it in?
• Why did it fail?
• Can it be retried safely?
• What happens when the provider slows down?
• What data did we retain?
• Can an operator understand the outcome?

The production boundary of an AI application is not the model endpoint.

It is the architecture that decides what happens before, during, and after the model call fails, retries, slows down, or returns something unexpected.

Comments

[Theo Valmis]

Durable acceptance is a much cleaner contract than keeping the HTTP request alive through inference. The subtle failure case is the ambiguous write between persisting the job and publishing to the queue. An outbox or equivalent atomic handoff belongs in the core design, otherwise idempotent clients can still create stranded work.

[Pasindu Lanka]

Great catch. The dual write boundary between the DynamoDB conditional put and the SQS publish is indeed the Achilles heel here. If the API crashes or SQS throttles immediately after persisting the record, the idempotency gate locks the job in a QUEUED state in DynamoDB without the message ever hitting the queue, leaving it stranded against client retries.

In production, tying DynamoDB Streams (acting as an outbox via CDC) to an EventBridge Pipe/SQS or running a reconciliation sweeper for stale QUEUED items solves this atomic handoff cleanly. Appreciate you highlighting this subtle edge case!