Part 5 — Cost, Latency, and Failure Are the Design

In GenAI systems, cost and latency are not optimizations.

They are design constraints.

Ignoring them early leads to brittle systems later.

Cost is proportional to thought

Every token has a price.

That means:

• Longer prompts cost more
• Larger context costs more
• Retries cost more
• Ambiguous design costs more

Unlike traditional systems, inefficiency shows up directly on the bill.

Latency compounds quickly

GenAI latency is additive:

• Retrieval latency
• Model latency
• Post-processing latency
• Retry latency

Each step feels reasonable in isolation. Together, they define user experience.

Systems that feel slow rarely have a single bottleneck. They have accumulated assumptions.

Failure is normal, not exceptional

GenAI systems fail differently.

They:

• Return partial answers
• Degrade silently
• Produce confident nonsense
• Time out under load

This means failure must be anticipated, not handled reactively.

Designing for degradation

Resilient systems:

• Fall back to smaller models
• Reduce context under pressure
• Return partial results
• Fail explicitly when needed

This is not pessimism. It’s engineering.

The next post looks at observability and evaluation, and why GenAI systems need different signals than traditional services.