Day 1: The AI Infrastructure Shift — Why the Model is Only 10% of the Production System

In enterprise AI, a common belief is that business value comes mainly from the choice of foundation model. Teams spend weeks benchmarking a 7-billion-parameter model against a 70-billion one, debating fine-tuning curves and polishing system prompts. Then, once a model is chosen, they hit a wall trying to move from a prototype on a laptop to a real production environment.

In real-world deployments the model is roughly 10% of the total engineering effort. The other 90% is the platform infrastructure needed to serve, scale, observe and secure that intelligence. This series is about that 90%.

In plain terms: A model is the engine. Nobody ships an engine to customers. They ship a car: fuel system, brakes, dashboard, locks, service schedule. This series builds the car.

Diagram: FIG 1 · The model is one block in the grid. The other eight blocks are the platform that serves, scales, observes and secures it. See the animated version.

Walk through the grid and you see the real job. Data has to flow in (ingestion and pipelines). Traffic has to be routed safely (ingress and a service mesh). State has to survive restarts (persistent storage and vector databases). Every change must ship safely (CI/CD and GitOps), every resource must be scheduled and capped, every request must be traceable (observability), and untrusted model-generated code must be contained (sandboxes and egress control). The model is one block of nine.

Diagram: FIG 2 · Teams spend weeks choosing a model, then discover that most of the work is the platform around it. See the animated version.

The hidden technical debt in machine learning

In 2015, Google researchers published a now-classic paper, Hidden Technical Debt in Machine Learning Systems. It showed that the machine-learning code in a real system is a small box surrounded by a much larger maze of infrastructure plumbing. With today's large language models (LLMs), that maze has grown:

• Infrastructure plumbing: API rate limits, load balancing for high-concurrency prompts, context-window exhaustion, and database connections.
• Resource allocation: inference is compute-bound and memory-bound. Unlike a stateless Node.js or Go service that sips memory, an LLM needs continuous GPU memory (VRAM), careful KV-cache management and hardware acceleration passed through to the container.
• State and synchronization: real-time systems must keep vector databases, caches such as Redis, and model weights in step with low latency.

The memory problem, in numbers

The KV-cache is the model's working memory for the conversation. Every token a request holds in its context adds to it, for every layer of the network. That is why a model that fits comfortably for one user can run out of memory at ten.

Diagram: FIG 4 · Weights are a fixed cost. The KV-cache grows with every long request, so a model that fits one user can run out of memory at ten. Approximate, before runtime overhead. See the animated version.

Here is the arithmetic as a few lines of Python. The architecture numbers match a typical 8-billion-parameter model (32 layers, 8 key-value heads, a head size of 128). It is a rough budget that ignores runtime overhead, and I have run it:

# Rough GPU memory budget for serving one 8-billion-parameter model.
GIB = 2**30

def weights_gib(params: float, bytes_per_param: float) -> float:
    return params * bytes_per_param / GIB

def kv_cache_gib(tokens: int, layers=32, kv_heads=8, head_dim=128, bytes_per_value=2) -> float:
    # Keys and values (x2), for every layer, for every token held in context.
    return 2 * layers * kv_heads * head_dim * bytes_per_value * tokens / GIB

gpu_gib = 24
for name, bytes_per_param in [("FP16", 2), ("INT8", 1), ("INT4", 0.5)]:
    w = weights_gib(8e9, bytes_per_param)
    per_request = kv_cache_gib(8192)
    fits = int((gpu_gib - w) // per_request)
    print(f"{name}: weights {w:5.1f} GiB, one 8k-token request {per_request:.1f} GiB, "
          f"about {fits} concurrent requests fit in {gpu_gib} GiB")

Notice what changes the answer: quantization shrinks the weights, which leaves room for more concurrent requests. We cover that in Day 4.

The fragility of unmanaged inference

The fastest way to get a model online is to load it in a Python script and wrap it in Flask or a basic FastAPI app. It works in a demo. In production it fails in three predictable ways:

• Memory leaks and out-of-memory panics: concurrent long prompts exhaust memory, and the Linux kernel's OOM-killer terminates the process with no warning.
• No fault isolation: if the model runtime crashes, the whole application gateway goes with it, dropping live connections and losing in-flight work.
• Security gaps: unsandboxed execution lets a prompt-injection attack reach the host filesystem, internal network sockets or environment variables, which often hold secrets.

Diagram: FIG 3 · A raw script takes the whole gateway down when it runs out of memory. Isolated pods contain the failure, and the platform replaces the failed one. See the animated version.

The platform engineering mandate

Reliable AI systems come from platform engineering habits. Stop treating AI as an isolated data-science experiment, and treat the inference runtime as a mission-critical infrastructure tier:

• Containerization: package the runtime as a deterministic, immutable, multi-stage Docker image (Days 2 and 3).
• Resource throttling: enforce memory and CPU quotas with Linux kernel cgroups, so one runaway request cannot take down a node (Day 3).
• Network isolation: use a zero-trust service mesh and eBPF-based networking to restrict what a model or agent can reach (Week 3).

Diagram: FIG 5 · Containerize for repeatability, throttle with kernel quotas, and isolate the network, so one misbehaving model cannot hurt anything else. See the animated version.

A quick readiness check

Before you call a model "production-ready", ask:

1. What happens when the process runs out of memory, and who restarts it?
2. How many concurrent long requests fit before it fails, and has that been measured?
3. Can the image be rebuilt identically next month, on any machine?
4. What can the model, or an agent running its code, reach on the network?
5. If an answer is wrong at 3 a.m., can you trace which step caused it?
6. How does a new version reach production, and how do you roll it back?

If you cannot answer most of these, the model is not the problem. The platform is.

What the next 30 days build

Each week wraps another ring around the model, until you have something you could actually run in production, on a laptop with 16 GB of RAM.

Diagram: FIG 6 · Each week adds a ring around the model: containers, Kubernetes, network and security, then agents and CI/CD. See the animated version.

Reliable enterprise AI begins at the operating system and networking layer, not in the prompt window.

Coming Up Next

Day 2: Advanced Docker for AI: multi-stage builds for heavy Python runtimes.

AIInfrastructure #MLOps #PlatformEngineering #CloudNative #DevOps #EnterpriseAI #SystemDesign #TechLeadership

---

Originally published at https://sureshpallapothu.in/blog/ai-int-day-1-ai-infrastructure-shift, where this post includes animated diagrams.