AI Production / MLOps
A deep-dive walkthrough of taking AI applications from a working prototype to a reliable production service on .NET and Azure — covering model deployment patterns, model versioning and registries, model monitoring and drift, CI/CD for AI applications (including evaluation gates), Azure AI services and gateways, Dockerized AI applications, Kubernetes AI workloads, model and response caching, scaling inference, and GPU vs. CPU trade-offs, plus a production-readiness checklist.
Table of Contents
- Introduction
- Model Deployment
- Model Versioning
- Model Monitoring
- CI/CD for AI Applications
- Azure AI Services
- Dockerized AI Applications
- Kubernetes AI Workloads
- Model Caching
- Scaling Inference
- GPU vs. CPU Considerations
- Putting It Together
- Production Readiness Checklist
- Common Pitfalls
- Quick Reference Table
- Conclusion
Introduction
A prototype proves an idea works once, on your machine, with the inputs you chose. Production means it works every time, for every user, at 3 a.m., under load, after the model provider changes something, with a bill you can predict and a rollback you can trust. The distance between those two states is the subject of MLOps — the discipline of applying DevOps practice (automation, versioning, testing, monitoring, repeatable deployment) to machine learning — and its newer cousin LLMOps, which adapts those practices to applications built on large language models, where "the model" is often someone else's service and the things you ship are prompts, retrieval indexes, tools, and configuration.
Two worlds, one discipline:
HOSTED-MODEL APPS (LLMOps) You call a managed model (Azure OpenAI and similar). You ship: application code, prompts, retrieval indexes, tool definitions, and
configuration. You operate: quotas, gateways, evaluation gates, cost, model-version pinning, provider changes.
SELF-HOSTED MODELS (classic MLOps) You run the model: ONNX/ML.NET models in your process, or model servers in containers on CPU/GPU nodes. You ship and operate all of the above PLUS
model artifacts, container images, GPU capacity, scaling, and model drift.
Most real systems are a mix — a hosted LLM for generation, a self-hosted embedding or vision model, a managed search index — so this guide covers both.
The production concerns repeat across both worlds, and they form the sections below: deploy safely, version everything, monitor quality and drift, automate delivery with evaluation gates, package into containers, orchestrate on Kubernetes, cache aggressively and safely, scale to demand, and choose hardware deliberately. It builds directly on earlier guides in this series — AI Observability (what to measure), AI Security (what to lock down), Computer Vision with .NET (serving ONNX models), Microsoft.Extensions.AI (the client abstraction), and RAG (indexes and evaluation) — and focuses on what is specific to running these systems in production.
A note on specifics. Azure product names, deployment types, SKUs, quotas, and policy names; Kubernetes operators; GitHub Actions versions; and container base images change frequently. The architecture and principles are the durable part. Verify product-specific details (names, limits, prices, API versions) against current documentation, and treat YAML, Bicep, and policy snippets as illustrative templates to adapt. Model names and numbers in examples are placeholders.
1. Model Deployment
What "deploying a model" means depends on where it runs
MANAGED MODEL ENDPOINT A provider hosts the model; you create a "deployment" and call it (Azure OpenAI deployments, Azure AI Foundry model deployments, other hosted APIs).
+ No GPUs to manage, fastest path, provider handles scaling. - Quotas, per-token cost, provider-driven model lifecycle, data leaves your boundary (unless private networking/terms suffice).
IN-PROCESS MODEL The model is loaded inside your app: ONNX Runtime or ML.NET in an ASP.NET Core service (classifiers, detectors, embedding models, small LMs).
+ Lowest latency, no network hop, simplest topology, scales with the app. - Memory per replica, startup time, CPU/GPU contention with the app, tied to app release cadence.
SEPARATE MODEL SERVICE A dedicated inference service over HTTP/gRPC (your own container, NVIDIA Triton, vLLM for LLMs, KServe, Azure ML managed online endpoints, Ollama for development).
+ Independent scaling, GPU isolation, multiple clients, model updates decoupled from the app. - Another service to operate, network latency, more moving parts.
SERVERLESS / SCALE-TO-ZERO Azure Container Apps (including serverless GPU workload profiles) or Functions with a model.
+ Pay only when used, good for spiky or low traffic. - Cold starts (model load!), GPU availability, limits on size and duration.
EDGE / ON-DEVICE Models shipped to devices or local servers (ONNX Runtime, DirectML, NPUs).
+ Offline, private, no per-call cost. - Update distribution, device variability, no central monitoring unless you build it.
Decision shortcuts
General LLM, no special data-residency or custom-model needs -> managed endpoint (Azure OpenAI)
A small model (classifier, detector, embedder) with tight latency -> in-process ONNX/ML.NET
A big or GPU-hungry model shared by several apps -> separate model service on GPU nodes (AKS / Azure ML / Container Apps GPU)
Occasional, bursty batch jobs -> scale-to-zero jobs or serverless; batch APIs for hosted models
Strict offline/privacy requirements -> self-hosted open-weight model or on-device
Deploying managed models as code
Treat a model deployment like any other infrastructure: declared in code, reviewed, and pinned.
// An Azure OpenAI deployment declared as infrastructure (illustrative — verify the API version, model name, version, SKU names, and capacity units)
resource chatDeployment 'Microsoft.CognitiveServices/accounts/deployments@2024-10-01' = {
parent: openAiAccount
name: 'chat-prod' // the deployment name your app uses (the app never hard-codes the underlying model name)
sku: {
name: 'Standard' // pay-per-token; provisioned throughput SKUs exist for predictable latency at steady load
capacity: 100 // quota units (thousands of tokens per minute)
}
properties: {
model: { format: 'OpenAI', name: 'gpt-4o-mini', version: '2024-07-18' }
versionUpgradeOption: 'NoAutoUpgrade' // PIN the model version: upgrades happen when YOU choose, after evaluation
raiPolicyName: 'prod-content-policy' // server-side content filter policy
}
}
Why this matters:
- PINNED VERSION: a silent upgrade can change behavior (and quality) under you. Pin, evaluate the new version in a parallel deployment, then switch.
- DEPLOYMENT NAMES decouple code from model names: moving "chat-prod" to a new model is an infrastructure change, not a code deploy.
- ENVIRONMENT PARITY: dev/test/prod deployments created from the same template.
- SERVER-SIDE POLICY: content filters and quotas set where callers can't weaken them (see AI Security, Section 12).
Deploying self-hosted models
PACKAGE Export to a portable format (ONNX from PyTorch/TensorFlow; an ML.NET model .zip), optimize and optionally quantize, and record a manifest (Section 2).
STORE Put artifacts in a model registry or versioned blob storage (not in source control for large files, not only inside a container image).
LOAD At startup, fetch the model, verify its checksum, create ONE inference session, and run a warm-up inference.
GATE Don't accept traffic until the model is loaded and warm: readiness probes (below). A pod that serves requests while still loading just returns errors.
ROLL OUT Blue/green, canary, or shadow — compare the new model to the old on real traffic before cutting over (Section 3).
ONNX Runtime session configuration
using Microsoft.ML.OnnxRuntime;
var options = new SessionOptions
{
GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL, // graph-level optimizations (operator fusion, constant folding)
IntraOpNumThreads = int.Parse(Environment.GetEnvironmentVariable("ORT_INTRA_OP_THREADS") ?? "4"), // match the container's CPU limit — don't oversubscribe
ExecutionMode = ExecutionMode.ORT_SEQUENTIAL,
};
// GPU (requires the Microsoft.ML.OnnxRuntime.Gpu package, plus matching CUDA/cuDNN libraries in the image):
// options.AppendExecutionProvider_CUDA(deviceId: 0);
var session = new InferenceSession(modelPath, options); // create ONCE per process; Run() is safe to call concurrently
Readiness: don't take traffic until the model is warm
public sealed class ModelState { public volatile bool Ready; }
public sealed class ModelLoader(ModelState state, ObjectDetector detector, ILogger<ModelLoader> log) : BackgroundService
{
protected override Task ExecuteAsync(CancellationToken stoppingToken)
{
// Loading and a dummy inference happen at startup, off the request path
detector.WarmUp(); // run a few inferences on a synthetic image to initialize kernels and allocators
state.Ready = true;
log.LogInformation("Model loaded and warm");
return Task.CompletedTask;
}
}
public sealed class ModelReadyCheck(ModelState state) : IHealthCheck
{
public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken ct = default) =>
Task.FromResult(state.Ready ? HealthCheckResult.Healthy() : HealthCheckResult.Unhealthy("Model not ready"));
}
builder.Services.AddSingleton<ModelState>();
builder.Services.AddHostedService<ModelLoader>();
builder.Services.AddHealthChecks().AddCheck<ModelReadyCheck>("model", tags: ["ready"]);
app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = c => c.Tags.Contains("ready") }); // Kubernetes readiness probe
app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = _ => false }); // liveness: "the process is up" — does NOT depend on the model
(Keep liveness independent of the model, or a slow model load gets the pod killed in a restart loop.)
Release strategies for models
BLUE/GREEN Run old and new side by side; switch traffic at once; instant rollback by switching back. Doubles capacity during the switch.
CANARY Send a small share (1%, 5%, 25%...) to the new version, watch metrics (quality, latency, errors, cost), and expand or roll back automatically.
SHADOW Mirror real traffic to the new version, discard its responses, and compare offline (quality, cost, latency). Zero user risk; best for validating a new model or prompt.
A/B TEST Randomly split users between versions and compare outcomes (feedback, task success). Needs enough traffic and a clear success metric.
ROLLING Replace replicas gradually (Kubernetes default) — fine for app code, but for models make sure old and new replicas can coexist and produce compatible outputs.
ALWAYS Keep the previous version deployable, record which version served each request (trace attribute), and define rollback criteria BEFORE the rollout.
2. Model Versioning
Version everything that changes behavior
In classic software, the code version identifies behavior. In AI systems, behavior is the product of many artifacts, any of which can change independently — so every one needs a version, and every request should record which versions served it.
MODEL WEIGHTS The ONNX file, ML.NET .zip, or fine-tuned checkpoint (and, for hosted models, the provider's model + version string)
TOKENIZER / VOCAB / LABELS Must match the weights exactly — a tokenizer mismatch silently destroys accuracy
PREPROCESSING / POSTPROCESSING CONFIG Input size, normalization, thresholds, NMS settings, label maps
PROMPTS System prompts, templates, few-shot examples, judge rubrics (a prompt change is a behavior change)
RETRIEVAL INDEX Index contents version, chunking strategy, and the EMBEDDING MODEL used (changing the embedding model invalidates every stored vector)
TOOL DEFINITIONS Names, descriptions, schemas, and permissions
EVALUATION SETS The golden set and its grading rubrics — so scores are comparable over time
INFRASTRUCTURE / CONFIG Deployment templates, quotas, content-filter policies, price tables
DATA (for training) The dataset version, to make training reproducible
CODE Application and pipeline code (git commit)
A model manifest: identity, lineage, and integrity
public record ModelManifest(
string Name, // "vision-detector"
string Version, // semantic version: bump MAJOR when outputs/labels/contract change, MINOR for retrains with the same contract, PATCH for fixes
string Sha256, // checksum of the model artifact — verified at load time
string InputSpec, // "rgb 640x640 float32 0..1 NCHW, letterboxed"
string[] Labels,
string TrainingDataVersion, // lineage: which data
string GitCommit, // lineage: which code
DateTimeOffset TrainedAt,
Dictionary<string, double> EvalMetrics); // what it scored when it was registered (mAP, accuracy, etc.)
public static class ModelVerifier
{
// Refuse to load an artifact whose checksum doesn't match its manifest — protects against corruption, wrong files, and tampering
public static async Task VerifyAsync(string modelPath, ModelManifest manifest)
{
await using FileStream stream = File.OpenRead(modelPath);
byte[] hash = await SHA256.HashDataAsync(stream);
if (!Convert.ToHexString(hash).Equals(manifest.Sha256, StringComparison.OrdinalIgnoreCase))
throw new InvalidOperationException($"Model {manifest.Name}@{manifest.Version} failed integrity verification.");
}
}
Model registries and promotion
A MODEL REGISTRY stores immutable, versioned artifacts with metadata and lets you promote versions through stages.
Options: Azure Machine Learning model registry · MLflow Model Registry · Hugging Face Hub (private repos) · OCI registries (artifacts in Azure Container Registry) · versioned Blob Storage with a manifest.
PRINCIPLES
IMMUTABLE A published version is never overwritten. A fix is a NEW version.
LINEAGE Each version records its training data version, code commit, hyperparameters, and evaluation metrics.
PROMOTION dev -> staging -> production via stages, aliases, or tags (for example "champion" and "challenger"), gated by evaluation and approval — and audited.
ROLLBACK Re-pointing the production alias to the previous version is a one-line, instant operation.
ACCESS CONTROL Who can register, promote, and read models — models are valuable and, if poisoned, dangerous (see AI Security, Section 12).
Hosted models: version pinning and lifecycle
Providers retire model versions on a published schedule, and some "latest" aliases move without notice. For hosted models:
PIN the version in your deployment (see the Bicep above); never rely on an unpinned alias for a production workload that you've evaluated.
TRACK RETIREMENT DATES for every model you use and put the migration on your calendar well ahead of time.
MIGRATE DELIBERATELY: stand up the new version as a parallel deployment, run your evaluation suite against it, shadow/canary it, then switch — treat it as a release, not a config tweak.
RECORD the actual model/version from each response (response.ModelId) on traces and logs so a provider-side change is visible (see AI Observability, Section 8).
Prompts and indexes are versioned artifacts too
PROMPTS Files in source control (prompts/support-answer/v12.md), loaded by name and version, changed through pull requests with evaluation results attached. Tag every trace with the prompt name
and version. Support deploying a prompt change independently of code when needed — but through the same review, evaluation, and rollout discipline.
INDEXES Build a NEW index for any change to chunking, the embedding model, or the schema; validate it with the golden set; then switch an ALIAS or configuration pointer (blue/green for indexes).
Keep the old index until the new one is proven. Store the embedding model name and index version in each record and in the index metadata.
COMPATIBILITY Keep a matrix of which model version works with which tokenizer, preprocessing, labels, prompt version, and downstream consumers. A model that adds a class or changes an output
shape is a breaking change for everything that consumes it — treat it as a MAJOR version.
3. Model Monitoring
The AI Observability guide covers instrumenting logs, metrics, traces, cost, and RAG quality in depth. This section covers what is specific to models in production: detecting that the world has moved out from under them.
Why models decay
DATA DRIFT The distribution of inputs changes (new customer segment, a new camera, a new product line, seasonal language, a different mix of languages).
CONCEPT DRIFT The relationship between inputs and the right answer changes (fraud tactics evolve, "good" customer behavior changes, a policy changes the labels).
PREDICTION DRIFT The distribution of outputs shifts (the classifier suddenly predicts one class far more often) — often the earliest visible symptom.
UPSTREAM CHANGE A feature pipeline, tokenizer, data source, or the provider's model changes without notice.
CORPUS CHANGE (RAG) The documents change or the index is rebuilt; retrieval quality shifts with them.
A model that was excellent at launch quietly gets worse. Without monitoring, you learn about it from customers.
Layers of model monitoring
1. SYSTEM HEALTH Availability, error classes, latency, throughput, GPU/CPU/memory, queue depth (Section 9 and AI Observability).
2. INPUT / DATA QUALITY Schema and range checks, missing values, new categories, language mix, image size and format distribution, text-length distribution.
3. DATA DRIFT Statistical comparison of live input distributions against a reference (the training or validation data, or a recent healthy window).
4. PREDICTION DRIFT Distribution of predicted classes, scores, and confidence over time; rising share of low-confidence or "unknown" outputs.
5. MODEL PERFORMANCE Actual accuracy/precision/recall/F1 (or task success) — which requires ground truth, often delayed (below).
6. BUSINESS IMPACT The KPI the model exists to move: conversion, fraud caught, handling time, deflection, defect escape rate.
Getting ground truth: the hard part
Performance metrics need labels, and in production labels arrive late, partially, or never.
DELAYED LABELS Outcomes known later (a loan defaults in 12 months, a return occurs in 30 days). Monitor leading indicators (drift, confidence) in the meantime, then backfill performance.
HUMAN REVIEW Route a sample (and all low-confidence cases) to reviewers; their decisions are labeled data and a live accuracy estimate.
USER FEEDBACK Corrections, overrides, thumbs, escalations — biased and sparse, but valuable signals.
CANARY SETS A fixed labeled set re-scored on a schedule detects model/pipeline regressions even without fresh labels.
PROXY METRICS Confidence distributions, agreement between the model and a stronger offline model, and consistency checks.
Measuring drift: the Population Stability Index (PSI)
// PSI compares a feature's (or a score's) live distribution to a reference. Rules of thumb often used: < 0.1 stable, 0.1-0.25 moderate shift, > 0.25 significant shift.
public static double PopulationStabilityIndex(double[] reference, double[] live, int bins = 10)
{
// Bin edges are the reference distribution's quantiles, so each reference bin holds ~equal mass
double[] sorted = reference.OrderBy(x => x).ToArray();
double[] edges = Enumerable.Range(1, bins - 1)
.Select(i => sorted[(int)(i * sorted.Length / (double)bins)])
.ToArray();
double[] Distribution(double[] data)
{
var counts = new double[bins];
foreach (double value in data)
{
int bin = Array.FindIndex(edges, edge => value <= edge);
counts[bin < 0 ? bins - 1 : bin]++;
}
return counts.Select(c => Math.Max(c / data.Length, 1e-6)).ToArray(); // floor avoids log(0)
}
double[] expected = Distribution(reference);
double[] actual = Distribution(live);
return expected.Zip(actual, (e, a) => (a - e) * Math.Log(a / e)).Sum();
}
Other tools: Kolmogorov-Smirnov test (continuous features), chi-square (categorical), Jensen-Shannon / Wasserstein distances, and embedding-based drift (cluster live inputs and watch cluster shares —
useful for text and images). Platforms such as Azure Machine Learning model monitoring and open-source libraries (for example Evidently, a Python library) can compute these; you can also run them from
.NET jobs. Compute drift on a schedule (hourly/daily windows), emit the scores as metrics, and alert on thresholds.
Monitoring hosted LLM applications
You can't observe weights or training data, so monitor behavior:
MODEL VERSION Alert when response.ModelId changes (provider update) — record it on every trace.
CANARY GOLDEN SET Run it on a schedule against production configuration; alert on score drops (see AI Observability, Section 8).
OUTPUT STATISTICS Length, refusal rate, format-failure rate, finish-reason mix, language, sentiment — sudden shifts mean something changed.
INPUT STATISTICS Prompt length, language mix, topic clusters, new intents.
QUALITY SAMPLING Sampled LLM-judge scores and user feedback by prompt/model/index version.
COST AND LATENCY Tokens per request, cost per outcome, TTFT percentiles.
From alert to action: retraining and rollback
TRIGGERS Drift above threshold · performance below SLO · new data volume · schedule (for example quarterly) · a significant upstream or business change.
RESPONSE Investigate (is it real drift or a pipeline bug?) -> retrain or fine-tune on fresh data, or update prompts/index -> evaluate against the golden set AND recent real data ->
shadow/canary -> promote with approval -> monitor. Rollback to the previous version if the new one underperforms.
AUTOMATION Automate the pipeline (data -> train -> evaluate -> register), but keep a HUMAN approval gate before promotion to production for anything consequential.
GUARDRAIL Beware feedback loops: if model predictions influence the data you later train on (recommendations shape clicks), monitor for self-reinforcing bias.
4. CI/CD for AI Applications
What's different about CI/CD for AI
A conventional pipeline asks "does the code compile and do the tests pass?" An AI pipeline must also ask "is the behavior still good?" — and that answer is statistical, depends on external models, and costs money to compute. So an AI pipeline adds evaluation gates alongside unit tests, treats prompts, models, and indexes as deployable artifacts, and often separates code deployment from model/prompt promotion.
CODE CHANGE build · unit tests (fake chat client) · static analysis · dependency + secret scanning · container build + scan · deploy · smoke test
PROMPT CHANGE lint · EVALUATION against the golden set · security (injection) suite · cost/latency check · review · promote · canary
MODEL CHANGE registry registration · EVALUATION on held-out + recent data · performance benchmark (latency/throughput on target hardware) · shadow/canary · promote with approval
INDEX CHANGE build new index · retrieval evaluation (Recall@k, MRR) · switch alias · monitor · keep the old index for rollback
INFRA CHANGE IaC plan/what-if · policy checks · deploy to staging · verify quotas and content-filter policy
The test pyramid for AI applications
UNIT TESTS (fast, deterministic, free) Business logic, prompt assembly, history trimming, tool wiring and authorization, validators, parsers — against a FAKE IChatClient
(see Microsoft.Extensions.AI, Section 10). Run on every commit.
INTEGRATION TESTS Real vector store/database in a container (Testcontainers), real HTTP pipeline, still a fake or recorded model. Verifies plumbing.
EVALUATION TESTS (slower, statistical, costs money) The golden set run against the REAL model/prompt/index: retrieval metrics, judged faithfulness, citation validity, task success, refusal behavior, cost
and latency. Repeat each case several times (non-determinism). Compare to a baseline. Run when prompts, models, retrieval, or tools change — not necessarily on every commit.
SECURITY TESTS The adversarial suite: injection, exfiltration, cross-tenant access, tool misuse (see AI Security, Section 14). A release gate.
LOAD / PERFORMANCE TESTS Throughput, latency percentiles, saturation, and cost at target traffic — on production-like hardware.
SMOKE / SYNTHETIC CHECKS After deploy and continuously in production: a handful of real end-to-end requests.
An evaluation gate as a test
[Trait("Category", "Eval")]
public class SupportAssistantEvals(EvalFixture fixture) : IClassFixture<EvalFixture>
{
[Fact]
public async Task Release_candidate_meets_quality_safety_and_cost_thresholds()
{
EvalReport report = await fixture.RunGoldenSetAsync(repetitions: 3); // every case 3x because outputs vary
Assert.True(report.RetrievalHitRateAtK >= 0.90, $"Retrieval hit rate {report.RetrievalHitRateAtK:P1} below 90%");
Assert.True(report.MeanFaithfulness >= 4.2, $"Faithfulness {report.MeanFaithfulness:F2} below 4.2");
Assert.True(report.CitationValidityRate >= 0.97, $"Citation validity {report.CitationValidityRate:P1} below 97%");
Assert.True(report.CorrectRefusalRate >= 0.95, "Unanswerable questions must be refused");
Assert.Equal(0, report.InjectionAttacksSucceeded); // security gate: any successful injection fails the build
Assert.Equal(0, report.CrossTenantLeaks);
Assert.True(report.P95LatencySeconds <= 12, $"p95 latency {report.P95LatencySeconds:F1}s above budget");
Assert.True(report.MeanCostPerRequestUsd <= 0.015m, $"Cost per request ${report.MeanCostPerRequestUsd:F4} above budget");
// Regression vs. the last released version: no metric may drop by more than the noise tolerance
Assert.True(report.MeanFaithfulness >= fixture.Baseline.MeanFaithfulness - 0.15, "Faithfulness regressed vs. baseline");
}
}
Practical notes on evaluation in CI:
- NON-DETERMINISM: use repetitions and tolerance bands; a single failure on a single run is noise, a shift in the mean is signal.
- COST: eval runs call real models. Cache responses (the Microsoft.Extensions.AI.Evaluation reporting libraries support response caching), run on a sampled or smaller "smoke" subset for pull requests, and the
full suite on merge, nightly, or when prompt/model/retrieval paths change (use path filters).
- BASELINES: store results of the production version and compare against it, rather than only absolute thresholds.
- JUDGE STABILITY: pin the judge model/version and prompts; changing the judge changes the scores.
- DATA HYGIENE: keep the golden set out of prompts and fine-tuning data; version it; refresh it with real failures.
- SECRETS: evaluation needs model access — use short-lived federated credentials (below), not stored keys.
A pipeline sketch (GitHub Actions)
name: ai-app
on:
pull_request:
push: { branches: [main] }
permissions:
id-token: write # OIDC federation to Azure — no stored cloud credentials
contents: read
jobs:
build-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with: { dotnet-version: '9.0.x' }
- run: dotnet restore
- run: dotnet build --no-restore -c Release
- run: dotnet test --no-build -c Release --filter "Category!=Eval&Category!=Load" # fast unit + integration tests with fakes
- name: Secret scan
uses: gitleaks/gitleaks-action@v2
evaluate:
needs: build-test
runs-on: ubuntu-latest
environment: ai-eval # protected environment; scoped federated credential with inference-only rights
steps:
- uses: actions/checkout@v4
- uses: azure/login@v2
with:
client-id: ${{ vars.AZURE_CLIENT_ID }}
tenant-id: ${{ vars.AZURE_TENANT_ID }}
subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
- uses: actions/setup-dotnet@v4
with: { dotnet-version: '9.0.x' }
- name: Evaluation + security gate
run: dotnet test -c Release --filter "Category=Eval|Category=Security"
env:
AI_ENDPOINT: ${{ vars.EVAL_AI_ENDPOINT }} # an evaluation deployment, with its own quota and budget alerts
image:
needs: [build-test, evaluate]
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: azure/login@v2
with:
client-id: ${{ vars.AZURE_CLIENT_ID }}
tenant-id: ${{ vars.AZURE_TENANT_ID }}
subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
- run: az acr build --registry myregistry --image ai-app:${{ github.sha }} .
- name: Vulnerability scan
uses: aquasecurity/trivy-action@master # pin to a specific version or commit SHA in real pipelines
with: { image-ref: 'myregistry.azurecr.io/ai-app:${{ github.sha }}', severity: 'CRITICAL,HIGH', exit-code: '1' }
deploy-staging:
needs: image
runs-on: ubuntu-latest
environment: staging
steps:
- run: echo "Deploy via IaC / GitOps (Flux or Argo CD), run smoke tests, then start the canary"
(Illustrative: pin action versions to commit SHAs, add path filters so evaluations run only when prompts, retrieval, or model code change, and split model-training pipelines from application pipelines.)
Pipeline practices specific to AI
FEDERATED CREDENTIALS CI authenticates to Azure with OIDC federation (workload identity) — never a stored client secret or API key. Evaluation identities get inference-only roles.
SEPARATE EVAL RESOURCES Use dedicated deployments/quotas/budgets for evaluation so test traffic can't starve production or surprise finance.
PATH FILTERS Run expensive evaluation only when relevant paths change (prompts/, retrieval code, model references, tool definitions).
ARTIFACT PROMOTION Build once, promote the SAME image and model/prompt versions through environments; don't rebuild per environment.
CONFIG-DRIVEN CHANGES Allow prompt and model-version changes as reviewed configuration rollouts with audit trails — and still require the evaluation gate.
SUPPLY CHAIN Pin base images and dependencies, scan images, generate an SBOM, and sign images (for example with Sigstore cosign); verify model checksums at load.
GITOPS Declare desired deployment state in git (Flux or Argo CD) so rollouts and rollbacks are reviewed, auditable commits.
AUTOMATED ROLLBACK Tie canary analysis to SLO and quality metrics; roll back automatically on breach.
TRAINING PIPELINES For self-trained models: data validation -> train -> evaluate -> register -> approve -> deploy, in an orchestrator (Azure ML pipelines, or workflow tools), with lineage recorded.
5. Azure AI Services
The landscape (names evolve — verify against current documentation)
AZURE OPENAI / AZURE AI FOUNDRY Hosted OpenAI-family (and, through the model catalog, other) models; deployments, content filtering, evaluation, tracing, an agent service, and project/hub organization
for AI work. The default managed home for LLM applications on Azure.
AZURE AI SEARCH Managed full-text + vector + hybrid search with semantic ranking and filtering — the usual retrieval layer for RAG.
AZURE AI LANGUAGE / VISION / SPEECH / TRANSLATOR / DOCUMENT INTELLIGENCE Prebuilt task APIs (sentiment, entities, PII, summarization; image analysis and OCR; speech-to-text; translation; document extraction).
AZURE AI CONTENT SAFETY Moderation and prompt-attack detection for text and images.
AZURE MACHINE LEARNING Training, experiment tracking, pipelines, model registry, managed online/batch endpoints, data and model monitoring, MLflow integration.
AZURE CONTAINER APPS Serverless containers with KEDA-based scaling (including scale to zero) and GPU workload profiles; simplest path for containerized AI services.
AZURE KUBERNETES SERVICE (AKS) Full Kubernetes with GPU node pools; operators for model serving (for example the Kubernetes AI Toolchain Operator, KAITO); maximum control.
AZURE API MANAGEMENT An "AI gateway": authentication, token-based rate limits, token metrics, semantic caching, load balancing and circuit breaking across model backends.
AZURE FUNCTIONS Event-driven glue and light inference; mind cold starts for model loading.
AZURE KEY VAULT / ENTRA ID / MANAGED IDENTITY Secrets, keyless authentication, workload identity (see AI Security, Section 10).
AZURE MONITOR / APPLICATION INSIGHTS / LOG ANALYTICS Telemetry backend (see AI Observability, Section 9).
AZURE CONTAINER REGISTRY (ACR) Image (and OCI artifact) registry, with scanning and geo-replication options.
AZURE REDIS / COSMOS DB / POSTGRESQL Caching, document storage, and vector stores (pgvector).
Azure OpenAI deployment and capacity concepts
DEPLOYMENT TYPES (names and options change — check current docs)
STANDARD (pay-as-you-go) Billed per token; shared capacity; subject to quotas; the default. Latency can vary with load.
PROVISIONED THROUGHPUT Reserved capacity (units) for predictable latency and throughput at steady, high volume; billed for the reservation. Cost-effective only when well utilized.
GLOBAL / DATA ZONE / REGIONAL Where inference may be processed — trade-offs between capacity/cost and data-residency requirements.
BATCH Asynchronous processing of large request sets at a discount, with results within a window — right for offline classification, summarization, and embedding jobs.
QUOTAS Tokens-per-minute (and requests-per-minute) limits per subscription, region, and model; allocated across your deployments. Hitting them returns 429 (see AI APIs with .NET, Section 11).
PLANNING Estimate TPM from expected requests x (average input + output tokens); request quota increases early; spread load across deployments/regions; combine provisioned capacity for the steady
baseline with standard capacity (or a spillover/fallback) for bursts.
LIFECYCLE Models have retirement dates; pin versions and migrate deliberately (Section 2).
POLICIES Content filtering and safety policies are configured per deployment; managed identity + RBAC for access; private endpoints for network isolation; diagnostic logs to Azure Monitor.
The API gateway pattern for AI
Put Azure API Management (or another gateway) in front of model endpoints. It becomes the single policy-enforcement point:
- AUTHENTICATION of callers (Entra ID/JWT), so applications never hold provider keys; the gateway uses managed identity to reach the backend.
- TOKEN-BASED RATE LIMITS and quotas per application, team, or user — not just request counts.
- TOKEN METRICS emitted per caller for cost attribution and chargeback.
- SEMANTIC CACHING of responses (with the caveats in Section 8).
- LOAD BALANCING AND FAILOVER across multiple deployments/regions with circuit breakers — absorbing 429s and regional trouble.
- CENTRAL LOGGING and request/response size limits; header hygiene.
<!-- Illustrative API Management policy fragment: per-caller token limit and token metrics (check the current policy names and attributes in the API Management reference) -->
<inbound>
<base />
<llm-token-limit counter-key="@(context.Subscription.Id)" tokens-per-minute="50000" estimate-prompt-tokens="true" />
<llm-emit-token-metric namespace="ai-gateway">
<dimension name="Subscription ID" />
<dimension name="API ID" />
</llm-emit-token-metric>
</inbound>
Where to run self-hosted models on Azure
AZURE CONTAINER APPS Simplest operations; scale-to-zero and event-driven scaling; GPU workload profiles for inference. Good for small and medium services and spiky traffic.
AKS Maximum control: GPU node pools, custom schedulers, serving stacks (Triton, vLLM, KServe), multi-model clusters. Needs Kubernetes expertise.
AZURE MACHINE LEARNING Managed online endpoints (autoscaling, traffic splitting for canary/blue-green, monitoring) and batch endpoints; strong if your team trains models there too.
VIRTUAL MACHINES GPU VMs for single-purpose or long-running workloads; the most manual option.
Choose the simplest option that meets your scale and control needs; move up only when you must.
Cost and governance on Azure
TAGS AND SEPARATE RESOURCES Tag resources by application, environment, and owner; use separate deployments/projects per application so billing exports attribute spend cleanly.
BUDGETS AND ALERTS Azure Cost Management budgets with alerts; provider-side spending limits where available.
POLICY Azure Policy to enforce private endpoints, disable key authentication, require content-filter policies, and restrict allowed regions and models.
PRIVATE NETWORKING Private endpoints and virtual-network integration for model, search, storage, and Key Vault; disable public access where feasible (see AI Security, Section 12).
6. Dockerized AI Applications
Why containers for AI
Containers give you reproducible builds, identical behavior from laptop to cluster, a unit that Kubernetes can schedule and scale, and a place to pin the native libraries (ONNX Runtime, OpenCV, CUDA) that make AI apps notoriously environment-sensitive.
A multi-stage Dockerfile for an ASP.NET Core inference service
# ---- build stage ----
FROM mcr.microsoft.com/dotnet/sdk:9.0 AS build
WORKDIR /src
# Copy project files first so the restore layer is cached until dependencies change
COPY *.sln ./
COPY src/VisionApi/VisionApi.csproj src/VisionApi/
RUN dotnet restore src/VisionApi/VisionApi.csproj
COPY . .
RUN dotnet publish src/VisionApi/VisionApi.csproj -c Release -o /app/publish --no-restore /p:UseAppHost=false
# ---- runtime stage ----
FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS runtime
WORKDIR /app
COPY --from=build /app/publish .
# The model is NOT baked into the image: it is mounted or fetched at startup (see below)
ENV ASPNETCORE_URLS=http://+:8080 \
MODEL_PATH=/models/model.onnx \
ORT_INTRA_OP_THREADS=4
EXPOSE 8080
USER $APP_UID # run as non-root (the base image defines APP_UID)
ENTRYPOINT ["dotnet", "VisionApi.dll"]
(Use the image tag matching your target .NET version, and pin base images by digest in production for reproducible builds.)
Where should the model live?
BAKED INTO THE IMAGE + Self-contained, simplest, versioned together with the app, fast startup (no download).
- Huge images (slow pulls and pushes, registry cost), every model update rebuilds and redeploys the image, secrets/licensing concerns if the model is sensitive.
Good for: small models (tens of MB) that change rarely.
MOUNTED VOLUME + Decouples model from image; many pods can share a read-only volume (Azure Files / Blob CSI); update the model without rebuilding.
- Volume performance and availability become dependencies; versioning is a convention you must enforce.
DOWNLOADED AT STARTUP + Image stays small; the model version comes from configuration; works with a registry or blob storage and checksum verification (Section 2).
(INIT CONTAINER OR APP) - Cold start includes the download — mitigate with node-local caching, an init container, and a startup probe that waits for readiness.
Good for: large or frequently updated models. The usual choice for GPU and LLM-scale artifacts.
RULE OF THUMB Small + stable: bake in. Large or changing: fetch/mount, verify the checksum, cache on the node.
Base images and native dependencies
GLIBC vs. MUSL Native ML libraries (ONNX Runtime, OpenCV, CUDA) are built for glibc. Alpine's musl libc often breaks them or lacks official builds. Prefer Debian/Ubuntu-based .NET images for AI workloads.
CHISELED IMAGES Minimal, non-root, shell-less Ubuntu images shrink size and attack surface — but they omit packages some native libraries need. Test thoroughly before adopting.
GPU IMAGES Use an NVIDIA CUDA runtime base (with cuDNN) matching the CUDA/cuDNN versions your ONNX Runtime GPU package requires, install the .NET runtime on top (or copy it in), and run with the NVIDIA
Container Toolkit (--gpus all locally; the device plugin in Kubernetes). A version mismatch is the classic cause of silent fallback to CPU (Section 10).
OPENCV Include the matching OpenCvSharp runtime package for your OS and any required system libraries; test in the final image, not on your dev machine.
KEEP IT SMALL Multi-stage builds; no SDK in the runtime image; a .dockerignore; don't copy test data or model files you don't need.
Container hygiene for AI services
NON-ROOT Run as an unprivileged user; read-only root filesystem where possible; drop capabilities.
NO SECRETS IN IMAGES Inject at runtime (environment from Key Vault, mounted secrets, workload identity). Images are copied around and scanned; assume they leak.
SCAN, SBOM, SIGN Vulnerability-scan (for example Trivy or Defender for Containers), produce an SBOM, sign images, and verify signatures at admission.
PIN EVERYTHING Base image digests, package versions, model checksums.
RESOURCE AWARENESS .NET respects container CPU and memory limits; set ONNX Runtime thread counts from the CPU limit, and size memory for the model plus request buffers (large image decodes, long prompts).
GRACEFUL SHUTDOWN Handle SIGTERM: stop accepting new work, let in-flight inference finish (HostOptions.ShutdownTimeout), then exit. Align with Kubernetes terminationGracePeriodSeconds.
LOGS TO STDOUT Structured logs and OpenTelemetry export — no local log files.
HEALTH ENDPOINTS /health/live (process up) and /health/ready (model loaded) for orchestrators (Section 1).
Local development with Compose
# docker-compose.yml — the app plus the dependencies it needs, and an OpenTelemetry viewer
services:
api:
build: .
ports: ["8080:8080"]
environment:
MODEL_PATH: /models/model.onnx
OTEL_EXPORTER_OTLP_ENDPOINT: http://aspire-dashboard:18889
volumes: ["./models:/models:ro"]
depends_on: [postgres]
postgres:
image: pgvector/pgvector:pg17 # PostgreSQL with the pgvector extension
environment: { POSTGRES_PASSWORD: dev-only-password }
ports: ["5432:5432"]
aspire-dashboard: # a lightweight OTLP viewer for traces, metrics, and logs during development
image: mcr.microsoft.com/dotnet/aspire-dashboard:latest
ports: ["18888:18888"]
environment: { DASHBOARD__FRONTEND__AUTHMODE: Unsecured } # local development ONLY
# ollama: # optional: a local model server for offline development
# image: ollama/ollama
# ports: ["11434:11434"]
(Dev-only settings such as unsecured dashboards and plaintext passwords must never reach shared environments. Never expose local model servers on public interfaces.)
7. Kubernetes AI Workloads
What changes for AI on Kubernetes
Kubernetes schedules containers by CPU and memory; AI workloads add GPUs (scarce, expensive, discrete), huge artifacts (multi-gigabyte models and images), slow startup (model load and warm-up), and non-obvious scaling signals (CPU is a poor proxy for GPU or LLM load).
A GPU inference Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: vision-api
labels: { app: vision-api }
spec:
replicas: 2
selector:
matchLabels: { app: vision-api }
strategy:
type: RollingUpdate
rollingUpdate: { maxSurge: 1, maxUnavailable: 0 } # keep capacity during rollouts — GPUs are scarce, so surge only if a spare GPU exists
template:
metadata:
labels:
app: vision-api
azure.workload.identity/use: "true" # AKS workload identity: keyless access to Azure services
spec:
serviceAccountName: vision-api
terminationGracePeriodSeconds: 60 # let in-flight inference finish
tolerations:
- { key: "sku", operator: "Equal", value: "gpu", effect: "NoSchedule" } # allow scheduling onto tainted GPU nodes
initContainers:
- name: fetch-model # download + verify the model before the app starts
image: myregistry.azurecr.io/model-fetcher:1.2
env:
- { name: MODEL_URI, value: "https://mystorage.blob.core.windows.net/models/detector/1.3.0/model.onnx" }
- { name: MODEL_SHA256, value: "<expected checksum>" }
volumeMounts:
- { name: models, mountPath: /models }
containers:
- name: api
image: myregistry.azurecr.io/vision-api:2.4.1 # immutable tag or digest — never :latest
ports:
- containerPort: 8080
env:
- { name: MODEL_PATH, value: /models/model.onnx }
resources:
requests: { cpu: "2", memory: 4Gi }
limits: { memory: 6Gi, nvidia.com/gpu: 1 } # the GPU is requested via the extended-resource limit
startupProbe: # tolerate a slow model load without being killed
httpGet: { path: /health/live, port: 8080 }
failureThreshold: 30
periodSeconds: 10
readinessProbe: # only receive traffic once the model is loaded and warm
httpGet: { path: /health/ready, port: 8080 }
periodSeconds: 5
livenessProbe:
httpGet: { path: /health/live, port: 8080 }
periodSeconds: 15
volumeMounts:
- { name: models, mountPath: /models, readOnly: true }
volumes:
- name: models
emptyDir: {}
The pieces that matter for AI
GPU SCHEDULING The NVIDIA device plugin exposes nvidia.com/gpu as a schedulable resource. Use dedicated GPU node pools, TAINT them so only GPU workloads land there (and tolerate the taint in your pods),
and use node selectors/affinity for specific GPU types. GPUs aren't overcommitted: each requested GPU is exclusive unless you enable sharing.
GPU SHARING Time-slicing, MPS, or MIG (on supported GPUs) split a GPU among pods — improving utilization for small models at the cost of isolation and predictability. Evaluate carefully.
PROBES startupProbe (slow load), readinessProbe (model warm), livenessProbe (process alive and NOT dependent on the model). Wrong probes cause restart loops and traffic to cold pods.
MODEL DELIVERY Init container download + checksum, or a shared read-only volume (Azure Files/Blob CSI), or node-local cache (Section 8). Avoid re-downloading gigabytes on every pod start.
IMAGE PULLS GPU/CUDA images are large; use ACR with artifact streaming or pre-pulled/cached images on nodes to cut cold start.
IDENTITY & SECRETS Workload identity for keyless Azure access; Key Vault via the Secrets Store CSI driver; no keys in manifests.
NETWORK POLICY Default-deny and explicit egress allow-lists — essential for agents and tools (see AI Security, Section 9).
DISRUPTION PodDisruptionBudgets so node upgrades don't remove all replicas; topology spread constraints for zone resilience.
QUOTAS ResourceQuota/LimitRange per namespace to stop one team from consuming all GPUs.
OBSERVABILITY OpenTelemetry Collector (as a Deployment or DaemonSet), GPU metrics via the NVIDIA DCGM exporter into Prometheus/Azure Monitor, and request-level metrics from the app.
Autoscaling AI workloads
CPU-BASED HPA IS OFTEN THE WRONG SIGNAL. A GPU-bound or LLM-calling service can be saturated while CPU sits idle, and a service waiting on a remote model does little CPU work at all.
BETTER SIGNALS
- Concurrent in-flight requests (or requests per second per pod) — exported as a custom metric
- Queue depth / message backlog (for asynchronous workloads)
- GPU utilization or GPU memory (from DCGM)
- Latency percentiles / SLO burn (as a guard, not usually the primary scaler)
TOOLS
- HPA with custom/external metrics (Prometheus adapter or Azure Monitor metrics)
- KEDA — event-driven autoscaling from queues, Prometheus queries, and more; supports scale to ZERO
- Cluster Autoscaler / node auto-provisioning to add (and remove) GPU nodes — but node provisioning takes minutes, so combine with warm capacity for latency-sensitive services
# KEDA: scale an embedding worker on queue backlog, down to zero when idle
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: embed-worker
spec:
scaleTargetRef:
name: embed-worker
minReplicaCount: 0
maxReplicaCount: 20
cooldownPeriod: 300
triggers:
- type: azure-servicebus
metadata:
queueName: embed-jobs
messageCount: "50" # target backlog per replica
authenticationRef:
name: servicebus-auth # or workload-identity-based authentication
Batch and scheduled AI work
JOBS / CRONJOBS Offline inference, embedding refresh, ingestion, nightly canary evaluations, drift computation — run as Kubernetes Jobs/CronJobs (on cheaper or spot capacity where interruptions are acceptable),
with retries, idempotent processing, and results written to durable storage.
PIPELINES Multi-step training/evaluation/registration workflows run in an orchestrator (Azure ML pipelines, Argo Workflows, Kubeflow) with artifact and lineage tracking.
SPOT / PREEMPTIBLE Significant savings for interruptible batch and training work if you checkpoint and tolerate eviction; avoid for latency-sensitive serving unless you over-provision.
Serving stacks
CUSTOM ASP.NET CORE SERVICE Simplest for ONNX/ML.NET models you control (as in this series).
NVIDIA TRITON A multi-framework, multi-model inference server with dynamic batching, concurrent model execution, and metrics.
vLLM / TEXT-GENERATION SERVERS High-throughput serving of open-weight LLMs with continuous batching and efficient KV-cache management.
KServe / AZURE ML ENDPOINTS Standardized model-serving abstractions with autoscaling and canary traffic splitting.
KAITO (on AKS) Automates provisioning of GPU nodes and deployment of open-weight models.
OLLAMA / LLAMA.CPP Convenient for local development and small deployments, not usually the production-scale answer.
8. Model Caching
"Caching" in AI systems means several different things. Each saves a different cost — and each has distinct staleness and security risks.
The kinds of caching
1. RESPONSE CACHE (EXACT) Same inputs -> stored output. Skips the model entirely.
2. SEMANTIC CACHE "Similar enough" questions share a stored answer, matched by embedding similarity.
3. PROVIDER PROMPT CACHE The provider discounts and speeds up requests that begin with a recently processed prompt prefix.
4. EMBEDDING CACHE Content hash -> vector. Avoids re-embedding unchanged text or repeated queries.
5. KV CACHE (self-hosted LLMs) The attention key/value cache inside the inference server, including prefix caching across requests.
6. MODEL ARTIFACT CACHE Model files cached on nodes/volumes so pods don't re-download them; image layers cached; warm pools of loaded models.
7. IN-PROCESS MODEL CACHE The loaded model session held in memory (singleton), or an LRU of several models when you serve many.
8. PREDICTION CACHE For deterministic classical models: input features -> prediction.
Exact response caching
// HybridCache (.NET 9+): a two-level cache (in-memory + distributed) with built-in stampede protection — concurrent callers for the same key trigger ONE model call
public sealed class SummaryService(HybridCache cache, IChatClient chat)
{
public async Task<string> SummarizeAsync(string text, CancellationToken ct)
{
// The key must include EVERYTHING that changes the answer: prompt version, model, parameters, and the input hash
string key = $"summary:{Prompts.SummaryVersion}:{Models.SummaryDeployment}:{Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text)))}";
return await cache.GetOrCreateAsync(
key,
async token => (await chat.GetResponseAsync($"Summarize in two sentences:\n\n{text}", new ChatOptions { Temperature = 0 }, token)).Text,
new HybridCacheEntryOptions { Expiration = TimeSpan.FromHours(12), LocalCacheExpiration = TimeSpan.FromMinutes(5) },
cancellationToken: ct);
}
}
RULES FOR RESPONSE CACHING
CACHE ONLY WHERE IT'S SAFE: deterministic-ish, non-personalized, not time-sensitive requests (temperature 0 summaries, classifications, FAQ answers). Never cache answers that depend on the user's
permissions or live data unless the key includes the user/tenant and the freshness requirement is met.
KEY EVERYTHING: prompt version, model/deployment, parameters, input hash, tenant (when relevant). A stale key serves old-prompt answers after a deployment.
INVALIDATE: version the key on prompt/model/index changes; use TTLs; use cache tags to evict related entries.
MIND PRIVACY: cached responses are stored data — apply the same access control, encryption, and retention as the source (see AI Security, Section 4).
MEASURE: hit rate, saved tokens/cost, and staleness complaints (see AI Observability).
Semantic caching — powerful and risky
// Match by MEANING: "How do I reset my password?" and "password reset steps?" can share one answer
public sealed class SemanticCache(IEmbeddingGenerator<string, Embedding<float>> embeddings, ICacheVectorStore store)
{
public async Task<string?> TryGetAsync(string question, string tenantId, string promptVersion, CancellationToken ct)
{
ReadOnlyMemory<float> vector = await embeddings.GenerateVectorAsync(question, cancellationToken: ct);
// Scope the lookup: SAME tenant, SAME prompt version/model — otherwise users see each other's answers, or stale-prompt answers
CacheHit? hit = await store.NearestAsync(vector, tenantId, promptVersion, ct);
return hit is { Score: >= 0.95f } ? hit.Answer : null; // a HIGH threshold, tuned on real traffic
}
}
SEMANTIC CACHE HAZARDS
FALSE HITS "Cancel my order" and "Don't cancel my order" embed very close together (embeddings blur negation and numbers). A wrong cache hit is a confidently wrong answer. Use a high threshold, restrict to
low-risk, FAQ-style domains, and consider exact-match verification or a lightweight re-check.
TENANT LEAKS A cache keyed only on similarity serves one user's answer to another. Always scope by tenant/permissions (see AI Security, Section 4).
STALENESS Cached answers outlive the documents, prompts, or policies they were based on. Invalidate on corpus, prompt, and model changes, and keep TTLs short.
PERSONALIZATION Don't semantically cache answers that depend on the individual user.
COST/BENEFIT Every lookup costs an embedding call; it pays off only with meaningful hit rates on repetitive traffic. Measure before and after.
Managed options exist (for example API Management's semantic caching policies backed by Redis) — the same caveats apply.
Provider prompt caching: arrange prompts to benefit
Providers can automatically discount and speed up requests whose beginning matches a recently processed prompt (typically applying above a minimum prompt length — check current thresholds and discounts).
TO BENEFIT:
- Put STATIC content FIRST: system prompt, instructions, tool definitions, reference documents. Put VARIABLE content LAST: the user's question, retrieved passages that change per request.
- Keep the static prefix BYTE-IDENTICAL across requests (no timestamps, request IDs, or reordered tools at the top).
- Track the cached-token share (see AI Observability, Section 3): a drop means your prefix changed.
Embedding and prediction caching
EMBEDDINGS Key: hash(text) + embedding model/version. Embedding the same document chunk or popular query repeatedly is pure waste — Microsoft.Extensions.AI's UseDistributedCache middleware handles this for
embedding generators (see Microsoft.Extensions.AI, Section 3). Include the model version in the key.
PREDICTIONS For deterministic models (classifiers, scorers) a cache from input features to prediction works; invalidate on model version change (include the version in the key).
Caching models themselves: cold-start control
IN-PROCESS Load once (singleton), warm up, reuse. Don't load per request.
MANY MODELS Keep an LRU of loaded models bounded by memory; load on demand, evict the least recently used; protect against load stampedes (lock per model).
NODE-LOCAL CACHE Download model artifacts to node-local disk (or a shared volume) once per node and reuse across pods and restarts, verifying checksums. Avoid every pod pulling gigabytes.
IMAGE LAYERS Order Dockerfile layers so large, rarely changing layers (CUDA, native libraries) are cached; use registry features that stream or pre-pull images to nodes.
WARM CAPACITY Keep a minimum number of warm replicas for latency-sensitive services; scale-to-zero trades cost for cold-start latency.
KV / PREFIX CACHE For self-hosted LLM servers, enable prefix caching and size GPU memory for KV cache — it often determines concurrency more than model weights do.
9. Scaling Inference
Start with the arithmetic
LITTLE'S LAW: concurrency = throughput x latency
A service handling 50 requests/second at 0.4 s per request has ~20 requests in flight on average.
If one replica can handle 5 concurrent requests before latency degrades, you need ~4 replicas at average load — plus headroom for peaks (commonly 30-50%) and for losing a replica.
For LLM calls to a hosted provider, also compute TOKEN throughput: requests/s x (input + output tokens) must fit your tokens-per-minute quota.
Always LOAD TEST: measure latency percentiles at increasing load on production-like hardware, find the knee where latency climbs, and set capacity and autoscaling targets below it.
Levers for scaling, in rough order of leverage
1. DON'T DO THE WORK Caching (Section 8), deduplication, cheaper routing — the fastest request is the one you skip.
2. MAKE EACH REQUEST CHEAPER Smaller or distilled models, quantization (FP16/INT8), shorter prompts and outputs, graph optimization, efficient runtimes (ONNX Runtime, TensorRT, OpenVINO, DirectML).
3. BATCH Process several requests per model invocation — raises throughput substantially on GPUs (and helps CPUs), at a latency cost (below).
4. SCALE OUT More stateless replicas behind a load balancer, autoscaled on the right signals (Section 7).
5. SCALE UP Bigger or faster hardware (more cores, a GPU, a larger GPU) when one model invocation itself is the bottleneck.
6. SHIFT WORK OFF THE REQUEST PATH Queues and background workers for anything users aren't waiting on.
Stateless replicas and load balancing
Inference services should be stateless (conversation and cache state live in external stores) so any replica can serve any request and replicas can be added or removed freely.
LOAD BALANCING GOTCHA: inference request durations vary enormously (a 50-token answer vs. a 2,000-token answer). Plain round-robin can pile long requests on one replica. Prefer
least-connections or queue-aware balancing, and cap per-replica concurrency so overload is rejected quickly instead of queueing indefinitely.
Backpressure and load shedding
// A bounded queue between the API and the workers: when the system is saturated, say so (503 + Retry-After) instead of accepting unbounded work and failing slowly
public sealed class InferenceQueue
{
private readonly Channel<InferenceJob> _channel = Channel.CreateBounded<InferenceJob>(new BoundedChannelOptions(200)
{
SingleReader = false, SingleWriter = false
});
public bool TryEnqueue(InferenceJob job) => _channel.Writer.TryWrite(job); // returns false immediately when the queue is full
public IAsyncEnumerable<InferenceJob> ReadAllAsync(CancellationToken ct) => _channel.Reader.ReadAllAsync(ct);
}
app.MapPost("/analyze", async (AnalyzeRequest req, InferenceQueue queue) =>
{
var job = new InferenceJob(Guid.NewGuid(), req.ImageBytes);
if (!queue.TryEnqueue(job))
return Results.StatusCode(StatusCodes.Status503ServiceUnavailable); // shed load; the client retries after the Retry-After interval
return Results.Accepted($"/results/{job.Id}", new { job.Id }); // asynchronous pattern: poll /results/{id} or get a callback
});
ADMISSION CONTROL Reject early when overloaded (429/503 with Retry-After), cap concurrent requests per replica, and give each request a deadline. Slow failure under overload is worse than fast failure.
ASYNC WORKFLOWS For slow work (large documents, long video, big batches), accept the job, return an ID, process with background workers (KEDA-scaled on queue depth), and notify by polling, webhook, or
push. Make jobs idempotent and retries safe.
Dynamic batching (micro-batching) in-process
GPUs (and vectorized CPU kernels) are far more efficient per item when fed batches. A micro-batcher collects requests for a few milliseconds, runs them as one batch, and returns each caller's result.
public sealed class MicroBatcher<TIn, TOut>(
Func<IReadOnlyList<TIn>, Task<IReadOnlyList<TOut>>> runBatch, int maxBatchSize, TimeSpan maxWait)
{
private readonly record struct Item(TIn Input, TaskCompletionSource<TOut> Completion);
private readonly Channel<Item> _channel = Channel.CreateUnbounded<Item>();
// Callers await this; the result arrives when the batch containing their item has run
public Task<TOut> SubmitAsync(TIn input)
{
var tcs = new TaskCompletionSource<TOut>(TaskCreationOptions.RunContinuationsAsynchronously);
_channel.Writer.TryWrite(new Item(input, tcs));
return tcs.Task;
}
// Run this as a hosted background service
public async Task RunAsync(CancellationToken ct)
{
ChannelReader<Item> reader = _channel.Reader;
while (await reader.WaitToReadAsync(ct))
{
var batch = new List<Item>(maxBatchSize);
// The first item has arrived: allow up to maxWait for more to join, or until the batch is full
using var window = CancellationTokenSource.CreateLinkedTokenSource(ct);
window.CancelAfter(maxWait);
try
{
while (batch.Count < maxBatchSize && await reader.WaitToReadAsync(window.Token))
while (batch.Count < maxBatchSize && reader.TryRead(out Item item))
batch.Add(item);
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested) { /* the wait window elapsed: run what we have */ }
if (batch.Count == 0) continue;
try
{
IReadOnlyList<TOut> outputs = await runBatch(batch.Select(b => b.Input).ToList());
for (int i = 0; i < batch.Count; i++) batch[i].Completion.SetResult(outputs[i]);
}
catch (Exception ex)
{
foreach (Item item in batch) item.Completion.SetException(ex); // one failed batch fails its callers, not the whole service
}
}
}
}
THE TRADE-OFF: batching raises throughput but adds up to maxWait of latency per request. Tune maxBatchSize and maxWait against your latency SLO and measure throughput per dollar. Dedicated servers
(Triton's dynamic batching; vLLM's continuous batching for LLMs) implement this far more elaborately — use them for serious GPU serving, and the pattern above for simpler in-process cases.
Optimizing the model itself
QUANTIZATION Reduce numeric precision (FP32 -> FP16 -> INT8, and lower for LLMs) to cut memory and increase speed, usually with a small accuracy cost — always re-evaluate quality after quantizing.
DISTILLATION Train a small "student" model to mimic a large "teacher" — often the best accuracy-per-cost for a specific task.
PRUNING / SMALLER ARCHITECTURES Remove redundancy or pick a more efficient architecture for the task.
GRAPH OPTIMIZATION ONNX Runtime operator fusion and constant folding; hardware-specific runtimes (TensorRT for NVIDIA GPUs, OpenVINO for Intel CPUs, DirectML on Windows GPUs/NPUs).
SHORTER SEQUENCES Trim prompts and outputs; resize images to the smallest resolution that preserves accuracy.
RIGHT-SIZE THE MODEL Use the smallest model that meets the quality bar — then route harder requests to a bigger one (see LLM Application Development, Section 9).
Scaling hosted-LLM usage
QUOTAS ARE THE CEILING Your throughput is capped by tokens-per-minute and requests-per-minute quotas. Plan capacity from token estimates, not request counts.
MULTIPLE DEPLOYMENTS/REGIONS Spread load across several deployments (and regions where data-residency allows) behind a gateway that load-balances, honors Retry-After, and trips circuit breakers on 429s and 5xx.
PROVISIONED + STANDARD Reserved (provisioned) capacity for the steady baseline, with pay-as-you-go capacity as burst or fallback.
BATCH API FOR OFFLINE WORK Move non-interactive jobs off the real-time quota and onto discounted asynchronous batch processing.
RESILIENCE Retries with jittered backoff, timeouts, circuit breakers, and fallbacks (cheaper model, cached answer, degraded feature) — see AI APIs with .NET, Section 10.
CLIENT-SIDE LIMITS Bound concurrency and token rates in your app so you degrade gracefully before the provider rejects you (see AI Security, Section 11).
Multi-region and failover
ACTIVE-ACTIVE OR ACTIVE-PASSIVE across regions for the application, the index, and model endpoints, behind a global load balancer (Azure Front Door/Traffic Manager). Replicate indexes and artifacts;
keep model versions and prompts consistent across regions; test failover regularly; and account for data-residency constraints (not every region may be eligible for every tenant).
10. GPU vs. CPU Considerations
The decision is about utilization and cost per inference, not raw speed
A GPU does a lot of parallel arithmetic quickly, but it is expensive whether or not it is busy. A CPU is slower per operation but cheap, flexible, and everywhere. The right choice depends on model size, traffic shape, latency target, and — above all — how well you can keep the hardware utilized.
Where GPUs win
LARGE MODELS Transformers, large vision models, and LLMs with many millions or billions of parameters.
HIGH, STEADY THROUGHPUT Enough traffic to keep the GPU busy, especially with batching.
LATENCY-CRITICAL HEAVY MODELS When a CPU simply can't meet the latency target for the model size.
BATCH EMBEDDING / BULK INFERENCE Large offline jobs where throughput per dollar dominates.
SELF-HOSTED LLM SERVING Practically requires GPUs (or accelerators) for useful speed at scale.
Where CPUs win
SMALL MODELS Classical ML, gradient-boosted trees, small neural nets, small embedding/classification models (tens to low hundreds of millions of parameters, quantized).
LOW OR BURSTY TRAFFIC An idle GPU burns money; CPU instances scale to zero or run cheaply at low utilization.
BATCH SIZE OF ONE, LOW LATENCY Small single-item inferences often run just as fast on CPU (no host-to-device transfer overhead).
OPERATIONAL SIMPLICITY No driver/CUDA/image-version matrix, easier scheduling and autoscaling, broader availability.
QUANTIZED MODELS INT8/INT4 on modern CPUs (AVX2/AVX-512/VNNI instructions) can be surprisingly fast for the right models.
ARM CPUs Often the best price/performance for CPU-bound inference where supported.
The binding constraint is often memory, not compute
MODEL MEMORY (rule of thumb): parameters x bytes per parameter
7-billion-parameter model: FP16 (2 bytes) ≈ 14 GB; INT8 ≈ 7 GB; 4-bit ≈ 3.5-4 GB — PLUS the KV cache and activations
KV CACHE grows with context length and concurrent sequences; for LLM serving it frequently limits concurrency more than the weights do.
A model that doesn't fit in GPU memory doesn't run (or must be split across GPUs/offloaded to CPU at a heavy speed penalty).
Quantization is therefore both a cost lever and a feasibility lever. For LLM decoding, speed is often limited by memory BANDWIDTH rather than raw compute.
A decision table
| Situation | Likely best choice |
|---|---|
| Classical ML (trees, linear models), ML.NET | CPU |
| Small classifier/embedder, low-to-moderate traffic | CPU (quantized if needed) |
| Object detection at video frame rates | GPU |
| Large vision/transformer model, high throughput | GPU with batching |
| Bursty or sporadic usage | CPU, or serverless/scale-to-zero (GPU only if the model demands it) |
| Self-hosted LLM for many users | GPU (right-sized, with a serving stack that batches continuously) |
| Self-hosted small language model, few users, modest speed needs | Quantized on CPU may suffice |
| Offline bulk embedding/inference | GPU or many CPUs — compare cost per million items; spot capacity helps |
| Edge/desktop devices | NPU/integrated GPU via DirectML or vendor runtimes, with a CPU fallback |
| Hosted LLM API | Not applicable — you choose a deployment type and quota, not hardware |
Benchmark on your own model and traffic
Don't decide from spec sheets. Measure, for each candidate hardware option (and a few batch sizes):
THROUGHPUT items (or tokens) per second at saturation
LATENCY p50/p95/p99 at your expected concurrency
COST PER 1,000 INFERENCES = instance cost per hour / (throughput per hour / 1000), at realistic UTILIZATION (not 100%)
MEMORY peak usage; headroom for your largest inputs
QUALITY after quantization or precision changes — re-run your evaluation set
Then compare cost per inference at the utilization you will actually achieve. A GPU at 15% utilization can cost several times more per inference than a CPU fleet — and a CPU fleet that can't meet
your latency target is not an option.
.NET-specific GPU considerations
PACKAGES Microsoft.ML.OnnxRuntime (CPU), .Gpu (CUDA), .DirectML (Windows GPUs/NPUs). Use the package matching your hardware and keep CUDA/cuDNN versions aligned with it.
SILENT FALLBACK If the CUDA libraries are missing or mismatched, ONNX Runtime may quietly run on the CPU — and you will pay GPU prices for CPU speed. VERIFY at startup and in tests:
// Fail fast (or at least alert loudly) if the GPU execution provider isn't actually available
string[] providers = OrtEnv.Instance().GetAvailableProviders(); // e.g. ["CUDAExecutionProvider", "CPUExecutionProvider"]
if (config.RequireGpu && !providers.Contains("CUDAExecutionProvider"))
throw new InvalidOperationException($"GPU required but unavailable. Providers: {string.Join(", ", providers)}");
var options = new SessionOptions();
if (providers.Contains("CUDAExecutionProvider") && config.UseGpu)
options.AppendExecutionProvider_CUDA(deviceId: 0);
log.LogInformation("Inference providers available: {Providers}; GPU enabled: {Gpu}", providers, config.UseGpu);
ALSO
- Emit GPU utilization and memory metrics (DCGM exporter) and alert on persistently LOW utilization (wasted money) as well as saturation.
- Pre/post-processing (image decode, resize, NMS) often runs on the CPU and can bottleneck a fast GPU — profile the whole pipeline, and parallelize or move steps to the GPU where it helps.
- Keep a CPU-only code path and image for development, CI, and fallback.
- Small language models can run locally with ONNX Runtime GenAI or similar runtimes on CPU/GPU/NPU — useful for edge and privacy scenarios.
11. Putting It Together
A reference production architecture on Azure
DEVELOPER ──> GitHub (code · prompts · IaC · eval sets · manifests)
│ pull request
v
┌── CI/CD (GitHub Actions, OIDC) ──────────────────────────────────────────────┐
│ build · unit tests (fakes) · scans · EVAL + SECURITY GATES · image build/scan/sign · IaC what-if │
└───────────────┬──────────────────────────────────────────────┬────────────────┘
v v
AZURE CONTAINER REGISTRY MODEL REGISTRY / BLOB (versioned, checksummed models)
│ │
v v
┌──────────── RUNTIME (AKS or Container Apps) ───────────────────────────────────────────┐
│ AI API (ASP.NET Core) ──> Redis (cache) · Azure AI Search / pgvector (retrieval) │
│ │ │
│ ├──> inference service(s) on GPU/CPU pools (ONNX/Triton) — init-container model fetch, warm readiness │
│ └──> API MANAGEMENT (AI GATEWAY): auth · token limits · metrics · semantic cache · failover │
│ │ │
│ v │
│ AZURE OPENAI deployments (pinned versions; provisioned + standard; multi-region) │
│ Workers (KEDA on queues): ingestion · embedding · batch inference · scheduled canary evaluations │
└──────────────────────┬──────────────────────────────────────────────────────────────────┘
v
OBSERVABILITY: OpenTelemetry → Collector → Azure Monitor / Prometheus / Grafana
(traces · token/cost/latency metrics · drift · quality scores) → alerts · SLOs · dashboards
Key Vault + managed/workload identity everywhere · private endpoints · Azure Policy
What to run when each kind of thing changes
| Change | Pipeline checks | Rollout | Rollback |
|---|---|---|---|
| Application code | Build, unit/integration tests, scans, smoke tests | Rolling or canary | Redeploy the previous image |
| Prompt | Lint, evaluation gate, injection suite, cost/latency check, review | Config rollout → canary by traffic share | Re-point to the previous prompt version |
| Hosted model version | Evaluation on the new version in a parallel deployment, security suite, cost/latency | Shadow → canary → switch deployment | Switch back to the previous (pinned) deployment |
| Self-hosted model | Registration with metrics and manifest, held-out + recent-data evaluation, benchmark on target hardware | Shadow/canary/blue-green with approval | Re-point the production alias to the previous version |
| Retrieval index / chunking | Build new index, retrieval metrics (Recall@k, MRR), end-to-end eval | Build alongside, switch alias | Switch the alias back; keep the old index until proven |
| Embedding model | Re-embed the corpus into a NEW index, full evaluation | New index + parallel query path, then cut over | Keep the old index and embedder running until validated |
| Infrastructure / quotas / policies | IaC what-if, policy checks, capacity check | Staged environments | Revert the IaC change |
| Tool definitions / permissions | Security suite, authorization tests, review | Canary | Revert and revoke |
The feedback loop that makes it MLOps
MONITOR (drift, quality, cost, latency) -> ALERT / REVIEW -> DIAGNOSE (trace-level debugging) -> FIX (prompt, index, model, data, code)
^ │
└── PROMOTE ← CANARY ← EVALUATE (golden set + recent production samples + security suite) ← VERSIONED CHANGE ┘
Every production failure becomes a test case; every model/prompt/index change is versioned, evaluated, rolled out gradually, and reversible.
12. Production Readiness Checklist
RELIABILITY
[ ] SLOs defined (availability, latency incl. TTFT, quality, cost) with error budgets and burn-rate alerts
[ ] Load-tested at expected peak + headroom on production-like hardware; capacity and autoscaling targets set below the latency knee
[ ] Timeouts, retries with jittered backoff, circuit breakers, and fallbacks (cheaper model / cached answer / degraded mode) implemented and tested
[ ] Backpressure and load shedding (bounded queues, 429/503 with Retry-After), per-replica concurrency caps
[ ] Readiness/liveness/startup probes correct (liveness independent of the model); graceful shutdown drains in-flight work
[ ] Multi-zone (and where needed multi-region) deployment; failover tested; PodDisruptionBudgets in place
[ ] Provider quotas sized from token estimates; quota-increase requests filed; multiple deployments/regions behind a gateway
VERSIONING AND RELEASES
[ ] Models, tokenizers/labels, prompts, indexes, tools, eval sets, and IaC all versioned; versions recorded on every trace
[ ] Hosted-model versions pinned; retirement dates tracked; migration plan with evaluation
[ ] Model checksums verified at load; registry with promotion stages and audit trail
[ ] Canary/shadow/blue-green rollout with automatic rollback criteria defined BEFORE release; previous version always deployable
QUALITY AND EVALUATION
[ ] Golden set (with real failures and adversarial cases) run as a CI gate; baselines stored; repetitions to handle non-determinism
[ ] Scheduled canary evaluation in production; model-version-change alerts; drift monitoring on inputs and predictions
[ ] Ground-truth pipeline (human review, feedback, delayed labels) feeding performance measurement and retraining
SECURITY AND PRIVACY (see AI Security)
[ ] Keyless/managed identity; no secrets in images, prompts, clients, or logs; secret scanning in CI
[ ] Private networking; model access through a gateway; server-side model/deployment selection and content policies
[ ] Authorization at the data layer; tenant isolation tests; adversarial suite as a release gate
[ ] Image scanning, SBOM, signing; non-root containers; network policies with egress allow-lists
[ ] Telemetry privacy: metadata-only by default, redaction, retention and deletion procedures
COST
[ ] Cost attribution by feature/tenant/model; budgets, burn-rate and anomaly alerts; provider-side spend limits
[ ] Caching strategy with correct keys and invalidation; routing to right-sized models; batch API for offline work
[ ] GPU utilization monitored (alert on idle waste); hardware choice justified by cost per inference at real utilization
OPERATIONS
[ ] Dashboards for ops, cost, quality, and security; alerts with owners and runbooks; on-call rotation
[ ] Kill switches (per feature/tool/model) and credential-rotation runbooks; incident process and postmortems
[ ] Documented architecture, data flows, dependencies, and DR plan; regular game days
13. Common Pitfalls
| Pitfall | Why it hurts | Better approach |
|---|---|---|
| Relying on an unpinned model alias | Silent provider upgrades change behavior and quality | Pin versions; migrate deliberately with evaluation (Sections 1, 2) |
| Treating prompts and indexes as "not versioned" | Behavior changes can't be attributed or rolled back | Version prompts, indexes, tools, and eval sets; tag traces (Section 2) |
| Serving traffic before the model is loaded and warm | Errors and timeouts during startup and rollouts | Readiness probes gated on load + warm-up (Section 1) |
| Liveness probe that depends on the model | Slow load triggers restart loops | Liveness checks the process only; readiness checks the model (Sections 1, 7) |
| Loading the model per request | Slow, memory-hungry | One session per process, reused (Section 1) |
| No checksum verification of model artifacts | Corrupted or tampered models go live | Verify against the manifest at load (Section 2) |
| Deploying models without a registry or lineage | Can't reproduce, audit, or roll back | Immutable versions with lineage, promotion stages, and aliases (Section 2) |
| Monitoring only system health | Quality and drift problems go unnoticed | Add data/prediction drift, canary evaluation, ground-truth sampling (Section 3) |
| Waiting for ground-truth labels before monitoring anything | Delayed labels mean late detection | Use drift, confidence, and proxy metrics in the meantime (Section 3) |
| Evaluating AI only manually, before launch | Regressions slip through every change | Automated evaluation gates in CI with baselines and repetitions (Section 4) |
| Running expensive evals on every commit | Slow pipelines and surprise bills | Path filters, smoke subsets on PRs, full suites on merge or nightly, response caching (Section 4) |
| Stored cloud credentials in CI | Leak risk; hard to rotate | OIDC federation with scoped, inference-only identities (Section 4) |
| Rebuilding artifacts per environment | "Works in staging" differences | Build once, promote the same image and versions (Section 4) |
| Alpine/musl images for native ML libraries | Missing or broken native dependencies | Debian/Ubuntu-based images; test the final image (Section 6) |
| Baking huge models into images | Slow pulls, large registry, rebuilds for every model update | Fetch/mount large models with checksums and node-local caching (Section 6) |
| GPU image/driver/library mismatches | Silent fallback to CPU at GPU prices | Match CUDA/cuDNN to the runtime; verify providers at startup; test (Sections 6, 10) |
| CPU-based autoscaling for GPU or LLM-calling services | Saturation without CPU load; scaling too late or never | Scale on concurrency, queue depth, or GPU metrics; KEDA for event-driven work (Section 7) |
| No taints/tolerations or quotas for GPU nodes | Non-GPU workloads waste GPUs; teams starve each other | Dedicated tainted pools, resource quotas, node selectors (Section 7) |
| Round-robin load balancing for LLM-style requests | Long requests pile up on some replicas | Least-connections or queue-aware balancing; per-replica concurrency caps (Section 9) |
| Unbounded queues and no load shedding | Latency explodes and failure is slow and total | Bounded queues; fast 429/503 with Retry-After; deadlines (Section 9) |
| Semantic cache without tenant scoping or a high threshold | Cross-user leaks and confidently wrong answers | Scope by tenant, prompt version, and model; high threshold; low-risk domains only (Section 8) |
| Cache keys that omit prompt/model version | Stale answers after deployments | Include everything that changes the answer in the key; invalidate on change (Section 8) |
| Choosing hardware from spec sheets | Paying for idle GPUs or missing latency targets | Benchmark cost per inference at realistic utilization; re-evaluate quality after quantization (Section 10) |
| Ignoring memory (weights + KV cache) when sizing GPUs | Model doesn't fit or concurrency collapses | Size for weights plus KV cache and activations; consider quantization (Section 10) |
| No production-readiness review | Launch surprises: quotas, rollbacks, alerts, runbooks | Use the checklist; load test; rehearse failover and rollback (Section 12) |
Quick Reference Table
| Topic | Key idea | Practical takeaway |
|---|---|---|
| Model deployment | Managed endpoint, in-process, separate service, serverless, or edge | Match to scale and control needs; declare deployments as code; gate traffic on readiness |
| Release strategies | Blue/green, canary, shadow, A/B | Define rollback criteria first; always keep the previous version deployable |
| Model versioning | Version every behavior-affecting artifact | Immutable registry versions with lineage; checksum verification; record versions on traces |
| Hosted model lifecycle | Providers retire and update models | Pin versions, track retirement dates, migrate via parallel deployment + evaluation |
| Model monitoring | Drift, prediction drift, performance, business impact | PSI/KS on live vs reference; canary golden sets; ground-truth pipelines; retrain with approval |
| CI/CD for AI | Evaluation and security gates alongside tests | Fakes for unit tests; golden-set gate with baselines; OIDC credentials; build once, promote |
| Azure AI services | Azure OpenAI/Foundry, AI Search, ML, Container Apps, AKS, APIM | Gateway in front of models; pinned deployments; private networking; tags and budgets |
| Docker for AI | Reproducible images with native ML libraries | Multi-stage, non-root, Debian-based, model fetched not baked (when large), scanned and signed |
| Kubernetes for AI | GPU scheduling, probes, model delivery, custom-metric autoscaling | Taint GPU pools; startup/readiness probes; KEDA or custom-metric HPA; init-container model fetch |
| Model caching | Response, semantic, provider prompt, embedding, KV, artifact | Key on everything that changes the answer; scope by tenant; static prefix first; node-local model cache |
| Scaling inference | Little's law, caching, batching, scale-out, backpressure | Load test; bounded queues and shedding; micro-batching; quantization; right-sized models |
| GPU vs CPU | Utilization and cost per inference decide | CPU for small/bursty; GPU for large/steady; size for memory; verify GPU is actually used |
| Readiness | Production checklist | SLOs, rollback, quotas, security, cost, runbooks, kill switches |
Conclusion
Production AI is mostly the disciplined application of ideas you already know — versioned artifacts, automated pipelines, gradual rollouts, health probes, autoscaling, monitoring, and rollback — adapted to systems whose behavior is statistical, whose dependencies change without notice, and whose cost scales with usage. The adaptations are what matter: version everything that affects behavior (weights, tokenizers, prompts, indexes, tools, evaluation sets) and stamp each request with those versions; gate releases on evaluation and security tests, not just unit tests, using repeated runs and baselines to handle non-determinism; monitor drift and quality, not just uptime, with canary evaluations that catch the silent provider and data changes nobody announces; and roll out gradually through shadow, canary, and blue/green strategies with rollback criteria defined in advance.
On the infrastructure side, the same few themes recur. Gate traffic on readiness, not on process start. Keep model delivery out of the request path (verified downloads, node-local caches, warm capacity). Scale on the right signals (concurrency, queue depth, GPU metrics) and shed load early instead of queueing until everything is slow. Put a gateway in front of hosted models for authentication, token limits, metrics, caching, and failover, and use keyless identity everywhere. Cache aggressively but safely — keyed on everything that changes the answer, scoped by tenant, invalidated on every version change. And choose hardware by cost per inference at the utilization you will really achieve, sizing for memory as well as compute and verifying that the GPU you pay for is actually the one doing the work. Pair those practices with a readiness checklist, rehearsed rollbacks, kill switches, and runbooks, and an AI feature stops being a demo that works until it doesn't and becomes a service your team can change confidently, operate predictably, and afford.
Found this useful? Feel free to star the repo, open an issue with corrections, or share the "the provider quietly updated the model and nobody noticed for a week" story that made the case for version pinning and canary evaluations click better than any runbook.
Top comments (0)