Chethan Ramaswamy

Posted on Nov 1 Edited on Nov 1

I Tested Building an Enterprise LLMOps Pipeline on Microsoft Azure. Here’s What Surprised Me.

#azure #openai #llm #ai

LLMOps on Microsoft Azure: A Practical Guide to Foundry, RAG, Evaluation, and Guardrails

Getting a Large Language Model (LLM) to answer questions on localhost takes about 10 minutes and a few lines of Python.

Getting that same model into production, where it doesn't invent company policy, leak user context, or run up a surprise API bill on a Tuesday night, is where things get real.

If your team builds on the Microsoft ecosystem, you've probably heard terms like Microsoft Foundry, Azure AI Search, and Agent Framework. This guide shows how those pieces fit together in a production-grade system, which parts have changed recently, and the traps worth avoiding.

Last verified: November 2026. Foundry changes quickly, so check model names, retirement dates, and feature status against the current Microsoft docs before you ship.

The core idea: MLOps is not LLMOps

If you come from classic MLOps, it's tempting to treat LLMOps as the same thing with extra steps. It isn't.

In traditional MLOps, your main artifact is a trained model, and you measure it with metrics like accuracy, F1, or MSE. In LLMOps, you usually consume a foundation model as a service. Much of your "code" becomes the prompt, the retrieval layer, the evaluation set, and the guardrails around the model.

+--------------------------------------------------------------+
|                  THE MODERN LLMOps PIPELINE                  |
+--------------------------------------------------------------+
| 1. Model selection -->  Foundry model catalog                |
| 2. Grounding (RAG) -->  Azure AI Search (hybrid + semantic)  |
| 3. Quality gates   -->  Foundry evaluators in CI             |
| 4. Guardrails      -->  Foundry guardrails (Content Safety)  |
| 5. Observability   -->  App Insights + OpenTelemetry         |
| 6. Cost control    -->  Budgets, quotas, token tracking      |
| 7. Lifecycle       -->  Retirement dates, version pinning    |
+--------------------------------------------------------------+

Quick comparison

Operational area	Traditional MLOps	Modern LLMOps
Primary code unit	Training scripts, feature pipelines	Prompts, system instructions, RAG and agent workflows
Data engine	Feature stores, SQL pipelines	Embeddings, hybrid search indexes
Customization	Training or retraining models	Prompting, RAG context, fine-tuning (including PEFT)
Testing strategy	Deterministic metrics (accuracy, loss)	LLM-as-judge, groundedness, and safety benchmarks
Main cost driver	GPU compute hours	Token volume, prompt size, and cache efficiency
Key latency metric	Batch or inference latency	Time to first token (TTFT) and tokens per second

The architecture: how the Microsoft pieces fit together

Azure AI Studio was renamed Azure AI Foundry and is now Microsoft Foundry. It is the umbrella for building, evaluating, and deploying generative AI apps and agents.

The core players

Microsoft Foundry: Your control plane for model deployments, playground testing, agents, evaluations, guardrails, and monitoring. Resources are organized into projects.
Foundry Models (including Azure OpenAI models): Managed model endpoints with enterprise networking and identity controls. Model names and versions change often, so keep the deployment name in configuration, not in code.
Azure AI Search: The retrieval engine behind your RAG pipeline. It combines keyword (BM25) search, vector similarity, and semantic reranking.
Microsoft Agent Framework and Semantic Kernel: Open-source SDKs you use to write orchestration code. They are build-time tools, not parts of the Foundry runtime. Microsoft recommends Agent Framework (GA since April 3, 2026) as the path forward for new orchestration work, so check its current guidance before choosing between the two.
Foundry Agent Service: The managed runtime and control plane for running, securing, observing, and publishing agents. It complements the SDKs: you design the agent with a framework and operate it in Agent Service.
Guardrails (Azure AI Content Safety): Filters and protections applied to model deployments and agents.
Application Insights: Telemetry and tracing across the app.

Note on Prompt flow: Older Microsoft LLMOps guides lean on Prompt flow. It is being retired on April 20, 2027 and is no longer recommended for new development. If you have existing flows, plan a migration to Agent Framework.

Five gotchas that bite in production

1. Traditional unit tests break down

You can't write assert response == "Hello World". LLM output is non-deterministic, and a prompt tweak can improve 90% of answers while silently breaking edge cases.

The fix: add LLM-as-judge evaluations to your pull request pipeline, scored against a small "golden" dataset of realistic questions. Foundry's built-in evaluators cover areas like:

Groundedness: Did the answer stick to the retrieved context?
Relevance: Did it answer what the user asked?
Coherence: Is the output well-structured and logical?
Safety: Does it avoid policy violations and jailbreak weaknesses?

Be explicit about what your threshold means. As far as I know, the built-in AI-assisted evaluators score on a 1-5 scale, so a gate like "0.85" has to be a pass rate (the share of test cases scoring at or above a per-item threshold), not a raw score.

# .github/workflows/llmops.yml (excerpt)
- name: Run evaluation gate
  run: |
    # Fails the build if fewer than 85% of golden-set answers
    # score at least 3 (on a 1-5 scale) for groundedness
    python scripts/run_eval.py \
      --dataset tests/gold_dataset.json \
      --min-item-score 3 \
      --min-pass-rate 0.85

A minimal sketch of the script (illustrative only; check the azure-ai-evaluation docs for current constructor arguments and result keys):

# scripts/run_eval.py (sketch)
import argparse, json, sys
from azure.ai.evaluation import GroundednessEvaluator

parser = argparse.ArgumentParser()
parser.add_argument("--dataset", required=True)
parser.add_argument("--min-item-score", type=float, default=3)
parser.add_argument("--min-pass-rate", type=float, default=0.85)
args = parser.parse_args()

# model_config points at the judge model deployment
evaluator = GroundednessEvaluator(model_config=load_model_config())

rows = json.load(open(args.dataset))
passed = 0
for row in rows:
    answer = call_my_app(row["query"])          # your application under test
    result = evaluator(query=row["query"], context=row["context"], response=answer)
    if result["groundedness"] >= args.min_item_score:
        passed += 1

pass_rate = passed / len(rows)
print(f"Groundedness pass rate: {pass_rate:.0%}")
sys.exit(0 if pass_rate >= args.min_pass_rate else 1)

Offline gates catch regressions before release. They don't tell you how the system behaves on real traffic, so also sample and score production conversations (see the checklist below).

2. Pure vector search is rarely enough

Many RAG prototypes embed text and query a vector index. In practice, pure vector search tends to struggle with product codes, exact names, and acronyms.

The fix: use hybrid search in Azure AI Search, combining keyword (BM25) and vector queries, with the semantic reranker on top. Measure retrieval quality on your own golden set so you know whether it helps for your data.

3. API keys in app settings are a risk (and managed identity has its own trap)

Hardcoded or loosely managed keys leak, and rotation is painful.

The fix: use Microsoft Entra ID and managed identities so your app authenticates without stored static keys.

// Keyless authentication with Semantic Kernel
var builder = Kernel.CreateBuilder();

builder.AddAzureOpenAIChatCompletion(
    deploymentName: config["Llm:ChatDeployment"],      // keep in config, not code
    endpoint: config["Llm:Endpoint"],                  // https://<your-resource>.openai.azure.com/
    credentials: new DefaultAzureCredential());        // no API key required

var kernel = builder.Build();

Two details the "no keys!" pitch often skips:

Role assignment is required. Keyless auth fails with 401/403 errors until the app's identity has an appropriate role on the resource (for example, Cognitive Services OpenAI User). Grant the least privilege your app needs.
Be deliberate about the credential. DefaultAzureCredential is convenient for local development because it tries several sources. In production, many teams use a specific credential, such as ManagedIdentityCredential, so behavior is predictable.

4. Models get retired on a fixed clock

Model versions don't live forever. At the time of writing, Microsoft's schedule showed o3-mini deprecated with a retirement in August 2026, and a widely used gpt-4o version (2024-11-20) retiring in October 2026 for many deployment types. After retirement, calls to the old deployment fail instead of quietly falling back. Dates vary by version and deployment type, and they are not always extendable, so treat the current schedule as the source of truth.

The fix:

Keep deployment names and model versions in configuration.
Subscribe to Azure Service Health alerts for your resources.
Re-run your evaluation set whenever you change or auto-upgrade a model.
Review the retirement schedule on a regular cadence.

5. Costs creep up quietly

An unbounded prompt, a retry loop, or a chatty agent can burn through tokens fast.

The fix:

Set budgets and alerts in Azure Cost Management for the subscription or resource group.
Apply quotas and rate limits on deployments, and consider an API gateway for per-app or per-user limits.
Cap max_tokens (or the equivalent) and trim retrieved context to what the model needs.
Use prompt caching where supported, and route simple tasks to smaller, cheaper models.
Track token usage per user, per feature, and per model so you can see who and what drives spend.

Production checklist

If you're taking your first LLM app to production on Azure, keep this handy:

[ ] Model strategy: Are small models (such as the Phi family) handling simple routing and summarization, with larger reasoning models reserved for complex tasks?
[ ] Model lifecycle: Are deployment names and versions in config, with retirement dates tracked and alerts on?
[ ] Prompt version control: Are prompt templates stored and versioned in Git alongside your code?
[ ] Evaluation gate: Does a batch evaluation on a golden dataset run before merging, with a clearly defined score and pass-rate threshold?
[ ] Production evaluation: Do you sample real conversations and score them for groundedness and safety over time?
[ ] Prompt-attack protection: Are Prompt Shields (via Foundry guardrails and Azure AI Content Safety) enabled to detect jailbreak and indirect prompt attacks?
[ ] Harmful content and PII handling: Are harm-category filters configured, and is PII detection or redaction in place where your data requires it? (Verify where and how PII detection runs for your setup.)
[ ] Identity: Are you using managed identities with least-privilege role assignments instead of static keys?
[ ] Observability: Are OpenTelemetry and Application Insights tracking latency (including TTFT), errors, and token spend per user?
[ ] Cost controls: Are budgets, alerts, quotas, and token limits configured?

What's next

Building with LLMs feels like magic at first. LLMOps is what turns that magic into software engineering. Set up evaluation gates early, ground your models with solid retrieval, and plan for model and platform change from day one.

Over to you: Are you running automated LLM-as-judge evaluations in CI/CD, or still spot-checking answers in the playground? Share your setup in the comments.

DEV Community