DEV Community: TheProdSDE

MCP for Backend Engineers: When to Use It (and When to Skip It)

TheProdSDE — Tue, 21 Apr 2026 14:12:51 +0000

"If you only have one consumer, MCP is a liability."

Subtitle: Real code, real cost, and an honest answer to the question every AI team gets wrong.

TL;DR

1 app · 1 team · 1 toolset → use plain tool calling
Multiple teams or AI surfaces → use MCP
Existing APIs → wrap them in MCP
Early-stage / MVP → skip MCP

MCP is a scaling decision, not a starting point.

Why This Exists

A client had a simple ask: "Let our AI assistant use our REST API."

Three teams built three integrations. All three were wrong:

- Model called /getCustomer when it needed /getOrders
- Auth header dropped silently on retry
- Three teams hardcoded three different endpoint assumptions

Result: 3 integrations. 3 slightly different bugs. 1 very awkward client call.

That's when MCP became the answer. One server wrapping the API. Any AI host connects to it. The client got exactly what they asked for — and we got a pattern reused three times since.

MCP is not about smarter models — it's about cleaner boundaries.

MCP in One Line

MCP = API Gateway + Plugin System + Contract Layer for LLMs

Without MCP	With MCP
Duplicate integrations	Single contract
Auth bugs everywhere	Centralized auth
Hardcoded tools	Dynamic discovery
Fragile wrappers	Reusable servers
Rebuild per host	Plug into existing system

The Hero Diagram: MCP vs Tool Calling

This is the architecture difference that actually matters.

graph TD
    subgraph TC["❌ Tool Calling — Tightly Coupled"]
        direction TB
        TC_H["AI Host"]
        TC_T1["Tool: get_customer()"]
        TC_T2["Tool: list_orders()"]
        TC_T3["Tool: check_inventory()"]
        TC_H --> TC_T1
        TC_H --> TC_T2
        TC_H --> TC_T3
    end

    subgraph MCP["✅ MCP — Loosely Coupled via Contract"]
        direction TB
        MH1["Host: Web Chat"]
        MH2["Host: Slack Bot"]
        MH3["Host: VS Code"]

        MS1["customer-mcp-server\nOwner: CRM Team"]
        MS2["payments-mcp-server\nOwner: Payments Team"]
        MS3["infra-mcp-server\nOwner: Platform Team"]

        MH1 --> MS1
        MH1 --> MS2
        MH2 --> MS1
        MH2 --> MS3
        MH3 --> MS2
        MH3 --> MS3
    end

The shift: Tool calling is a local decision. MCP is an organizational contract.

What MCP Actually Is

Model Context Protocol (MCP) is an open protocol that standardises how AI applications connect to external tools and data sources. Think of it as USB-C for AI tools — one connector, many servers, any host.

Instead of every IDE, chat UI, or agent framework inventing its own plugin format, MCP defines a common contract using JSON-RPC 2.0 over Streamable HTTP or stdio.

Three roles in the spec:

Host — the application the user sees (IDE, CLI, chat UI, agent framework)
Client — a connector inside the host that speaks the MCP protocol
Server — a service that exposes tools, resources, and prompts to the model

The client sits between host and server. It connects to one server at a time and speaks the protocol; the host can orchestrate multiple clients simultaneously.

graph TD
    subgraph HOST["AI Host (Orchestrates Everything)"]
        direction TB
        H["Orchestrator Logic + LLM Calls"]
        C1["MCP Client 1"]
        C2["MCP Client 2"]
        H --> C1
        H --> C2
    end

    subgraph SERVERS["MCP Servers (Owned per Team)"]
        S1["customer-mcp-server"]
        S2["infra-mcp-server"]
    end

    C1 --> S1
    C2 --> S2

Key Insight: The model never talks to your backend directly. It only talks to tools exposed by MCP servers. That's the boundary.

Case Study: Wrapping a Client REST API

Before: Three separate UIs (web, Slack bot, IDE) called the REST API directly. Each integrator made slightly different assumptions about endpoints and auth — producing three brittle integrations and repeated bugs.

After: One MCP server wrapping the existing API. Same tool list and prompts exposed to every host.

Results in production:

Integration time reduced by ~60% for new hosts
Tool-call auth failures dropped from multiple incidents to zero in the first month (root cause: centralized token handling)
Two new hosts onboarded with zero backend changes

This is the practical payoff: a small upfront cost for long-term reduction in duplicated integration work and clearer ownership.

Should You Use MCP? — Decide Before Reading Further

Most tutorials make you read 80% before answering this. Here it is upfront.

If you pick MCP too early, you're trading ~1–2 extra days of setup for zero immediate benefit. Start with the simplest thing. Reach for MCP when the "three wrappers" problem is already real.

Three questions that determine everything:

How many consumers will call these tools?
- One app, one team → plain tool calling, done
- Multiple apps or teams → MCP starts making sense
Who owns the tools vs who owns the AI host?
- Same team owns both → plain tool calling, tight coupling is fine
- Different teams → MCP gives you the clean boundary
Do these tools already exist as APIs?
- Yes → wrap them as an MCP server, zero changes to existing backend
- No → build them either way, MCP adds structure from day one

Five Real Scenarios

Scenario 1 — Internal chatbot for one team
A team wants an AI that queries their own database and sends Slack alerts. One app, one team, they control everything.
→ Plain tool calling. MCP adds overhead with zero benefit here.

Scenario 2 — Your client's REST API (the exact story above)
Client has an existing API. Wants multiple AI surfaces to consume it — web app today, Slack bot next month, VS Code extension later.
→ MCP. Wrap the API once, every host connects via the same protocol.

Scenario 3 — Enterprise platform, multiple teams
Payments team, infra team, CRM team — each owns their domain. One central AI console orchestrates across all three.
→ MCP per domain. Each team ships and secures their own server.

Scenario 4 — Quick prototype / MVP
You're validating an idea over a weekend. Speed matters more than architecture.
→ Plain tool calling always. Get the idea working first. MCP is a refactor you do when it proves valuable.

Scenario 5 — Agentic workflow with long-running tasks
Agent coordinates across multiple systems, tasks need to be traceable and retryable across sessions.
→ MCP. The clean server/client boundary makes observability and retry logic significantly easier to implement.

Decision Flow

flowchart TD
    Start([Start])
    Start --> Prototype{Prototype / MVP?}
    Prototype -- Yes --> Plain[Plain tool calling]
    Prototype -- No --> Multi{Multiple hosts / teams?}
    Multi -- No --> Plain
    Multi -- Yes --> Ownership{Same team owns tools & host?}
    Ownership -- Yes --> Plain
    Ownership -- No --> APICheck{APIs already exist?}
    APICheck -- Yes --> MCP[Use MCP]
    APICheck -- No --> Consider[Build either way — MCP adds structure]
    classDef recommend fill:#f9f,stroke:#333,stroke-width:1px;
    class MCP recommend;

Key Insight: MCP is triggered by scale of ownership and consumption — not complexity.

Where MCP Is a Bad Idea

MCP is the wrong choice when:

You're at the early-stage or MVP phase
Only one host consumes the tools
One team owns the full stack
Latency is business-critical
Tool shapes are still evolving

If you only have one consumer, MCP is architecture cosplay.

Core MCP Concepts

Server-side primitives

1. Tools
Executable actions — often with side effects. The LLM decides when to call them.
Examples: create_invoice, deploy_service, run_sql_query.

2. Resources
Read-only data and context the model is allowed to see — not actions.
Examples: docs, config files, logs, user profile JSON, query results.

3. Prompts
Reusable workflow templates — Postman collections for LLM behaviour. Server-defined recipes the host can invoke.
Examples: bug_report_prompt, deployment_checklist_prompt.

Client-side capabilities

4. Sampling
The server asks the client: "Run a model completion for me using your model access." Useful when servers don't hold model keys.

5. Roots
The server asks: "What file paths or URIs can I operate on?" Organises where tools can act.

6. Elicitation
The server asks the client to collect more information from the user.
Example: "Ask the user which environment to deploy to: dev/staging/prod."

Request / Response Lifecycle

The most important thing to understand: the Host orchestrates LLM calls — not the MCP Client. The client only handles server communication. This distinction is what most sequence diagrams get wrong.

sequenceDiagram
    autonumber
    participant U as User
    participant H as Host / Orchestrator
    participant L as LLM
    participant C as MCP Client
    participant S as MCP Server

    U ->> H : Query
    H ->> C : discover tools
    C ->> S : GET /mcp/tools
    S -->> C : tool schemas
    C -->> H : return schemas
    H ->> L : prompt + tool schemas
    L -->> H : tool call decision
    H ->> C : execute tool call
    C ->> S : call_tool(name, args)
    S -->> C : tool result
    C -->> H : return result
    H ->> L : result + continue prompt
    L -->> H : final answer
    H -->> U : output

Key Insight: The model decides what to call, but the server controls what exists. That's the contract.

Minimal MCP Server (Python)

Three things to watch in this block:

@mcp.tool() is an executable action with side effects
@mcp.resource() is read-only context — the model sees it, doesn't call it
The /health endpoint is not optional — Container Apps and AKS need it for readiness probes. Skip it and your pods restart on a loop.

# server.py
# requires: fastmcp>=2.0.0, python-dotenv

import os
import uuid
from dotenv import load_dotenv
from fastmcp import FastMCP, Resource

load_dotenv()

mcp = FastMCP(
    name="customer-service",
    instructions="Customer service MCP server.",
)

# ── Tool: get_customer ────────────────────────────────────────────────────────
@mcp.tool()
async def get_customer(customer_id: str) -> dict:
    """Fetch customer profile by ID."""
    # Replace with your real DB call
    return {
        "customer_id": customer_id,
        "name": "Priya Sharma",
        "email": "priya@example.com",
        "tier": "premium",
        "last_order_date": "2026-03-15",
    }

# ── Tool: get_orders ─────────────────────────────────────────────────────────
@mcp.tool()
async def get_orders(customer_id: str, limit: int = 5) -> dict:
    """Fetch recent orders for a customer."""
    return {
        "customer_id": customer_id,
        "orders": [
            {"order_id": f"ORD-{i}", "status": "delivered", "amount": 1200 + i * 50}
            for i in range(1, limit + 1)
        ],
    }

# ── Tool: create_support_ticket ───────────────────────────────────────────────
@mcp.tool()
async def create_support_ticket(
    customer_id: str,
    issue: str,
    priority: str = "medium",
) -> dict:
    """Create a support ticket for a customer issue."""
    return {
        "ticket_id": f"TKT-{uuid.uuid4().hex[:8].upper()}",
        "customer_id": customer_id,
        "issue": issue,
        "priority": priority,
        "status": "open",
    }

# ── Resource: customer playbook ───────────────────────────────────────────────
@mcp.resource("customer-playbook")
async def customer_playbook() -> Resource:
    """Internal guidance for handling customer incidents."""
    content = """
# Customer Handling Playbook

- Always greet the customer by name.
- Premium customers: consider goodwill credit for severe issues.
- Escalate to L3 if incident impacts > 10 customers.
- SLA: 4 hours for premium, 24 hours for standard.
"""
    return Resource(
        name="customer_playbook",
        description="Internal guidance for handling customer incidents.",
        mimeType="text/markdown",
        content=content,
    )

# ── Prompt: email draft ───────────────────────────────────────────────────────
@mcp.prompt("email_draft_prompt")
async def email_draft_prompt():
    """Reusable workflow template for customer apology emails."""
    return {
        "name": "email_draft_prompt",
        "description": "Template for customer apology emails.",
        "messages": [{
            "role": "system",
            "content": (
                "You are a senior customer success manager.\n"
                "Write a short apology email to {{customer_name}} about: {{issue}}.\n"
                "Tone: professional, empathetic. Under 100 words."
            ),
        }],
    }

# ── Health check — required for ACA and AKS readiness probes ─────────────────
@mcp.custom_route("/health", methods=["GET"])
async def health():
    from fastapi.responses import JSONResponse
    return JSONResponse({"status": "ok", "server": "customer-mcp"})

if __name__ == "__main__":
    mcp.run(
        transport="streamable-http",
        host="0.0.0.0",
        port=int(os.getenv("PORT", 8000)),
    )

Start it:

pip install "fastmcp>=2.0.0" python-dotenv
python server.py
# Live at http://localhost:8000/mcp

Inspect without writing a single line of client code:

npx @modelcontextprotocol/inspector http://localhost:8000/mcp

MCP Client + Agent Loop

The key line is client.list_tools() — tools are discovered dynamically from the server, not hardcoded in the client. Every host that connects to this server gets the same tool list automatically, even as you add new tools.

# client.py
# requires: fastmcp>=2.0.0, openai>=1.0.0, python-dotenv

import asyncio
import json
import os
from dotenv import load_dotenv
from fastmcp import Client
from openai import AsyncOpenAI

load_dotenv()

openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
MCP_URL = os.getenv("MCP_SERVER_URL", "http://localhost:8000/mcp")


async def run(query: str) -> str:
    async with Client(MCP_URL) as client:

        # Discover tools from the server — not hardcoded here
        tools = await client.list_tools()
        openai_tools = [
            {
                "type": "function",
                "function": {
                    "name": t.name,
                    "description": t.description,
                    "parameters": t.inputSchema,
                },
            }
            for t in tools
        ]

        print(f"\n[MCP] {len(tools)} tools discovered: {[t.name for t in tools]}")
        messages = [{"role": "user", "content": query}]

        while True:
            resp = await openai_client.chat.completions.create(
                model="gpt-4o",
                messages=messages,
                tools=openai_tools,
                tool_choice="auto",
            )
            msg = resp.choices[0].message
            messages.append(msg)

            # No tool calls = final answer
            if not msg.tool_calls:
                return msg.content

            # Execute each tool call via the MCP server
            for tc in msg.tool_calls:
                args = json.loads(tc.function.arguments)
                print(f"[Tool] {tc.function.name}({args})")

                result = await client.call_tool(tc.function.name, args)
                tool_content = (
                    result.content[0].text if result.content else "{}"
                )
                messages.append({
                    "role": "tool",
                    "tool_call_id": tc.id,
                    "content": tool_content,
                })


if __name__ == "__main__":
    queries = [
        "Look up customer C-123 and tell me their tier.",
        "Get the last 3 orders for customer C-123.",
        "C-123 had a late delivery. Open a ticket and draft an apology email.",
    ]
    for q in queries:
        print(f"\n[Query] {q}")
        print(asyncio.run(run(q)))
        print("─" * 60)

Run it (server must be up):

python client.py

Auth — The Part Most Tutorials Skip

TL;DR:
Internal / pod-to-pod → skip auth entirely, use network policy.
Public-facing → OAuth 2.1 + PKCE, no exceptions. The spec mandates it.
Everything else is a practical middle ground.

For remote MCP servers (Streamable HTTP), the spec prescribes OAuth 2.1 Authorization Code + PKCE. The MCP client acts as the OAuth client; your server acts as the resource server; an Authorization Server (Keycloak, Auth0, Azure AD) issues the tokens.

The flow:

Client hits server without a token → 401 with WWW-Authenticate pointing to /.well-known/oauth-protected-resource
Client discovers the auth server from that metadata
Client runs OAuth 2.1 Authorization Code + PKCE
Client retries MCP requests with Authorization: Bearer <token>
Server validates the token and serves tools/resources

Here's the token validation middleware — the piece that took longest to get right because the aud (audience) claim check is easy to miss. Without it, a token issued for a different service will pass validation silently.

# auth_middleware.py
import os
import httpx
from authlib.jose import jwt, JsonWebKey
from fastapi import Request
from fastapi.responses import JSONResponse

MCP_RESOURCE_ID = os.getenv("MCP_RESOURCE_ID", "https://your-mcp-server/")
AUTH_SERVER_URL  = os.getenv("AUTH_SERVER_URL", "")
_jwks_cache: dict | None = None

SKIP_PATHS = {"/health", "/.well-known/oauth-protected-resource"}


async def fetch_jwks() -> dict:
    global _jwks_cache
    if _jwks_cache:
        return _jwks_cache
    async with httpx.AsyncClient() as http:
        r = await http.get(f"{AUTH_SERVER_URL}/protocol/openid-connect/certs")
        r.raise_for_status()
        _jwks_cache = r.json()
    return _jwks_cache


async def oauth_middleware(request: Request, call_next):
    if request.url.path in SKIP_PATHS:
        return await call_next(request)

    auth = request.headers.get("Authorization", "")
    if not auth.startswith("Bearer "):
        return JSONResponse(
            status_code=401,
            content={"error": "missing_token"},
            headers={"WWW-Authenticate": f'Bearer realm="{MCP_RESOURCE_ID}"'},
        )

    token = auth.removeprefix("Bearer ").strip()

    try:
        jwks   = await fetch_jwks()
        claims = jwt.decode(token, JsonWebKey.import_key_set(jwks))
        claims.validate()

        # ── The check most tutorials miss ───────────────────────────
        aud = claims.get("aud", [])
        if isinstance(aud, str):
            aud = [aud]
        if MCP_RESOURCE_ID not in aud:
            return JSONResponse(status_code=403, content={"error": "wrong_audience"})

        request.state.user = dict(claims)
        return await call_next(request)

    except Exception as e:
        return JSONResponse(status_code=401, content={"error": str(e)})

Wire it into your server:

# Add to server.py after: mcp = FastMCP(...)
from auth_middleware import oauth_middleware
mcp.app.middleware("http")(oauth_middleware)

Real-World Cost (Numbers, Not Theory)

Expect:

+3–100ms MCP protocol overhead per call (varies by gateway)
+280–950ms total round-trip latency including tool execution (p50–p95)
+1.8–4.2 seconds for multi-step agentic flows (3-agent chains, p50–p95)
Extra infra: containers, auth service, monitoring
Non-trivial engineering overhead during setup

If your system is latency-sensitive or single-tool focused, MCP will feel slower — because it is.

You're buying reuse and consistency, not speed.

Observability & Debugging

Make MCP servers and client-host interactions observable from day one:

Trace every request — propagate a correlation ID across host → client → server → backend and include it in logs and responses
Metrics to track — mcp_tool_call_latency, mcp_tool_call_errors, mcp_discovery_time, mcp_auth_failures, per-tool success rates
Structured JSON logs — (timestamp, level, correlation_id, tool, args, duration, error) to make postmortems fast
Distributed tracing — use OpenTelemetry to connect host traces to server traces for full request/response visibility
Troubleshooting checklist — confirm /.well-known/oauth-protected-resource, check token aud claim, verify /health readiness, replay failing tool calls against a local mock server

Testing & QA

Treat the MCP server like a public contract:

Contract tests — verify the tool list and input/output schemas remain compatible across releases. Fail CI on breaking schema changes.
Unit tests — each @mcp.tool() should have unit tests for edge cases
Integration tests — run the server against a mocked backend and a simulated client that performs discovery + tool calls
Smoke tests — lightweight end-to-end checks that run on deploy (e.g., a small request to get_customer and get_orders)

Production Checklist

[ ] Health/readiness: expose /health and use readiness probes (ACA/AKS)
[ ] Replicas: min-replicas >= 1 for interactive workloads to avoid cold starts
[ ] Auth: validate aud claim and enforce least-privilege scopes
[ ] Rate limiting & quotas: protect downstream systems from runaway agents
[ ] Secrets: store credentials in Key Vault / secret manager — avoid plain env vars for DB credentials
[ ] Observability: metrics, logs, and traces wired to your platform
[ ] Cost & latency: budget for an extra 280–950ms network hop per tool call

Deploying on Azure

MCP is not Azure-specific — it runs on any platform that supports HTTP. But if you're already in the Azure ecosystem, here's the honest breakdown.

flowchart TD
    A["Infra context?"]
    A -->|Greenfield| B["Azure Container Apps\n✅ Default choice"]
    A -->|Existing cluster| C["AKS + Workload Identity"]
    A -->|Low usage / event-driven| D["Azure Functions"]
    A -->|Existing App Service| E["App Service /mcp"]

Azure Deployment Decision Table

Situation	Best fit	Why
Greenfield, no existing infra	Azure Container Apps	Least ops overhead, managed ingress
Already running AKS	AKS + Ingress	Reuse cluster auth, colocate workloads
Multiple servers, same team	ACA shared environment	Internal discovery, Dapr sidecars
Infrequent tools / cost-sensitive	Azure Functions	Per-invocation billing, zero idle cost
Existing App Service web app	App Service /mcp	Zero infrastructure change
Enterprise, Istio service mesh	AKS + mTLS	Zero trust alongside OAuth 2.1

Option 1 — Azure Container Apps (start here)

Recommended for most teams starting fresh. Handles autoscaling, ingress, Entra ID integration, and scale-to-zero with no MCP-specific platform knowledge needed.

RG="mcp-rg"
ACR="<your-acr-name>"

# Build and push
az acr build --registry $ACR --image customer-mcp-server:latest .

# Create environment
az containerapp env create \
  --name mcp-env --resource-group $RG --location eastus

# Store credentials in Key Vault — never directly in env vars
az keyvault secret set \
  --vault-name mcp-keyvault \
  --name database-url \
  --value "postgresql://user:pass@host:5432/db"

# Deploy
az containerapp create \
  --name customer-mcp-server \
  --resource-group $RG \
  --environment mcp-env \
  --image $ACR.azurecr.io/customer-mcp-server:latest \
  --target-port 8000 \
  --ingress external \
  --min-replicas 1 \
  --max-replicas 10 \
  --secrets db-url=keyvaultref:<key-vault-secret-uri> \
  --env-vars DATABASE_URL=secretref:db-url

min-replicas 1 is critical for interactive use — scale-to-zero causes cold start that breaks mid-conversation flow. Expect ~1–3s latency even on warm replicas; design your agent loop timeouts accordingly (30s is a safe outer bound).

Option 2 — AKS (if you already run a cluster)

Use Workload Identity over storing credentials in env vars. Reuse cluster auth and network policies.

# mcp-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: customer-mcp-server
  namespace: mcp-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: customer-mcp-server
  template:
    metadata:
      labels:
        app: customer-mcp-server
    spec:
      containers:
        - name: mcp-server
          image: <your-acr>.azurecr.io/customer-mcp-server:latest
          ports:
            - containerPort: 8000
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: mcp-secrets
                  key: database-url
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 15
            periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: customer-mcp-server
  namespace: mcp-system
spec:
  selector:
    app: customer-mcp-server
  ports:
    - port: 8000
      targetPort: 8000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mcp-ingress
  namespace: mcp-system
spec:
  rules:
    - host: mcp.yourdomain.internal
      http:
        paths:
          - path: /mcp
            pathType: Prefix
            backend:
              service:
                name: customer-mcp-server
                port:
                  number: 8000

kubectl create namespace mcp-system
kubectl apply -f mcp-deployment.yaml
kubectl get pods -n mcp-system

Option 3 — Azure Functions (infrequent, event-driven tools)

Best for tools called rarely — scheduled jobs, webhook-triggered actions, batch processors. Per-invocation billing means zero idle cost. Not suitable for interactive agent loops — cold start breaks conversational flow.

Option 4 — App Service (add `/mcp` to an existing service)

Mount the MCP server on /mcp alongside existing routes. Zero infrastructure change. Lowest friction path for adding MCP to a service that already exists.

Common Failure Modes

Token misconfiguration — auth breaks silently on retry
Tool explosion — too many vaguely named tools confuse the model
Latency stacking — chained MCP calls compound response time (budget 280–950ms per hop)
Business logic duplication — logic living in both the server and the calling agent
Weak observability — no tracing across the client–server boundary
No ownership boundaries — multiple teams modifying the same server

Critical rule: One MCP server = one owning team. Violating this recreates the exact problem MCP was meant to solve.

What MCP Does NOT Solve

Bad tool design
Weak or vague prompts
Latency at the model level
Capability gaps in the underlying LLM
Business logic duplication inside your tools

MCP standardises access. It does not improve what you expose.

Migration Path

Don't start with MCP. Migrate to it when the pain appears:

Start with plain tool calling
Identify tools being duplicated across teams or surfaces
Wrap those shared tools in an MCP server
Add auth, deployment, and observability incrementally

Avoid premature standardisation. It's just premature optimisation with a fancier name.

Further Resources

MCP spec: https://modelcontextprotocol.io/specification
Inspector: npx @modelcontextprotocol/inspector — discover and test any MCP server
FastMCP: https://github.com/jlowin/fastmcp — the Python server/client used in this guide

If this saved you from building an MCP server you didn't need — hit clap. It takes one second and tells the algorithm this is worth distributing. If you've already wired MCP into production, drop your setup in the comments — specifically curious what the auth layer looks like on your end.

The Honest Verdict

MCP won't make your architecture simpler. It will make it honest — every tool, resource, and auth scope exactly where it belongs, exposed through a contract any host can consume.

If you're building for one app, one team: skip it. Plain tool calling ships faster and is entirely adequate.

But if you're building something multiple systems or teams will consume — especially if those tools already exist as APIs — MCP ends the "three copies of the same integration, each slightly wrong" problem permanently.

That's the only problem it solves. It solves it very well.

Worth Each minute read if you work on RAG

TheProdSDE — Mon, 06 Apr 2026 09:59:14 +0000

TheProdSDE

Mar 24

Why Most RAG Systems Fail in Production (And How to Design One That Actually Works)

#ai #rag #agents #software

Comments 1

4 min read

Agentic AI Fails in Production for Simple Reasons — What MLDS 2026 Taught Me

TheProdSDE — Tue, 31 Mar 2026 14:56:35 +0000

TL;DR:
Most agentic AI failures in production are not caused by weak models, but by stale data, poor validation, lost context, and lack of governance. MLDS 2026 reinforced that enterprise‑grade agentic AI is a system design problem, requiring validation‑first agents, structural intelligence, strong observability, memory discipline, and cost‑aware orchestration—not just bigger LLMs.

I recently attended MLDS 2026 (Machine Learning Developer Summit) by Analytics India Magazine (AIM) in Bangalore. While many sessions featured advanced models and agentic frameworks, the most valuable insight was unexpected:

Most AI systems don’t fail in production because of bad models — they fail because of bad systems.

Across the summit, speakers repeatedly showed that issues like stale data, missing validation, poor observability, and uncontrolled execution are what derail agentic AI at scale—not lack of intelligence.

A recurring theme across sessions was clear: the hardest problem in AI today is no longer building impressive demos, but running AI systems reliably at enterprise scale. Many real-world failures stem from system design gaps rather than model limitations.

A Key Shift: From Models to Systems

One of the most important takeaways from the summit was that enterprise AI is fundamentally a system design problem, not a model selection problem.

Multiple speakers highlighted common failure modes seen in production:

Stale or outdated data
Poor data granularity
Context loss across multi-step workflows
False confidence and lack of validation
Black-box decisions with no observability

This explains why many AI solutions look powerful in prototypes but break down in real operational environments.

Policy Learning vs. Structural Intelligence

A particularly insightful discussion contrasted two approaches:

Runtime Policy Learning

Examples include Reinforcement Learning (RL), MADDPG, and Graph Neural Networks (GNNs):

Dynamic decision-making
GPU-intensive
Higher cost and latency
Harder to govern and observe

Structural Intelligence at Design Time

In this approach, intelligence is encoded into the system structure itself, often using graph-based designs:

Relationships are resolved at construction time
Minimal runtime inference
Deterministic behavior
Lower cost and faster response

Key insight: Not every intelligent system needs continuous runtime learning. When relationships are stable, embedding intelligence structurally can be more efficient and reliable.

Validation-First Agent Design

Another strong theme was the shift toward validation-first agents, not answer-first agents.

Successful agentic systems:

Ground every important output to source data
Track freshness and provenance
Validate semantics before taking actions
Plan explicitly before executing
Expose confidence where appropriate

Several talks emphasized that observability should evolve from “what happened?” to “was the result actually correct?”.

Agentic Memory: Accuracy, Cost, and Trust

Sessions on agentic memory highlighted how short-term memory, long-term memory, and pruning strategies directly influence:

Accuracy
Latency
Cost
User trust

The key takeaway was that memory should be treated as a first-class architectural concern, with explicit design choices and benchmarks—rather than an ad-hoc cache bolted on later.

Data Platforms and Practical Architecture Choices

The summit also covered modern data platforms that unify OLTP and OLAP workloads, with strong support for time-series data. These architectures reduce complexity and make near–real-time analytics more accessible.

A broader lesson emerged: cost, latency, reliability, and accuracy must be designed together. Choosing larger models without optimizing workflows, routing, and memory leads to unnecessary compute cost and slower systems.

Putting Agents into Production: Real-World Risks

One session focused entirely on lessons learned from deploying agents in production. Four recurring risks were highlighted:

Silent failures – systems appear healthy but produce wrong outputs
Black-box decisions – lack of explainability and traceability
Permission explosion – agents accumulating excessive access
Runaway execution – uncontrolled tool calls and rising costs

These issues reinforce the importance of governance, guardrails, observability, and scoped execution from day one.

AI-Assisted Development Needs Guardrails

Another notable takeaway was the need to pair AI-assisted code generation with strong static analysis and security validation. Integrations with tools like SonarQube demonstrate how AI-written and human-written code can be:

Validated automatically
Secured against vulnerabilities
Fixed via generated pull requests

This closes the gap between productivity gains and production reliability.

Final Reflections

MLDS 2026 reinforced a critical idea:

The future of AI in enterprises depends more on architecture, validation, and governance than on model strength alone.

Agentic AI succeeds when it is:

Grounded in reliable data
Observable and debuggable
Cost-aware and execution-bounded
Designed around real workflows
Rolled out with clear trust and adoption strategies

The biggest mindset shift is moving from “How powerful is the model?” to “How reliable and efficient is the end-to-end intelligent workflow?”

That, more than anything, was the most valuable learning from the summit.

If you’re working on agentic AI in production, I’d love to hear:

Where have agents broken down for you?
What controls or guardrails helped the most?
Are you handling validation and memory explicitly—or implicitly?

Let’s compare notes.

Why Most RAG Systems Fail in Production (And How to Design One That Actually Works)

TheProdSDE — Tue, 24 Mar 2026 12:23:04 +0000

A practical, system design–focused breakdown of why RAG systems degrade after launch—and what actually works in production.

Everyone builds a RAG system.

And almost all of them work — in demos.

Clean query
Relevant chunks
Decent answer

Ship it.

Then production happens.

Users ask vague follow-ups
Retrieval returns partial context
The model answers confidently… and incorrectly

And suddenly:

Your “working” RAG system becomes unreliable.

The Reality: RAG Fails Quietly

RAG doesn’t crash. It degrades.

Slightly wrong answers
Missing context
Hallucinated explanations with citations

Which is worse than a system that fails loudly.

Most teams blame:

embeddings
vector database
chunk size

But in real systems:

RAG failures are usually system design failures—not retrieval failures.

What a Production RAG System Actually Looks Like

Not this:

Query → Vector DB → LLM

But this:

Step 1: Parsing Matters More Than You Think

Most pipelines start like this:

text = pdf.read()
chunks = split(text)
embeddings = embed(chunks)

This is where things already break.

Problem

PDFs lose structure
Tables turn into noise
Headers/footers pollute chunks
Sections lose meaning

Production Approach

Document → Layout-aware parsing → Structured sections → Clean chunks

Key principles:

preserve headings and hierarchy
remove boilerplate
chunk by meaning, not length

If parsing is wrong, retrieval will always be wrong.

Step 2: Dense vs Sparse Retrieval (You Need Both)

Dense Retrieval (Embeddings)

semantic similarity
handles vague queries
fails on exact matches

Sparse Retrieval (BM25 / Keyword)

exact term matching
works for IDs, clauses
ignores meaning

Production Pattern: Hybrid Retrieval

This gives:

semantic understanding
exact precision

Using only vector search is a common production mistake.

Step 3: Reranking (The Accuracy Multiplier)

Top-K retrieval is noisy.

Add a reranker (cross-encoder):

evaluates (query, chunk) pairs
reorders by true relevance

This significantly improves answer quality without changing your database.

Step 4: Context Building (Where Systems Win or Lose)

Even with good retrieval, most failures happen here.

Common Mistakes

stuffing too many chunks
mixing unrelated documents
ignoring token limits

Production Approach

select top-ranked chunks only
preserve document structure
enforce token budget
maintain ordering

Better context > more context

Vector DB vs Graph DB — When to Use What

Use Vector Database When

unstructured data
semantic search
document retrieval

Use Graph Database When

relationships matter
multi-hop reasoning
structured entities

Hybrid (Real Systems)

Use graph when relationships matter.
Use vector when meaning matters.
Use both when systems get complex.

RAG Is Not Single-Turn — Managing Context Over Time

Most systems fail here.

RAG is not just:

retrieve → answer

It’s:

retrieve → answer → follow-up → correction → refinement

The Problem: Context Drift

If you blindly append chat history:

token usage explodes
wrong answers get reinforced
relevance drops

Production Strategy: Context Is a Filter

Not a dump.

Context Layers

Store full history
Select only relevant turns
Exclude invalid or corrected responses
Combine with retrieved context

When to Summarize vs Include Raw History

Include Raw

short conversations
active refinement
recent corrections

Summarize

long conversations (>5–7 turns)
approaching token limits

Critical Rule

Summarize facts—not hallucinations.

If a previous answer was wrong:

exclude it
prioritize user correction

Handling User Corrections (Critical for Trust)

Users will fix your system.

If you ignore that, the system feels broken.

Strategy

mark incorrect responses
exclude them from future context
boost corrected information

Example:

{
  "turn_id": 8,
  "valid": false,
  "corrected": true
}

Agentic RAG (When Retrieval Needs Reasoning)

Basic RAG is static.

Agentic RAG adds:

planning
iteration
tool usage

Architecture

Use It When

multi-step queries
missing context
dynamic retrieval

Avoid It When

simple Q&A
strict latency requirements

Otherwise you're adding complexity without ROI.

Confidence Scores and Citations (Trust Layer)

Without trust signals, users don’t trust answers.

Citations

Always return:

source document
section or chunk reference

Confidence Score (Simple Heuristic)

Combine:

retrieval score
reranker score
validation signal

Example:

confidence =
  0.4 * retrieval +
  0.4 * reranker +
  0.2 * validation

Optional Validation Step

Ask the model:

“Is this answer fully supported by the context?”

Lower confidence if not.

Guardrail: Don’t Trust the Model Alone

Even with RAG:

hallucinations still happen
citations can be fabricated

Enforce:

answers must reference retrieved chunks
no context → no answer

Final Architecture (Multi-Turn RAG System)

Production Checklist

If your system doesn’t have these, it will fail:

structured parsing
hybrid retrieval
reranking
controlled context building
memory filtering
correction handling
confidence + citations
observability

The Real Rule

RAG is not a retrieval problem. It’s a system design problem.

What Actually Works

The best RAG systems are:

simple
structured
observable
measurable

Not over-engineered.

Final Thought

If your system only works when:

the query is perfect
the data is clean
the demo is controlled

Then it doesn’t work.

What’s Next

Once RAG works, the next bottleneck is:

Cost.

Why LLM systems become expensive in production—and how to control it without killing performance.

Most AI Agent Frameworks Are Overkill — Here's How to Choose the Right One in 30 Seconds

TheProdSDE — Wed, 18 Mar 2026 12:58:03 +0000

A senior engineer's field-tested breakdown of LangGraph, AutoGen, CrewAI, Microsoft Agent Framework, and Haystack — from reviewing real production systems across teams.

Everyone is building AI agents right now.

LangGraph. AutoGen. CrewAI. Semantic Kernel. Microsoft Agent Framework.

But most production AI systems don't actually need an agent framework.

Across multiple teams and production codebases I've reviewed, the same two failure modes appear constantly — over-engineering and under-engineering. In one case, replacing a complex agent framework with ~200 lines of plain tool-calling code made the system 3× faster, easier to debug, and easier to maintain. In another, the absence of a framework caused a codebase to collapse under its own complexity.

Both failures had the same root cause:

The problem isn't choosing the wrong framework. It's choosing a framework before understanding the workflow.

This guide is the decision framework I now use before touching any agent tooling.

TL;DR — Pick Your Approach in 30 Seconds

Workflow Shape	Recommended Approach
Single request → tools → response	Plain tool calling (no framework)
Self-correcting loops, retries, re-evaluation	LangGraph
Parallel specialist agents	AutoGen 0.7.5
Enterprise persistent agents (Azure)	Microsoft Agent Framework RC
Sequential role-based task delegation	CrewAI 1.10.1
Document analysis and extraction pipelines	Haystack 2.x

Workflow Shapes at a Glance

Two Real Failure Modes I've Seen Across Teams

The over-engineering case.

During a cross-team architecture review, I found a financial data analysis pipeline — pull market metrics, cross-reference filings, produce a risk score — built with three microservices, a LangGraph orchestrator with twelve nodes, Redis for inter-agent memory, and a separate evaluation loop. Six weeks of engineering. It worked.

What it actually needed: 180 lines of FastAPI with direct OpenAI tool calling. Same output. 3× faster inference. Any engineer on the team could debug it in minutes.

After the review, the team simplified it together. The framework had been adopted before anyone mapped the workflow shape — a straight line — and LangGraph's graph model added cost with no return.

The under-engineering case.

In a separate review, I found the opposite problem. An infrastructure incident response system — Azure alerts, metric retrieval, remediation decisions, rollback logic, retry conditions, escalation thresholds — built with hand-rolled state machines, custom retry logic, and a bespoke tool orchestration layer. Three weeks in, the codebase was a maze. Every new remediation path required rewriting core routing logic.

A proper agent framework would have provided state management, conditional branching, retry handling, and checkpointing for free. Instead the team was reinventing those primitives by hand — the most expensive kind of technical debt.

The pattern is the same in both directions: the architecture was chosen before the workflow was understood.

The Core Question: Does Your Workflow Resist Simplicity?

Before touching any framework, draw the workflow on paper. Then answer these:

Does step N's output determine whether to redo step N-1? → You have a loop
Do multiple specialized agents need to run simultaneously? → You have parallelism
Does the workflow run for minutes or hours, surviving restarts? → You need persistent state
Does the agent need to decide its next action from intermediate results? → Dynamic planning
Do independent agents need to hand off context to each other? → Multi-agent delegation

None of these? Use plain tool calling. Here's what that looks like at production quality.

The Baseline: Plain Tool Calling (No Framework Needed)

Use case: Real-time ESG (Environmental, Social, Governance) risk scoring. Given a ticker, pull sustainability metrics, cross-reference regulatory filings, produce a risk-adjusted score, and persist an audit trail — one clean, observable service.

# requirements: fastapi, openai, psycopg2-binary, httpx, pydantic
from fastapi import FastAPI
from openai import OpenAI
from pydantic import BaseModel
import json, httpx, psycopg2, os

app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

class ESGRequest(BaseModel):
    ticker: str
    portfolio_id: str

def get_esg_metrics(ticker: str) -> dict:
    resp = httpx.get(
        f"https://api.sustainalytics.com/v1/esg/{ticker}",
        headers={"Authorization": f"Bearer {os.environ['ESG_API_KEY']}"},
        timeout=10
    )
    return resp.json()

def get_regulatory_flags(ticker: str) -> dict:
    with psycopg2.connect(os.environ["DATABASE_URL"]) as conn:
        with conn.cursor() as cur:
            cur.execute("""
                SELECT flag_type, severity, filing_date, description
                FROM regulatory_flags WHERE ticker = %s
                AND filing_date > NOW() - INTERVAL '2 years'
                ORDER BY severity DESC LIMIT 10
            """, (ticker,))
            rows = cur.fetchall()
    return {"flags": [{"type": r[0], "severity": r[1], "date": str(r[2]), "detail": r[3]} for r in rows]}

TOOLS = [
    {"type": "function", "function": {
        "name": "get_esg_metrics",
        "description": "Fetch ESG sustainability scores for a stock ticker",
        "parameters": {"type": "object", "properties": {"ticker": {"type": "string"}}, "required": ["ticker"]}
    }},
    {"type": "function", "function": {
        "name": "get_regulatory_flags",
        "description": "Retrieve regulatory violations and compliance flags",
        "parameters": {"type": "object", "properties": {"ticker": {"type": "string"}}, "required": ["ticker"]}
    }}
]

TOOL_MAP = {
    "get_esg_metrics": get_esg_metrics,
    "get_regulatory_flags": get_regulatory_flags
}

@app.post("/assess-esg")
async def assess_esg(req: ESGRequest):
    messages = [{"role": "user", "content": (
        f"Run a full ESG risk assessment for {req.ticker} in portfolio {req.portfolio_id}. "
        f"Produce a risk-adjusted score with recommendation: HOLD, REDUCE, or DIVEST."
    )}]
    while True:
        response = client.chat.completions.create(
            model="gpt-4o", messages=messages, tools=TOOLS, tool_choice="auto"
        )
        msg = response.choices[0].message
        if not msg.tool_calls:
            return {"ticker": req.ticker, "assessment": msg.content}
        messages.append(msg)
        for tc in msg.tool_calls:
            result = TOOL_MAP[tc.function.name](**json.loads(tc.function.arguments))
            messages.append({"role": "tool", "tool_call_id": tc.id, "content": json.dumps(result)})

Multi-step tool calling, audit trail, fully debuggable by any engineer. If this solves the problem — stop here. No framework needed.

1. LangGraph — Stateful Cyclic Workflows

Version: 1.1.2 | pip install langgraph langgraph-checkpoint-redis

Use when: Your workflow has genuine loops — the result of one step conditionally re-runs a previous step. Redis checkpointing lets state survive service restarts mid-workflow.

Avoid when: The workflow is strictly sequential with no conditional branching. The graph model adds measurable overhead for zero architectural return.

Use case: Automated cloud cost optimization — scan Azure VMs for underutilization, simulate right-sizing savings, apply low-risk changes, re-scan. The loop continues until no optimization exceeds the savings threshold.

# requirements: langgraph, langgraph-checkpoint-redis, openai
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.redis import RedisSaver
from typing import TypedDict, Annotated
import operator, json, os
from openai import OpenAI

oai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

class CostState(TypedDict):
    subscription_id: str
    resources: list[dict]
    candidates: Annotated[list, operator.add]
    applied: Annotated[list, operator.add]
    total_savings: float
    iteration: int

def scan_resources(state: CostState) -> dict:
    resources = azure_monitor_client.list_resources(state["subscription_id"])
    underutilized = [r for r in resources if r["avg_cpu_7d"] < 15 and r["avg_memory_7d"] < 30]
    return {"resources": underutilized, "iteration": state["iteration"] + 1}

def analyze_savings(state: CostState) -> dict:
    resp = oai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"""
            Analyze these underutilized Azure resources and recommend right-sizing.
            For each: resource_id, current_sku, recommended_sku, monthly_savings_usd, risk_level (LOW/MEDIUM/HIGH).
            Only include LOW or MEDIUM risk recommendations.
            Resources: {json.dumps(state["resources"])}
            Respond: {{"recommendations": [...]}}
        """}],
        response_format={"type": "json_object"}
    )
    candidates = json.loads(resp.choices[0].message.content)["recommendations"]
    return {"candidates": candidates, "total_savings": sum(c["monthly_savings_usd"] for c in candidates)}

def apply_changes(state: CostState) -> dict:
    applied = []
    for c in state["candidates"]:
        if c["risk_level"] == "LOW":
            azure_compute.resize_vm(c["resource_id"], c["recommended_sku"])
            applied.append(c)
    return {"applied": applied}

def check_threshold(state: CostState) -> str:
    return "loop" if state["total_savings"] > 500 and state["iteration"] < 5 else "done"

graph = StateGraph(CostState)
graph.add_node("scan", scan_resources)
graph.add_node("analyze", analyze_savings)
graph.add_node("apply", apply_changes)
graph.set_entry_point("scan")
graph.add_edge("scan", "analyze")
graph.add_edge("analyze", "apply")
graph.add_conditional_edges("apply", check_threshold, {"loop": "scan", "done": END})

optimizer = graph.compile(checkpointer=RedisSaver.from_conn_string(os.environ["REDIS_URL"]))
result = optimizer.invoke({
    "subscription_id": os.environ["AZURE_SUBSCRIPTION_ID"],
    "candidates": [], "applied": [], "total_savings": 0.0, "iteration": 0
})

Why this justifies LangGraph: The self-correcting re-scan is genuinely hard to express cleanly in plain tool calling without writing a custom state machine — which is exactly the under-engineering trap described earlier.

2. AutoGen 0.7.5 — Parallel Multi-Agent Collaboration

Version: 0.7.5 | pip install "autogen-agentchat>=0.7.5" "autogen-ext[openai,redis]"

Key additions in 0.7.5: linear memory via RedisMemory, fixed GraphFlow cycle detection, Anthropic thinking mode support, reasoning_effort parameter for GPT-5 models, improved Azure AI client streaming.

Use when: Multiple specialized agents run independent analysis simultaneously. Async, event-driven — agents can be deployed on separate containers with zero blocking between them.

Avoid when: The task is sequential and single-agent. Multi-agent coordination overhead only pays off when genuine parallelism exists.

Use case: Automated M&A due diligence — Legal, Financial, and Tech Audit agents work in parallel, then a Synthesis agent consolidates findings into an investment decision. Sequential execution would add hours of unnecessary latency to a time-sensitive deal process.

# requirements: autogen-agentchat>=0.7.5, autogen-ext[openai,redis]
import asyncio, os
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import SelectorGroupChat
from autogen_agentchat.conditions import TextMentionTermination
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.memory.redis import RedisMemory  

model_client = OpenAIChatCompletionClient(model="gpt-4o", api_key=os.environ["OPENAI_API_KEY"])

legal_agent = AssistantAgent(
    name="LegalAgent",
    model_client=model_client,
    memory=[RedisMemory(redis_url=os.environ["REDIS_URL"], session_id="ma-legal")],
    system_message="""M&A legal counsel. Identify: IP gaps, change-of-control clauses,
    litigation exposure, employment liabilities.
    Output JSON: {risk_items, severity: BLOCKER|HIGH|MEDIUM|LOW, deal_impact}""",
    tools=[fetch_contract_repository, search_litigation_database]
)

financial_agent = AssistantAgent(
    name="FinancialAgent",
    model_client=model_client,
    memory=[RedisMemory(redis_url=os.environ["REDIS_URL"], session_id="ma-financial")],
    system_message="""M&A financial analyst. Identify: revenue quality, off-balance-sheet
    liabilities, working capital needs post-acquisition, EBITDA normalization.
    Output JSON: {financial_flags, normalized_ebitda, recommended_valuation_range}""",
    tools=[fetch_financial_statements, compute_dcf_model]
)

tech_audit_agent = AssistantAgent(
    name="TechAuditAgent",
    model_client=model_client,
    system_message="""Technical due diligence expert. Assess: tech debt score (1–10),
    security vulnerabilities, scalability ceiling, bus-factor risk.
    Output JSON: {tech_risks, estimated_remediation_cost_usd, integration_complexity}""",
    tools=[clone_and_analyze_repo, run_dependency_scan]
)

synthesis_agent = AssistantAgent(
    name="SynthesisAgent",
    model_client=model_client,
    system_message="""M&A deal lead. Wait for all domain agents to complete.
    Synthesize: PROCEED / PROCEED_WITH_CONDITIONS / ABORT.
    Include: top 5 risks, price adjustment recommendation, 90-day priorities.
    End your message with: ANALYSIS_COMPLETE""",
    tools=[generate_pdf_memo, notify_deal_team_slack]
)

async def run_ma_due_diligence(target: str):
    team = SelectorGroupChat(
        participants=[legal_agent, financial_agent, tech_audit_agent, synthesis_agent],
        model_client=model_client,
        termination_condition=TextMentionTermination("ANALYSIS_COMPLETE"),
        selector_prompt="""Run LegalAgent, FinancialAgent, TechAuditAgent first (order flexible,
        can run in parallel). Only select SynthesisAgent after all three have reported."""
    )
    async for msg in team.run_stream(task=f"Full M&A due diligence for: {target}"):
        print(f"[{msg.source}] {str(msg.content)[:120]}...")

asyncio.run(run_ma_due_diligence("TargetCorp Inc."))

3. Microsoft Agent Framework RC — Enterprise Production Agents

Version: Release Candidate, Feb 19, 2026 | pip install agent-framework --pre

The most significant shift in the Microsoft AI ecosystem right now. This framework unifies Semantic Kernel and AutoGen into a single SDK — both are entering maintenance mode as of this writing, and all new Microsoft investment flows into Agent Framework first.

RC signals a frozen, stable API surface with GA targeting Q1 2026. Core capabilities: persistent threads (Cosmos DB), Service Bus integration, MCP + A2A protocol support, multi-agent orchestration with handoff and group chat patterns, streaming checkpointing for long-running agents, full .NET and Python support.

Use when: Azure-first enterprise teams, production 24/7 background agents, regulated systems requiring full audit trails, or any team currently building on Semantic Kernel or AutoGen.

Avoid when: Greenfield Python-only stacks with no Azure dependency, or where GA stability is a hard requirement before adoption.

Use case: Autonomous DevOps deployment agent — monitors CI/CD completion events, validates health via Application Insights, promotes through dev → staging → prod automatically, pages on-call only when a gate fails. Runs continuously as a persistent, resumable agent.

# requirements: agent-framework --pre, azure-identity, azure-monitor-query, kubernetes
import os
from agent_framework import AgentClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
agent_client = AgentClient(
    endpoint=os.environ["AZURE_AI_FOUNDRY_ENDPOINT"],
    credential=credential
)

def check_app_insights(resource_id: str, environment: str, window_minutes: int = 5) -> dict:
    from azure.monitor.query import MetricsQueryClient
    from datetime import timedelta
    monitor = MetricsQueryClient(credential)
    result = monitor.query_resource(
        resource_id,
        metrics=["requests/failed", "requests/duration"],
        timespan=timedelta(minutes=window_minutes)
    )
    return {
        "environment": environment,
        "error_rate_percent": _calculate_error_rate(result),
        "p99_latency_ms": _calculate_p99(result),
        "gate_passed": _calculate_error_rate(result) < 1.0 and _calculate_p99(result) < 500
    }

def promote_deployment(service: str, image_tag: str, environment: str) -> dict:
    from kubernetes import client as k8s, config
    config.load_incluster_config()
    k8s.AppsV1Api().patch_namespaced_deployment(
        name=service, namespace=environment,
        body={"spec": {"template": {"spec": {"containers": [{
            "name": service,
            "image": f"{os.environ['ACR_REGISTRY']}/{service}:{image_tag}"
        }]}}}}
    )
    return {"status": "promoted", "service": service, "tag": image_tag, "environment": environment}

# Persistent thread — Cosmos DB preserves state across restarts
agent = agent_client.create_agent(
    model="gpt-4o",
    name="DeploymentOrchestrator",
    instructions="""Autonomous DevOps deployment agent.
    1. Validate health: error rate < 1%, p99 < 500ms, zero pod restarts
    2. Run smoke tests on the deployed environment
    3. All gates pass → promote to next environment
    4. Any gate fails → halt, capture full diagnostics, page on-call
    5. After prod deploy: monitor 15 minutes, auto-rollback if error rate > 2%
    Log every action: timestamp, metric values, decision, outcome.""",
    tools=[check_app_insights, promote_deployment, run_smoke_tests,
           rollback_deployment, page_oncall, log_deployment_event]
)
thread = agent_client.create_thread()

async def handle_pipeline_completion(event: dict):
    """Azure Event Grid webhook — fires on every pipeline completion"""
    agent_client.create_message(
        thread_id=thread.id, role="user",
        content=(
            f"Deployment complete — Service: {event['service_name']}, "
            f"Tag: {event['image_tag']}, Environment: {event['environment']}. "
            f"Begin validation and promotion workflow."
        )
    )
    return agent_client.create_and_process_run(thread_id=thread.id, agent_id=agent.id)

4. CrewAI 1.10.1 — Role-Based Sequential Pipelines

Version: 1.10.1 (March 3, 2026) | pip install crewai==1.10.1

⚠️ Note: v1.10.0 was yanked from PyPI due to an AMP runtime issue. Always pin to 1.10.1 directly. This release lazy loading in Memory module and resolves a concurrent multi-process LockException in production flows.

Use when: Clearly defined specialist roles, sequential task handoff, and fastest path from idea to working prototype. CrewAI is the most opinionated framework and the fastest to build with.

Avoid when: Fine-grained conditional edge control is needed, or throughput-sensitive production paths where abstraction overhead is measurable.

Use case: Automated system architecture pipeline — an Architect designs from requirements, a Reviewer stress-tests for production failures, a Documentation Lead produces the Architecture Decision Record for the engineering wiki.

# requirements: crewai==1.10.1
from crewai import Agent, Task, Crew, Process

architect = Agent(
    role="Principal Solutions Architect",
    goal="Design a scalable, cloud-native system architecture from requirements",
    backstory="10+ years on Azure and AWS. Expert in event-driven architecture, CQRS, and zero-trust.",
    verbose=True
)
reviewer = Agent(
    role="Staff Engineer — Technical Reviewer",
    goal="Identify scalability bottlenecks, security vulnerabilities, and operational risks",
    backstory="Former SRE. Has seen every production failure mode. Only approves designs that hold at 10x load.",
    verbose=True
)
doc_lead = Agent(
    role="Technical Documentation Lead",
    goal="Produce a complete Architecture Decision Record capturing every design trade-off",
    backstory="Undocumented architecture is a liability. Writes for the engineer joining 2 years later.",
    verbose=True
)

design_task = Task(
    description="""Design a complete architecture for: {requirements}
    Output JSON: components (name + responsibility), data_flows (source → destination),
    tech_choices (with justification), scaling_strategy per component, security_controls.""",
    expected_output="JSON architecture specification",
    agent=architect
)
review_task = Task(
    description="""Stress-test the architecture:
    1. 10x traffic spike — what breaks first?
    2. Single region failure — what is the blast radius?
    3. Compromised service account — what can an attacker reach?
    Return: {approved: YES/NO, blocking_issues: [], recommended_changes: []}""",
    expected_output="Review JSON with approval and findings",
    agent=reviewer,
    context=[design_task]
)
adr_task = Task(
    description="Write a complete ADR: Context, Decision, Consequences, Alternatives Considered, Risk Register. Format: Markdown.",
    expected_output="Complete ADR in Markdown",
    agent=doc_lead,
    context=[design_task, review_task]
)

crew = Crew(
    agents=[architect, reviewer, doc_lead],
    tasks=[design_task, review_task, adr_task],
    process=Process.sequential,
    verbose=True
)
result = crew.kickoff(inputs={
    "requirements": "Real-time fraud detection — 50K TPS, sub-100ms decisions, 99.99% uptime, multi-region active-active"
})
print(result.raw)

5. Haystack 2.x — Document Intelligence Pipelines

Version: 2.x | pip install haystack-ai

Use when: Document processing, extraction, or compliance analysis at scale is the core product — not general agentic behavior. Purpose-built for this and measurably outperforms general frameworks on document-centric workloads.

Avoid when: The problem is general agentic orchestration, multi-agent coordination, or any real-time interactive system.

Use case: Automated SOC 2 evidence collection — ingests all internal policy documents, maps clauses against Trust Service Criteria, produces a compliance gap report showing which controls are missing or non-conformant.

The pipeline maps extracted content against all six SOC 2 Trust Service Criteria (CC6, CC7, CC8, CC9, A1, C1), flags NOT_COVERED gaps with descriptions, and returns a structured compliance report with overall coverage percentage — no custom parsing layer required.

The Practical Decision Rule

If your AI system needs:

  Retries or self-correction    → LangGraph
  Multiple parallel specialists → AutoGen
  Enterprise Azure persistence  → Microsoft Agent Framework
  Sequential role-based tasks   → CrewAI
  Document extraction at scale  → Haystack
  None of the above             → No framework. Plain tool calling.

Framework Reference

Framework	Version	Primary Strength	Use When	Avoid When
Plain tool calling	—	Speed, simplicity, debuggability	Straight-line workflow	Never — if no loops exist
LangGraph	1.1.2	Cyclic graphs, checkpointing	Self-correcting loops, retries	No conditional branching
AutoGen	0.7.5	Parallel async agents, RedisMemory	Multiple specialists in parallel	Sequential single-agent tasks
Microsoft Agent Framework	RC Feb 2026	Enterprise persistence, SK + AutoGen unified	Azure production 24/7 agents	Python-only, no Azure stack
CrewAI	1.10.1	Role-based prototyping, fast iteration	Sequential delegation, fast prototyping	Fine-grained production control
Haystack	2.x	Document extraction pipelines	Document processing as core product	General agentic tasks

Production Lessons From Real Systems

After reviewing AI systems across multiple teams, a few patterns appear consistently:

1. Most workflows are simpler than they look.
The instinct when building with LLMs is to reach for orchestration layers. That instinct is usually wrong. Start with the simplest thing that could work, measure it, then add complexity only where the problem resists simplicity.

2. Agent frameworks pay off only when the workflow has the right shape.
Loops, parallel specialists, or long-running persistent state. If none of these exist, the framework is cost with no architectural return.

3. Debuggability matters more than clever architecture.
A system the team can debug at 2AM is worth more than an elegant multi-agent pipeline nobody fully understands. Production incidents don't wait for framework comprehension.

4. Hand-rolling framework primitives is the most expensive mistake.
Custom state machines, retry logic, and checkpointing written to avoid a framework dependency consistently cost more engineering time than learning the framework properly. Both failure modes described earlier confirm this.

5. Decide the architecture before writing the first line.
Draw the workflow. Identify the shape. Pick the tool. That decision should take 30 minutes, not six weeks of refactoring.

The Real Rule

If you can draw your workflow as a straight line — plain tool calling is your production architecture.
If that line needs to loop, branch, or coordinate parallel specialists — match the tool to the shape.

The best production AI systems are architecturally boring. One clean service, typed tool schemas, structured output, observable logs. No framework overhead unless the problem demands it.

Add complexity only when the problem resists simplicity. That's the whole framework.

What's Next in This Series

This post focused on orchestration — how to structure and run AI workflows in production.

But once orchestration is solved, most teams run into a different problem:

Their RAG system doesn’t actually work.

Not in demos — those look fine.
In production — it breaks.

Wrong answers. Missing context. Hallucinations with high confidence.

And in most cases, the root cause is not the vector database or the embedding model.

It’s the architecture around it.

The next article breaks down:

Why most RAG systems fail after launch
The common design mistakes teams repeat
And the production architecture that actually fixes it

If you’re building anything on top of retrieval, this is where things either scale — or quietly fail.

Are you running an agent framework in production — or did you strip one out and go back to basics?

What made that decision clear? Drop your stack and the reason in the comments. The real production stories are always more useful than the official docs.

Tags: #ai #llm #machinelearning #softwareengineering #devops

Version data sourced from official release channels: AutoGen 0.7.5 , Microsoft Agent Framework RC, CrewAI 1.10.1, LangGraph, Haystack

[Boost]

TheProdSDE — Wed, 18 Mar 2026 09:37:06 +0000

Stop Guessing Your LLM Replacement

TheProdSDE ・ Mar 14

#ai #llm #cloud #systemdesign

Stop Guessing Your LLM Replacement

TheProdSDE — Sat, 14 Mar 2026 11:32:29 +0000

A Practical Guide to Migrating GPT Apps Across Azure, AWS, and GCP

TL;DR — Most LLM migrations are not caused by model performance. They happen because of data residency laws, enterprise deployment requirements, or cloud standardisation decisions. This guide helps you narrow the search space to the right replacement candidates — not replace real testing.

Why Companies Actually Migrate LLMs

Engineers rarely wake up and decide to migrate their AI stack. Most migrations are triggered by business constraints, not technical ones.

Common Migration Scenarios

1️⃣ Expanding into Regions With Data Residency Laws

Your product currently runs on Azure OpenAI (gpt-4o-mini), but a new region requires:

EU sovereign cloud
Local data processing
Provider-specific compliance certifications

You may need to move to AWS Bedrock, Google Vertex AI, or Azure AI Foundry with open models — even though your application logic stays identical.

2️⃣ Enterprise Customers Want AI Inside Their Environment

This is extremely common in B2B SaaS. Enterprise customers often require:

Private cloud deployment
VPC-only access
On-prem inference
Sovereign cloud environments

Your API-based model suddenly needs to become Llama, Qwen, DeepSeek, or Mistral — running inside their infrastructure.

3️⃣ Corporate Cloud Standardisation

A classic scenario:

AI team → Azure
Platform team → AWS

Leadership decides: "All workloads must run on AWS."

Now your team must translate gpt-4o-mini into an AWS Bedrock equivalent — and the model catalog doesn't make that obvious.

The Problem: Model Names Don't Translate

Each vendor uses completely different naming schemes. There is no official cross-provider model map.

Vendor	Entry Model	Mid Model	Reasoning Model
OpenAI	`gpt-4o-mini`	`gpt-4o`	o-series
Anthropic	Haiku	Sonnet	Opus
Google	Flash	Pro	Pro reasoning
Meta	Scout	Maverick	Large variants
Qwen	Small (7B–14B)	72B–110B	235B Thinking
DeepSeek	V3 (non-thinking)	V3 standard	R1 (reasoning)

Common mistakes teams make

❌ Picking the cheapest model in the new catalog
❌ Picking the newest model by release date
❌ Picking the highest benchmark model regardless of tier

All three approaches can silently break production behaviour.

The Tier Model That Actually Works

Instead of comparing names, compare capability tiers. Every major provider follows the same five-tier structure.

Tier	Typical Use Cases	Latency	Relative Cost
Mini / Flash / Small	Chatbots, RAG, classification	Fastest	Lowest
Standard / Mid	Assistants, summarisation, coding	Medium	Moderate
Reasoning / Pro	Agents, planning, complex Q&A	Slower	Higher
Frontier / Flagship	Research workloads, safety-critical	Slowest	Highest

Once you know your current model's tier, finding candidates becomes systematic — not guesswork.

Tier 1 — Replacing `gpt-4o-mini`

Typical workloads: chat assistants, RAG pipelines, tool calling, lightweight coding

Candidates by cloud

Cloud	Replacement Candidates
Azure AI Foundry	Llama 4 Scout, Qwen3-8B/14B, DeepSeek V3, Claude Haiku
AWS Bedrock	Claude Haiku, Mistral Small
GCP Vertex AI	Gemini Flash-Lite, Gemini Flash

Behaviour differences at this tier

Model	Strengths	Watch out for
Claude Haiku	Reliable, low hallucination rate	~7× more expensive than `gpt-4o-mini`
Gemini Flash	Extremely fast, 1M token context	GCP-only; not available on Azure
Llama 4 Scout	Open-weight, 10M token context, Azure-hosted	Not a pure reasoning-tuned model
DeepSeek V3	Unusually strong reasoning (MMLU-Pro ~75.9, GPQA ~59.1) for this price tier	Direct API or Azure Foundry; no native AWS/GCP
Qwen3-8B/14B	Strong multilingual + math, Apache 2.0	Smaller context than Gemini/Llama

Tier 2 — Replacing `gpt-4o`

Typical workloads: document summarisation, coding assistance, enterprise chat assistants

Candidates by cloud

Cloud	Replacement Candidates
Azure AI Foundry	Claude Sonnet, Llama 4 Maverick, DeepSeek V3
AWS Bedrock	Claude Sonnet, Mistral Medium
GCP Vertex AI	Gemini Flash, Gemini 2.5 Pro

Benchmark reference (reasoning quality)

Model	MMLU	MMLU-Pro	GPQA-Diamond	Notes
Claude Sonnet 4.x	~88+	Strong	Competitive	Best SWE-bench coding score at this tier
Llama 4 Maverick	~85+	~80.5	~69.8	Beats GPT-4o on Meta's coding benchmarks
DeepSeek V3	88.5	75.9–81.2	59.1–68.4	Frontier-class at mid-tier pricing
Gemini Flash (GCP)	Strong	Competitive	~78% SWE-bench	GCP-only; fastest in this tier

DeepSeek V3 on Azure often outperforms gpt-4o on raw reasoning benchmarks at significantly lower cost. Treat it as a tier upgrade, not just a replacement.

Tier 3 — Replacing Reasoning Models (`gpt-4.1` / `gpt-5` / o-series)

Typical workloads: agent systems, research workflows, complex multi-step reasoning

Candidates by cloud

Cloud	Replacement Candidates
Azure AI Foundry	Claude Opus, DeepSeek R1, Qwen3-235B Thinking (via Foundry)
AWS Bedrock	Claude Opus
GCP Vertex AI	Gemini 2.5 Pro, Gemini 3.1 Pro

Reasoning benchmark reference (HLE + advanced)

Model	HLE Score	MMLU-Pro	GPQA-Diamond	AIME-25
Claude Opus 4.x	Top-tier (Anthropic reports #1 on HLE leaderboard)	~90+	Strong	Strong
Qwen3-235B-A22B Thinking	~18% (one of few published open-weight HLE scores)	~84.4%	~81%	~92%
DeepSeek R1	Not widely published	~81.2	~68.4	Strong
Gemini 2.5 / 3.1 Pro	Competitive	Strong	Strong	Strong

Qwen3-235B-A22B Thinking is currently one of the few open-weight models with a published Humanity's Last Exam score (~18%) — putting it in the same conversation as frontier closed models for reasoning-heavy tasks.

Architecture Pattern: Make LLMs Replaceable

The biggest mistake teams make is hard-coding a model into their architecture.

❌ The fragile pattern

Application → GPT-4o-mini (hardcoded)

Any migration requires touching application logic, service config, and prompt templates.

✅ The replaceable pattern

Benefits:

Vendor independence — swap providers via config
Easier model upgrades without app rewrites
Enables cost-optimised routing

Cost Optimisation: Intelligent Request Routing

Many production AI systems at scale route requests by task complexity rather than using one model for everything.

This pattern can reduce LLM costs by 60–90% in workloads with a mix of simple and complex requests.

Prompt Regression Testing — Non-Negotiable

Before committing to any model swap, run prompt regression tests on your real production prompts.

# Simple regression harness
test_prompts = load_production_samples(n=200)

results = []
for prompt in test_prompts:
    output_a = old_model.invoke(prompt)
    output_b = new_model.invoke(prompt)

    results.append({
        "prompt": prompt,
        "score_a": evaluate(output_a),   # correctness, format, hallucination
        "score_b": evaluate(output_b),
        "regression": score_b < score_a * 0.95
    })

regressions = [r for r in results if r["regression"]]
print(f"Regression rate: {len(regressions)/len(results)*100:.1f}%")

Check for:

Correctness — does the answer change?
Format compliance — does it still follow your output structure?
Hallucination rate — does it fabricate facts?
Latency — does it still meet your SLA?

⚠️ Important Disclaimer

The mappings in this guide are a starting point, not guaranteed drop-in replacements.

Models in the same tier can have meaningfully different behaviour on:

Your specific domain vocabulary
Your prompt style
Edge cases in your data

Every migration must include:

Prompt regression testing on real data
Human evaluation of sampled outputs
Shadow traffic validation (run both models in parallel, compare outputs)
Gradual rollout (5% → 25% → 100%)

Recommended Migration Workflow

Do not skip shadow traffic. It catches subtle regressions that prompt tests miss.

📋 LLM Migration Cheat Sheet

Mini Tier (`gpt-4o-mini` equivalent)

Azure → Llama 4 Scout, Qwen3-8B/14B, DeepSeek V3, Claude Haiku
AWS → Claude Haiku, Mistral Small
GCP → Gemini Flash-Lite, Gemini Flash

Standard Tier (`gpt-4o` equivalent)

Azure → Claude Sonnet, Llama 4 Maverick, DeepSeek V3
AWS → Claude Sonnet, Mistral Medium
GCP → Gemini Flash, Gemini 2.5 Pro

Reasoning Tier (o-series / `gpt-5` equivalent)

Azure → Claude Opus, DeepSeek R1, Qwen3-235B Thinking
AWS → Claude Opus
GCP → Gemini 2.5 Pro, Gemini 3.1 Pro

Final Takeaway

The LLM ecosystem is evolving too fast to depend on a single provider.

Design your systems so that:

GPT → Claude → Gemini → DeepSeek

…is a configuration change, not a system rewrite.

When that happens, migrations become boring infrastructure work.

And boring infrastructure is exactly what you want in production.

Found this useful? Follow TheProdSDE for more practical engineering guides on AI systems, cloud architecture, and developer tooling.

Tags: #ai #llm #cloud #azure #systemdesign

Worth your 10min of time if deciding which framework suits your use case better

TheProdSDE — Fri, 13 Mar 2026 14:48:04 +0000

LangGraph vs Semantic Kernel: Python AI Agents in 2026

TheProdSDE ・ Mar 11

#python #ai #agents #langchain

[Boost]

TheProdSDE — Fri, 13 Mar 2026 14:47:02 +0000

7 AI Prompts Senior Engineers Use to Build Production Code Faster

TheProdSDE ・ Mar 13

#ai #programming #productivity #software

7 AI Prompts Senior Engineers Use to Build Production Code Faster

TheProdSDE — Fri, 13 Mar 2026 13:15:10 +0000

AI coding tools are powerful — but most developers still use them like autocomplete.
The difference between average AI usage and 10x productivity often comes down to one thing:
how you prompt the system.

In a previous article, I explained why teams should stop treating AI like a magic architect and instead treat it like a fast junior engineer inside a well-designed system.

But that raises an important question:

How do experienced engineers actually prompt AI tools in real production workflows?

Below are 7 practical prompts senior engineers use to guide AI coding assistants toward reliable, production-quality output.

What You'll Learn

How senior engineers structure prompts for AI coding tools
Why constraints improve AI-generated code
Practical prompts you can reuse in real projects
How to avoid common AI development mistakes

Prompt 1 — Spec → Implementation

One of the most effective patterns is defining the specification first.

Instead of asking the AI to build a feature from scratch, you provide clear constraints and structure.

Example prompt:

You are a senior backend engineer.

Implement the following interface in TypeScript.

Constraints:
- Follow the existing repository pattern
- Do not introduce new dependencies
- Use the existing error handling system
- Write clean, maintainable code

Interface:
[PASTE INTERFACE HERE]

Why this works:

The engineer controls architecture
AI performs mechanical implementation
Code stays consistent with the existing system

Prompt 2 — Safe Refactoring

Refactoring is one of the safest and most valuable uses of AI coding tools.

Example prompt:

Refactor this service to extract the email logic into a NotificationService.

Constraints:
- Do not change public APIs
- Preserve existing behavior
- Maintain current tests
- Avoid introducing circular dependencies

Benefits:

Faster multi-file refactoring
Lower risk of manual mistakes
Improved code organization

AI becomes a high-speed refactoring assistant.

Prompt 3 — Test Generation

Writing test scaffolding is another task where AI shines.

Example prompt:

Generate Jest unit tests for this service.

Constraints:
- Cover success and failure cases
- Mock external dependencies
- Follow our existing test structure
- Focus on behavior rather than implementation details

AI can generate:

basic test structures
mocks
scenario coverage

You still review tests, but the initial draft appears instantly.

Prompt 4 — Code Review Assistant

AI can act as a secondary reviewer for pull requests.

Example prompt:

Review the following code as a senior engineer.

Focus on:
- architecture issues
- potential bugs
- performance concerns
- unnecessary complexity
- security risks

This helps engineers quickly surface:

hidden edge cases
inefficient patterns
potential race conditions

It does not replace human review, but it improves signal detection.

Prompt 5 — Documentation Generation

Documentation is often neglected in fast-moving teams.

AI can generate a first draft instantly.

Example prompt:

Generate documentation for this API endpoint.

Include:
- request format
- response format
- error conditions
- example usage
- explanation of business logic

Engineers then refine the output instead of writing everything from scratch.

Prompt 6 — Guided Multi-File Refactors

Large refactors across many files are tedious and error-prone.

AI can help coordinate them.

Example prompt:

Find all usages of sendEmail across the repository.

Refactor them to use NotificationService.

Constraints:
- Preserve existing behavior
- Do not modify unrelated logic
- Update imports if necessary

AI tools integrated with IDEs can perform these changes much faster than manual edits.

Prompt 7 — Architecture Sanity Check

One of the most useful prompts is asking AI to analyze architectural risks.

Example prompt:

Analyze this module for potential architecture issues.

Look for:
- tight coupling
- duplicated logic
- boundary violations
- hidden dependencies

This works surprisingly well when reviewing complex modules.

AI often catches patterns that are easy to miss during routine coding.

Why Constraints Make AI Better

A common mistake developers make is using vague prompts.

Example of a weak prompt:

Build authentication for my app

The AI must guess too much.

Better prompts:

define constraints
reference existing patterns
specify architecture boundaries

The clearer the prompt, the more reliable the output.

The Key Insight

AI coding tools are not magic architects.

They are execution engines.

Senior engineers do not ask AI to design systems.

They ask AI to implement clearly defined pieces of a system.

Final Thoughts

AI coding assistants are becoming a permanent part of the developer toolbox.

But the biggest productivity gains do not come from the tool itself.

They come from learning how to guide the tool effectively.

The engineers who benefit most from AI are not the ones who rely on it blindly.

They are the ones who combine strong engineering discipline with precise prompts.

And in that workflow, AI becomes less like autocomplete…

…and more like a very fast junior engineer.

Written by TheProdSDE — sharing insights on AI engineering, system design, and developer productivity.

Stop Treating AI Like Autocomplete — Design an AI-First Developer Workflow Instead

TheProdSDE — Thu, 12 Mar 2026 13:51:28 +0000

AI coding tools are everywhere — but most teams still use them like autocomplete.
The result? Faster code generation… and faster technical debt.

In 2026, almost every developer works with AI tools inside their IDE.

You type a function name.
Your AI assistant confidently generates 40 lines of code.

You accept it because the deadline is tight.

Two weeks later your team is debugging the same AI-generated code.

The issue isn't AI.

The issue is the workflow around it.

The teams getting massive productivity gains are not the ones simply using AI tools.

They are the ones engineering a workflow around AI.

💬 Quick question for you:

How are you currently using AI coding tools?

autocomplete replacement
debugging helper
architecture assistant
full feature generation

Drop your workflow in the comments — I'm curious how different teams are approaching this.

What You'll Learn

Why AI coding tools create hidden technical debt
The correct mental model for AI-assisted development
A practical AI-first development workflow
Real engineering patterns that work

* Rules for keeping humans in control

TL;DR

Most teams use AI coding tools like autocomplete.

That works for small tasks — but breaks down in real systems.

The better workflow:

Engineers design the architecture
AI generates implementation
Humans review the output
Tests validate behavior

AI shouldn't design your system.

It should accelerate implementation inside a system you already designed.

The Real Problem

AI coding assistants are now part of everyday development.

Developers use them to:

generate boilerplate
explain unfamiliar code
write tests
refactor functions
scaffold APIs

But problems appear when teams move from:

"AI helps write code"

"AI probably knows what the system should do."

That shift introduces hidden risk.

What AI Is Very Good At

Generating repetitive code
Translating intent into a first draft
Creating test scaffolding
Performing large refactors
Accelerating low-risk implementation

What AI Is Still Bad At

Understanding business rules
Making architectural trade-offs
Preserving system boundaries
Long-term maintainability
Handling ambiguity safely

AI doesn't remove engineering discipline.
It amplifies whatever discipline already exists in the team.

If the workflow is weak, AI accelerates bad architecture.

If the workflow is strong, AI accelerates productivity.

The Correct Mental Model

The best way to think about AI coding tools is simple:

Treat AI like a fast junior engineer inside your system.

A good junior engineer can:

implement well-defined tasks
write boilerplate
perform refactors
generate first drafts

But they still require:

clear requirements
architecture constraints
code review

AI works exactly the same way.

This workflow works especially well with AI-enabled IDEs like Cursor or coding assistants like GitHub Copilot.

A healthy AI development workflow looks like this:

Problem Definition
      ↓
Interface Design
      ↓
AI Implementation
      ↓
Human Code Review
      ↓
Testing & Validation
      ↓
Merge

Humans control architecture.

AI accelerates execution.

The Most Dangerous AI Workflow

Many teams accidentally follow this workflow:

Open IDE
Ask AI to build a feature
Accept generated code
Ship it

It feels fast.

But it introduces:

hidden coupling
duplicated logic
architecture violations
security risks

Speed without discipline is technical debt at machine speed.

A Practical AI-First Workflow

A better engineering model looks like this:

Define the problem clearly
Design interfaces first
Ask AI to implement inside constraints
Review like a pull request
Standardize repeatable prompts

Human-Owned Work

Engineers must still own:

system architecture
service boundaries
security decisions
data contracts
performance trade-offs
business rules

AI-Assisted Work

AI works best for:

boilerplate code
test scaffolding
repetitive refactors
documentation drafts
DTO and mapper generation
CRUD scaffolding

AI executes inside the system you designed.

Practical Pattern #1 — Spec → Design → AI Implementation

Weak prompt:

Build authentication for my app

Too vague.

Better workflow:

Step 1 — Lightweight spec

Feature: Email/password login

POST /api/auth/login
Request: { email, password }

Response:
{
  accessToken,
  refreshToken,
  user: { id, email, roles }
}

Step 2 — Define interface

export interface AuthService {
  login(email: string, password: string): Promise<LoginResult>;
  refresh(token: string): Promise<LoginResult>;
}

export interface LoginResult {
  accessToken: string;
  refreshToken: string;
  user: {
    id: string;
    email: string;
    roles: string[];
  };
}

Step 3 — AI implementation

Prompt:

Implement AuthService using UserRepository, bcrypt password checks, and the existing JWT helper.

Now:

You control the contract
AI handles mechanical work
Architecture remains stable

Practical Pattern #2 — Guided Refactors

Example: centralizing email logic behind a service.

Define abstraction first:

export interface NotificationService {
  sendWelcomeEmail(userId: string): Promise<void>;
  sendPasswordResetEmail(userId: string, resetToken: string): Promise<void>;
}

Then instruct AI:

Find direct sendEmail calls and route them through NotificationService.

Now the AI performs the mechanical refactor while architecture stays consistent.

Practical Pattern #3 — AI First Draft, Human Final Draft

AI can generate boilerplate quickly.

Example:

export function toUserResponse(entity: UserEntity): UserResponse {
  return {
    id: entity.id,
    email: entity.email,
    roles: entity.roles,
    createdAt: entity.createdAt.toISOString(),
    updatedAt: entity.updatedAt.toISOString(),
  };
}

Human refinement:

import { formatIsoDate } from "../utils/date";

export function toUserResponse(entity: UserEntity): UserResponse {
  return {
    id: entity.id,
    email: entity.email,
    roles: [...entity.roles],
    createdAt: formatIsoDate(entity.createdAt),
    updatedAt: formatIsoDate(entity.updatedAt),
  };
}

AI saves typing.

Engineers enforce consistency.

Practical Pattern #4 — AI as Test Author

Define test strategy first.

// Scenarios:
// valid credentials → success
// invalid email → AUTH_INVALID_CREDENTIALS
// invalid password → AUTH_INVALID_CREDENTIALS
// locked account → AUTH_ACCOUNT_LOCKED

Then prompt AI:

Generate Jest tests covering these scenarios.

You then review for:

missing edge cases
flaky tests
incorrect assumptions

The AI Development Rules I Follow

Simple rules that dramatically improve outcomes:

Never generate architecture with AI
Always define interfaces first
Constrain AI to specific files
Review AI code like junior code
Never merge AI code without tests
Standardize prompts for repeatable workflows

These guardrails keep velocity high without sacrificing quality.

Real Tools Supporting This Workflow

Modern tools supporting this approach include:

GitHub Copilot
Claude Code
Cursor

These tools are powerful — but the workflow around them matters more than the tool itself.

One Insight That Changes Everything

AI does not remove engineering responsibility.

It amplifies the engineering discipline already present in the team.

Bad workflows produce bad code faster.

Good workflows produce good code faster.

The difference is not the AI.

The difference is the system around it.

Before adopting AI agents across your team, make sure the workflow is clear.

Otherwise you are simply scaling chaos faster.

AI coding agents are now a standard part of software development.

The real competitive advantage is not using AI.

It is engineering workflows around AI.

If you treat AI like a magic senior engineer, you will be disappointed.

If you treat it like a fast junior inside a strong engineering system, you can ship faster without sacrificing code quality.

The next generation of software development is not AI-powered.

It is AI-orchestrated.

If you're experimenting with AI development workflows, I'd love to hear what has worked (or failed) for your team.

Drop your experiences in the comments 👇

Follow TheProdSDE for more content on:

AI engineering
system design
developer productivity

LangGraph vs Semantic Kernel: Python AI Agents in 2026

TheProdSDE — Wed, 11 Mar 2026 12:41:03 +0000

Why This Comparison Matters Right Now

Here's the honest reality of building Python AI agents in 2026: you have two genuinely good framework choices, and picking the wrong one for the wrong problem will cost you architecture refactors, not just a few hours of code changes.

LangGraph and Semantic Kernel have both crossed major milestones since most popular comparisons were written:

LangGraph hit v1.0 in October 2025 — with a formal stability commitment and no breaking changes until 2.0. (official announcement)
LangChain 1.0's create_agent now runs on the LangGraph runtime underneath, making LangGraph the execution engine of the LangChain ecosystem. (LangChain/LangGraph 1.0 blog)
Semantic Kernel shipped first-class MCP support for Python in v1.28.1 — SK can now act as both an MCP client and server natively in the SDK. (official SK dev blog)

If you're still reading comparisons that call LangGraph "unstable" or Semantic Kernel "too tied to .NET", you're reading old content.

This post is grounded in the official LangGraph docs, the official Semantic Kernel docs, and both framework changelogs. Let's go.

TL;DR: The One-Line Decision Rule

Problem Type	Framework
Stateful multi-step agent workflows	LangGraph
Plugin-based AI platforms & MCP interoperability	Semantic Kernel

That distinction explains every trade-off in this article.

Architecture: Two Very Different Mental Models

LangGraph — The Graph Runtime

LangGraph models your agent system as a stateful graph where you explicitly define state, nodes, and edges. Nodes are Python callables or subgraphs. Edges are transitions. State is a typed object that flows through the graph and gets updated at each step.

That's not an internal implementation detail — it's the primary abstraction you work with every day.

The official LangGraph v1 docs describe the framework around three core ideas: durable execution, controllability, and human-in-the-loop. Resuming a workflow from the last checkpoint after a crash, inserting a human review step, or branching into parallel sub-agents are first-class operations — not workarounds.

Since LangGraph v1, LangChain's create_agent lives on top of this runtime. The stack now has a clean separation:

Start with create_agent for standard tool-calling loops.
Drop down to raw LangGraph when you need explicit workflow topology.

Semantic Kernel — The Kernel-Plugin Middleware

Semantic Kernel starts from the Kernel abstraction, which holds AI services, plugins, and functions. Plugins are groups of functions exposed to the model and to agents, and can come from native Python code, prompt templates, or imported external schemas.

The official SK agent-functions docs state:

"Any Plugin available to an Agent is managed within its respective Kernel instance — this enables each Agent to access distinct functionalities based on its specific role."

Orchestration emerges from agents choosing functions and planners sequencing capability calls — rather than from a graph topology you define up front.

This makes Semantic Kernel feel more like AI middleware. You shape what your agent can do, then let function calling and the agent framework decide how to do it.

Architectural Difference — Quick Reference

Dimension	LangGraph	Semantic Kernel
Primary abstraction	Typed state graph (nodes + edges)	Kernel + plugins + agents
Workflow control	You define topology explicitly	Emerges from agent function calling
State management	First-class typed state + checkpointing	Externalized per service or plugin
Best mental model	Durable state machine for agents	AI middleware with composable capabilities

Code: The Same Agent in Both Frameworks

To make the architectural differences concrete, let's build the same agent in both: a multi-turn weather assistant with memory and a system prompt.

LangGraph — Weather Agent with Checkpointing

Pattern from the official LangGraph agents quickstart

# pip install -U langgraph "langchain[openai]"

from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import InMemorySaver
from langchain.chat_models import init_chat_model

# --- Tool: plain Python function ---
def get_weather(city: str) -> str:
    """Get the current weather for a given city."""
    # Replace with a real API call in production
    return f"It's sunny and 28°C in {city}."

# --- LLM ---
model = init_chat_model("openai:gpt-4o-mini", temperature=0)

# --- Checkpointer enables durable multi-turn memory ---
# Swap InMemorySaver for SqliteSaver or PostgresSaver in production
checkpointer = InMemorySaver()

# --- Compile graph agent ---
agent = create_react_agent(
    model=model,
    tools=[get_weather],
    prompt="You are a helpful weather assistant.",
    checkpointer=checkpointer,
)

# --- thread_id binds this conversation to a persistent checkpoint ---
config = {"configurable": {"thread_id": "user-session-1"}}

# Turn 1
response = agent.invoke(
    {"messages": [{"role": "user", "content": "What is the weather in Mumbai?"}]},
    config=config,
)
print(response["messages"][-1].content)

# Turn 2 — agent remembers context automatically via checkpointer
followup = agent.invoke(
    {"messages": [{"role": "user", "content": "How about Delhi?"}]},
    config=config,
)
print(followup["messages"][-1].content)

What is happening architecturally:

create_react_agent compiles a StateGraph with a model-tool loop under the hood.
The checkpointer persists state at every step; the same thread_id resumes from the last saved state automatically.
If the process crashes mid-run, restarting and invoking with the same thread_id picks up from the last checkpoint — durability is a runtime concern, not your concern.

Semantic Kernel — Weather Agent with Plugin

Pattern from the official SK agent-functions docs

# pip install semantic-kernel

import asyncio
from semantic_kernel import Kernel
from semantic_kernel.agents import ChatCompletionAgent
from semantic_kernel.connectors.ai.open_ai import (
    OpenAIChatCompletion,
    OpenAIChatPromptExecutionSettings,
)
from semantic_kernel.connectors.ai import FunctionChoiceBehavior
from semantic_kernel.functions import kernel_function
from semantic_kernel.contents import ChatHistory

# --- Plugin: class with @kernel_function decorators ---
class WeatherPlugin:
    @kernel_function(name="get_weather", description="Get the weather for a city.")
    def get_weather(self, city: str) -> str:
        # Replace with a real API call in production
        return f"It's sunny and 28°C in {city}."

# --- Kernel: holds services and plugins ---
kernel = Kernel()
kernel.add_service(OpenAIChatCompletion(ai_model_id="gpt-4o-mini"))

# --- Execution settings: enable auto function calling ---
settings = OpenAIChatPromptExecutionSettings()
settings.function_choice_behavior = FunctionChoiceBehavior.Auto()

# --- Register plugin ---
kernel.add_plugin(WeatherPlugin(), plugin_name="WeatherPlugin")

# --- Agent: kernel + instructions ---
agent = ChatCompletionAgent(
    kernel=kernel,
    name="WeatherAssistant",
    instructions="You are a helpful weather assistant.",
)

async def run_agent():
    # ChatHistory is your responsibility to maintain across turns
    history = ChatHistory()

    # Turn 1
    history.add_user_message("What is the weather in Mumbai?")
    async for message in agent.invoke(history):
        print(f"Agent: {message.content}")
        history.add_message(message)

    # Turn 2
    history.add_user_message("How about Delhi?")
    async for message in agent.invoke(history):
        print(f"Agent: {message.content}")
        history.add_message(message)

asyncio.run(run_agent())

What is happening architecturally:

The Kernel holds the AI service and plugins as a dependency container.
@kernel_function decorators make Python methods discoverable and invocable by the model automatically.
FunctionChoiceBehavior.Auto() tells the model to call functions when needed.
Memory is a ChatHistory object you manage and pass into each invocation. The runtime does not persist it for you.

The Most Revealing Difference in 6 Lines

# LangGraph — runtime owns durability
checkpointer = InMemorySaver()
config = {"configurable": {"thread_id": "session-1"}}
agent.invoke(messages, config)  # resumes from last checkpoint automatically

# Semantic Kernel — you own state
history = ChatHistory()
history.add_user_message("...")
agent.invoke(history)  # you pass and maintain state explicitly

In LangGraph, durability is a runtime concern. In Semantic Kernel, state management is your concern. Neither is wrong — they match different application models.

Protocol Support: MCP and A2A

This is where Semantic Kernel has made its most significant leap recently.

Semantic Kernel — Native MCP in the Python SDK

The official SK MCP announcement states:

"Python support for MCP has arrived... SK Python can act as both an MCP Host and an MCP Server, support multiple transport methods (stdio, SSE, WebSocket), chain multiple MCP servers together, and expose SK functions or agents as MCP servers."

That is not an adapter or community plugin. It is first-class SDK support from v1.28.1+. For teams building tools and agents that need to cross service boundaries via a standard protocol, this is a meaningful architectural upgrade.

LangGraph — Strong MCP at the Deployment Edge

LangGraph's MCP story is more about deployment than in-process integration. When deployed on the LangGraph Platform, every agent is automatically exposed as an MCP-accessible endpoint at /mcp with no extra code required. For self-hosted deployments, integration is available via the langchain-mcp-adapters package.

Bottom line:
SK is stronger when you want MCP semantics inside your Python process. LangGraph is stronger when you think of agents as deployed services that other clients consume via MCP.

Stability and Breaking Changes: The 2026 Reality

Here is what the official docs actually say now.

LangGraph v1 (October 2025): The official v1 release notes state that the core graph APIs and execution model are unchanged. The main migration note is deprecation of create_react_agent in langgraph.prebuilt in favour of LangChain's create_agent. The LangGraph 1.0 announcement explicitly commits to no breaking changes until 2.0.

Semantic Kernel 1.x: Most architectural disruption landed at 1.0 (namespace reorg, API renames, context variable changes). The H1 2025 SK roadmap and subsequent releases show an incremental, additive pattern with targeted fixes rather than structural breaks.

The old narrative of "LangGraph breaks every release" is no longer accurate. Both frameworks are now in a stability-first phase.

Updated Technical Ratings (March 2026)

Based on official docs and both frameworks' current stable releases:

Dimension	LangGraph	Semantic Kernel
Workflow control	⭐ 4.8 / 5	⭐ 4.0 / 5
Durable execution	⭐ 4.9 / 5	⭐ 4.1 / 5
Plugin/tool architecture	⭐ 4.2 / 5	⭐ 4.8 / 5
MCP interoperability	⭐ 3.9 / 5	⭐ 4.9 / 5
Flow debuggability	⭐ 4.7 / 5	⭐ 3.9 / 5
Start simple, scale complex	⭐ 4.8 / 5	⭐ 4.4 / 5
Python DX overall	⭐ 4.6 / 5	⭐ 4.5 / 5

The scores are intentionally close. Both are production-grade frameworks solving real problems well. The winner for your team is whichever abstraction maps better to how you think about the problem you are solving.

When to Choose Which

✅ Choose LangGraph when:

Your agent logic involves non-trivial branching, retries, human review, or approval steps that benefit from explicit graph topology.
You need durable execution — workflows that survive crashes, resume from checkpoints, and have auditable step history.
You are already invested in the LangChain ecosystem and want the clean create_agent → LangGraph stack with a clear upgrade path.
You want fine-grained observability into how execution moved through a workflow at the node level.

✅ Choose Semantic Kernel when:

You are building a platform or SDK where capabilities are composed as plugins and different agents consume different tool surfaces.
MCP or A2A interoperability is a core requirement and you want it natively in the Python SDK, not via adapters.
Your team already uses a DI/service-oriented architecture and the kernel-plugin model maps naturally to it.
You want lightweight deployment without a dedicated orchestration runtime and can manage state externally.

The One-Line Rule — Revisited

If your agent needs to behave like a durable state machine, use LangGraph.
If your agent needs to behave like a protocol-aware platform component, use Semantic Kernel.

That is the comparison most blog posts are not making. Hopefully this one was useful.

DEV Community: TheProdSDE

MCP for Backend Engineers: When to Use It (and When to Skip It)

TL;DR

Why This Exists

MCP in One Line

The Hero Diagram: MCP vs Tool Calling

What MCP Actually Is

Case Study: Wrapping a Client REST API

Should You Use MCP? — Decide Before Reading Further

Five Real Scenarios

Decision Flow

Where MCP Is a Bad Idea

Core MCP Concepts

Server-side primitives

Client-side capabilities

Request / Response Lifecycle

Minimal MCP Server (Python)

MCP Client + Agent Loop

Auth — The Part Most Tutorials Skip

Real-World Cost (Numbers, Not Theory)

Observability & Debugging

Testing & QA

Production Checklist

Deploying on Azure

Azure Deployment Decision Table

Option 1 — Azure Container Apps (start here)

Option 2 — AKS (if you already run a cluster)

Option 3 — Azure Functions (infrequent, event-driven tools)

Option 4 — App Service (add /mcp to an existing service)

Common Failure Modes

What MCP Does NOT Solve

Migration Path

Further Resources

The Honest Verdict

Related Reading

Worth Each minute read if you work on RAG

Why Most RAG Systems Fail in Production (And How to Design One That Actually Works)

Agentic AI Fails in Production for Simple Reasons — What MLDS 2026 Taught Me

A Key Shift: From Models to Systems

Policy Learning vs. Structural Intelligence

Runtime Policy Learning

Structural Intelligence at Design Time

Validation-First Agent Design

Agentic Memory: Accuracy, Cost, and Trust

Data Platforms and Practical Architecture Choices

Putting Agents into Production: Real-World Risks

AI-Assisted Development Needs Guardrails

Final Reflections

Why Most RAG Systems Fail in Production (And How to Design One That Actually Works)

The Reality: RAG Fails Quietly

What a Production RAG System Actually Looks Like

Step 1: Parsing Matters More Than You Think

Problem

Production Approach

Step 2: Dense vs Sparse Retrieval (You Need Both)

Dense Retrieval (Embeddings)

Sparse Retrieval (BM25 / Keyword)

Production Pattern: Hybrid Retrieval

Step 3: Reranking (The Accuracy Multiplier)

Step 4: Context Building (Where Systems Win or Lose)

Common Mistakes

Production Approach

Vector DB vs Graph DB — When to Use What

Use Vector Database When

Use Graph Database When

Hybrid (Real Systems)

RAG Is Not Single-Turn — Managing Context Over Time

The Problem: Context Drift

Production Strategy: Context Is a Filter

Context Layers

When to Summarize vs Include Raw History

Include Raw

Summarize

Critical Rule

Handling User Corrections (Critical for Trust)

Strategy

Agentic RAG (When Retrieval Needs Reasoning)

Architecture

Use It When

Avoid It When

Option 4 — App Service (add `/mcp` to an existing service)

Tier 1 — Replacing `gpt-4o-mini`

Tier 2 — Replacing `gpt-4o`

Tier 3 — Replacing Reasoning Models (`gpt-4.1` / `gpt-5` / o-series)

Mini Tier (`gpt-4o-mini` equivalent)

Standard Tier (`gpt-4o` equivalent)

Reasoning Tier (o-series / `gpt-5` equivalent)