DEV Community: TheProdSDE

Worth Each minute read if you work on RAG

TheProdSDE — Mon, 06 Apr 2026 09:59:14 +0000

TheProdSDE

Mar 24

Why Most RAG Systems Fail in Production (And How to Design One That Actually Works)

#ai #rag #agents #software

Comments 1

4 min read

Agentic AI Fails in Production for Simple Reasons — What MLDS 2026 Taught Me

TheProdSDE — Tue, 31 Mar 2026 14:56:35 +0000

TL;DR:
Most agentic AI failures in production are not caused by weak models, but by stale data, poor validation, lost context, and lack of governance. MLDS 2026 reinforced that enterprise‑grade agentic AI is a system design problem, requiring validation‑first agents, structural intelligence, strong observability, memory discipline, and cost‑aware orchestration—not just bigger LLMs.

I recently attended MLDS 2026 (Machine Learning Developer Summit) by Analytics India Magazine (AIM) in Bangalore. While many sessions featured advanced models and agentic frameworks, the most valuable insight was unexpected:

Most AI systems don’t fail in production because of bad models — they fail because of bad systems.

Across the summit, speakers repeatedly showed that issues like stale data, missing validation, poor observability, and uncontrolled execution are what derail agentic AI at scale—not lack of intelligence.

A recurring theme across sessions was clear: the hardest problem in AI today is no longer building impressive demos, but running AI systems reliably at enterprise scale. Many real-world failures stem from system design gaps rather than model limitations.

A Key Shift: From Models to Systems

One of the most important takeaways from the summit was that enterprise AI is fundamentally a system design problem, not a model selection problem.

Multiple speakers highlighted common failure modes seen in production:

Stale or outdated data
Poor data granularity
Context loss across multi-step workflows
False confidence and lack of validation
Black-box decisions with no observability

This explains why many AI solutions look powerful in prototypes but break down in real operational environments.

Policy Learning vs. Structural Intelligence

A particularly insightful discussion contrasted two approaches:

Runtime Policy Learning

Examples include Reinforcement Learning (RL), MADDPG, and Graph Neural Networks (GNNs):

Dynamic decision-making
GPU-intensive
Higher cost and latency
Harder to govern and observe

Structural Intelligence at Design Time

In this approach, intelligence is encoded into the system structure itself, often using graph-based designs:

Relationships are resolved at construction time
Minimal runtime inference
Deterministic behavior
Lower cost and faster response

Key insight: Not every intelligent system needs continuous runtime learning. When relationships are stable, embedding intelligence structurally can be more efficient and reliable.

Validation-First Agent Design

Another strong theme was the shift toward validation-first agents, not answer-first agents.

Successful agentic systems:

Ground every important output to source data
Track freshness and provenance
Validate semantics before taking actions
Plan explicitly before executing
Expose confidence where appropriate

Several talks emphasized that observability should evolve from “what happened?” to “was the result actually correct?”.

Agentic Memory: Accuracy, Cost, and Trust

Sessions on agentic memory highlighted how short-term memory, long-term memory, and pruning strategies directly influence:

Accuracy
Latency
Cost
User trust

The key takeaway was that memory should be treated as a first-class architectural concern, with explicit design choices and benchmarks—rather than an ad-hoc cache bolted on later.

Data Platforms and Practical Architecture Choices

The summit also covered modern data platforms that unify OLTP and OLAP workloads, with strong support for time-series data. These architectures reduce complexity and make near–real-time analytics more accessible.

A broader lesson emerged: cost, latency, reliability, and accuracy must be designed together. Choosing larger models without optimizing workflows, routing, and memory leads to unnecessary compute cost and slower systems.

Putting Agents into Production: Real-World Risks

One session focused entirely on lessons learned from deploying agents in production. Four recurring risks were highlighted:

Silent failures – systems appear healthy but produce wrong outputs
Black-box decisions – lack of explainability and traceability
Permission explosion – agents accumulating excessive access
Runaway execution – uncontrolled tool calls and rising costs

These issues reinforce the importance of governance, guardrails, observability, and scoped execution from day one.

AI-Assisted Development Needs Guardrails

Another notable takeaway was the need to pair AI-assisted code generation with strong static analysis and security validation. Integrations with tools like SonarQube demonstrate how AI-written and human-written code can be:

Validated automatically
Secured against vulnerabilities
Fixed via generated pull requests

This closes the gap between productivity gains and production reliability.

Final Reflections

MLDS 2026 reinforced a critical idea:

The future of AI in enterprises depends more on architecture, validation, and governance than on model strength alone.

Agentic AI succeeds when it is:

Grounded in reliable data
Observable and debuggable
Cost-aware and execution-bounded
Designed around real workflows
Rolled out with clear trust and adoption strategies

The biggest mindset shift is moving from “How powerful is the model?” to “How reliable and efficient is the end-to-end intelligent workflow?”

That, more than anything, was the most valuable learning from the summit.

If you’re working on agentic AI in production, I’d love to hear:

Where have agents broken down for you?
What controls or guardrails helped the most?
Are you handling validation and memory explicitly—or implicitly?

Let’s compare notes.

Why Most RAG Systems Fail in Production (And How to Design One That Actually Works)

TheProdSDE — Tue, 24 Mar 2026 12:23:04 +0000

A practical, system design–focused breakdown of why RAG systems degrade after launch—and what actually works in production.

Everyone builds a RAG system.

And almost all of them work — in demos.

Clean query
Relevant chunks
Decent answer

Ship it.

Then production happens.

Users ask vague follow-ups
Retrieval returns partial context
The model answers confidently… and incorrectly

And suddenly:

Your “working” RAG system becomes unreliable.

The Reality: RAG Fails Quietly

RAG doesn’t crash. It degrades.

Slightly wrong answers
Missing context
Hallucinated explanations with citations

Which is worse than a system that fails loudly.

Most teams blame:

embeddings
vector database
chunk size

But in real systems:

RAG failures are usually system design failures—not retrieval failures.

What a Production RAG System Actually Looks Like

Not this:

Query → Vector DB → LLM

But this:

Step 1: Parsing Matters More Than You Think

Most pipelines start like this:

text = pdf.read()
chunks = split(text)
embeddings = embed(chunks)

This is where things already break.

Problem

PDFs lose structure
Tables turn into noise
Headers/footers pollute chunks
Sections lose meaning

Production Approach

Document → Layout-aware parsing → Structured sections → Clean chunks

Key principles:

preserve headings and hierarchy
remove boilerplate
chunk by meaning, not length

If parsing is wrong, retrieval will always be wrong.

Step 2: Dense vs Sparse Retrieval (You Need Both)

Dense Retrieval (Embeddings)

semantic similarity
handles vague queries
fails on exact matches

Sparse Retrieval (BM25 / Keyword)

exact term matching
works for IDs, clauses
ignores meaning

Production Pattern: Hybrid Retrieval

This gives:

semantic understanding
exact precision

Using only vector search is a common production mistake.

Step 3: Reranking (The Accuracy Multiplier)

Top-K retrieval is noisy.

Add a reranker (cross-encoder):

evaluates (query, chunk) pairs
reorders by true relevance

This significantly improves answer quality without changing your database.

Step 4: Context Building (Where Systems Win or Lose)

Even with good retrieval, most failures happen here.

Common Mistakes

stuffing too many chunks
mixing unrelated documents
ignoring token limits

Production Approach

select top-ranked chunks only
preserve document structure
enforce token budget
maintain ordering

Better context > more context

Vector DB vs Graph DB — When to Use What

Use Vector Database When

unstructured data
semantic search
document retrieval

Use Graph Database When

relationships matter
multi-hop reasoning
structured entities

Hybrid (Real Systems)

Use graph when relationships matter.
Use vector when meaning matters.
Use both when systems get complex.

RAG Is Not Single-Turn — Managing Context Over Time

Most systems fail here.

RAG is not just:

retrieve → answer

It’s:

retrieve → answer → follow-up → correction → refinement

The Problem: Context Drift

If you blindly append chat history:

token usage explodes
wrong answers get reinforced
relevance drops

Production Strategy: Context Is a Filter

Not a dump.

Context Layers

Store full history
Select only relevant turns
Exclude invalid or corrected responses
Combine with retrieved context

When to Summarize vs Include Raw History

Include Raw

short conversations
active refinement
recent corrections

Summarize

long conversations (>5–7 turns)
approaching token limits

Critical Rule

Summarize facts—not hallucinations.

If a previous answer was wrong:

exclude it
prioritize user correction

Handling User Corrections (Critical for Trust)

Users will fix your system.

If you ignore that, the system feels broken.

Strategy

mark incorrect responses
exclude them from future context
boost corrected information

Example:

{
  "turn_id": 8,
  "valid": false,
  "corrected": true
}

Agentic RAG (When Retrieval Needs Reasoning)

Basic RAG is static.

Agentic RAG adds:

planning
iteration
tool usage

Architecture

Use It When

multi-step queries
missing context
dynamic retrieval

Avoid It When

simple Q&A
strict latency requirements

Otherwise you're adding complexity without ROI.

Confidence Scores and Citations (Trust Layer)

Without trust signals, users don’t trust answers.

Citations

Always return:

source document
section or chunk reference

Confidence Score (Simple Heuristic)

Combine:

retrieval score
reranker score
validation signal

Example:

confidence =
  0.4 * retrieval +
  0.4 * reranker +
  0.2 * validation

Optional Validation Step

Ask the model:

“Is this answer fully supported by the context?”

Lower confidence if not.

Guardrail: Don’t Trust the Model Alone

Even with RAG:

hallucinations still happen
citations can be fabricated

Enforce:

answers must reference retrieved chunks
no context → no answer

Final Architecture (Multi-Turn RAG System)

Production Checklist

If your system doesn’t have these, it will fail:

structured parsing
hybrid retrieval
reranking
controlled context building
memory filtering
correction handling
confidence + citations
observability

The Real Rule

RAG is not a retrieval problem. It’s a system design problem.

What Actually Works

The best RAG systems are:

simple
structured
observable
measurable

Not over-engineered.

Final Thought

If your system only works when:

the query is perfect
the data is clean
the demo is controlled

Then it doesn’t work.

What’s Next

Once RAG works, the next bottleneck is:

Cost.

Why LLM systems become expensive in production—and how to control it without killing performance.

Most AI Agent Frameworks Are Overkill — Here's How to Choose the Right One in 30 Seconds

TheProdSDE — Wed, 18 Mar 2026 12:58:03 +0000

A senior engineer's field-tested breakdown of LangGraph, AutoGen, CrewAI, Microsoft Agent Framework, and Haystack — from reviewing real production systems across teams.

Everyone is building AI agents right now.

LangGraph. AutoGen. CrewAI. Semantic Kernel. Microsoft Agent Framework.

But most production AI systems don't actually need an agent framework.

Across multiple teams and production codebases I've reviewed, the same two failure modes appear constantly — over-engineering and under-engineering. In one case, replacing a complex agent framework with ~200 lines of plain tool-calling code made the system 3× faster, easier to debug, and easier to maintain. In another, the absence of a framework caused a codebase to collapse under its own complexity.

Both failures had the same root cause:

The problem isn't choosing the wrong framework. It's choosing a framework before understanding the workflow.

This guide is the decision framework I now use before touching any agent tooling.

TL;DR — Pick Your Approach in 30 Seconds

Workflow Shape	Recommended Approach
Single request → tools → response	Plain tool calling (no framework)
Self-correcting loops, retries, re-evaluation	LangGraph
Parallel specialist agents	AutoGen 0.7.5
Enterprise persistent agents (Azure)	Microsoft Agent Framework RC
Sequential role-based task delegation	CrewAI 1.10.1
Document analysis and extraction pipelines	Haystack 2.x

Workflow Shapes at a Glance

Two Real Failure Modes I've Seen Across Teams

The over-engineering case.

During a cross-team architecture review, I found a financial data analysis pipeline — pull market metrics, cross-reference filings, produce a risk score — built with three microservices, a LangGraph orchestrator with twelve nodes, Redis for inter-agent memory, and a separate evaluation loop. Six weeks of engineering. It worked.

What it actually needed: 180 lines of FastAPI with direct OpenAI tool calling. Same output. 3× faster inference. Any engineer on the team could debug it in minutes.

After the review, the team simplified it together. The framework had been adopted before anyone mapped the workflow shape — a straight line — and LangGraph's graph model added cost with no return.

The under-engineering case.

In a separate review, I found the opposite problem. An infrastructure incident response system — Azure alerts, metric retrieval, remediation decisions, rollback logic, retry conditions, escalation thresholds — built with hand-rolled state machines, custom retry logic, and a bespoke tool orchestration layer. Three weeks in, the codebase was a maze. Every new remediation path required rewriting core routing logic.

A proper agent framework would have provided state management, conditional branching, retry handling, and checkpointing for free. Instead the team was reinventing those primitives by hand — the most expensive kind of technical debt.

The pattern is the same in both directions: the architecture was chosen before the workflow was understood.

The Core Question: Does Your Workflow Resist Simplicity?

Before touching any framework, draw the workflow on paper. Then answer these:

Does step N's output determine whether to redo step N-1? → You have a loop
Do multiple specialized agents need to run simultaneously? → You have parallelism
Does the workflow run for minutes or hours, surviving restarts? → You need persistent state
Does the agent need to decide its next action from intermediate results? → Dynamic planning
Do independent agents need to hand off context to each other? → Multi-agent delegation

None of these? Use plain tool calling. Here's what that looks like at production quality.

The Baseline: Plain Tool Calling (No Framework Needed)

Use case: Real-time ESG (Environmental, Social, Governance) risk scoring. Given a ticker, pull sustainability metrics, cross-reference regulatory filings, produce a risk-adjusted score, and persist an audit trail — one clean, observable service.

# requirements: fastapi, openai, psycopg2-binary, httpx, pydantic
from fastapi import FastAPI
from openai import OpenAI
from pydantic import BaseModel
import json, httpx, psycopg2, os

app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

class ESGRequest(BaseModel):
    ticker: str
    portfolio_id: str

def get_esg_metrics(ticker: str) -> dict:
    resp = httpx.get(
        f"https://api.sustainalytics.com/v1/esg/{ticker}",
        headers={"Authorization": f"Bearer {os.environ['ESG_API_KEY']}"},
        timeout=10
    )
    return resp.json()

def get_regulatory_flags(ticker: str) -> dict:
    with psycopg2.connect(os.environ["DATABASE_URL"]) as conn:
        with conn.cursor() as cur:
            cur.execute("""
                SELECT flag_type, severity, filing_date, description
                FROM regulatory_flags WHERE ticker = %s
                AND filing_date > NOW() - INTERVAL '2 years'
                ORDER BY severity DESC LIMIT 10
            """, (ticker,))
            rows = cur.fetchall()
    return {"flags": [{"type": r[0], "severity": r[1], "date": str(r[2]), "detail": r[3]} for r in rows]}

TOOLS = [
    {"type": "function", "function": {
        "name": "get_esg_metrics",
        "description": "Fetch ESG sustainability scores for a stock ticker",
        "parameters": {"type": "object", "properties": {"ticker": {"type": "string"}}, "required": ["ticker"]}
    }},
    {"type": "function", "function": {
        "name": "get_regulatory_flags",
        "description": "Retrieve regulatory violations and compliance flags",
        "parameters": {"type": "object", "properties": {"ticker": {"type": "string"}}, "required": ["ticker"]}
    }}
]

TOOL_MAP = {
    "get_esg_metrics": get_esg_metrics,
    "get_regulatory_flags": get_regulatory_flags
}

@app.post("/assess-esg")
async def assess_esg(req: ESGRequest):
    messages = [{"role": "user", "content": (
        f"Run a full ESG risk assessment for {req.ticker} in portfolio {req.portfolio_id}. "
        f"Produce a risk-adjusted score with recommendation: HOLD, REDUCE, or DIVEST."
    )}]
    while True:
        response = client.chat.completions.create(
            model="gpt-4o", messages=messages, tools=TOOLS, tool_choice="auto"
        )
        msg = response.choices[0].message
        if not msg.tool_calls:
            return {"ticker": req.ticker, "assessment": msg.content}
        messages.append(msg)
        for tc in msg.tool_calls:
            result = TOOL_MAP[tc.function.name](**json.loads(tc.function.arguments))
            messages.append({"role": "tool", "tool_call_id": tc.id, "content": json.dumps(result)})

Multi-step tool calling, audit trail, fully debuggable by any engineer. If this solves the problem — stop here. No framework needed.

1. LangGraph — Stateful Cyclic Workflows

Version: 1.1.2 | pip install langgraph langgraph-checkpoint-redis

Use when: Your workflow has genuine loops — the result of one step conditionally re-runs a previous step. Redis checkpointing lets state survive service restarts mid-workflow.

Avoid when: The workflow is strictly sequential with no conditional branching. The graph model adds measurable overhead for zero architectural return.

Use case: Automated cloud cost optimization — scan Azure VMs for underutilization, simulate right-sizing savings, apply low-risk changes, re-scan. The loop continues until no optimization exceeds the savings threshold.

# requirements: langgraph, langgraph-checkpoint-redis, openai
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.redis import RedisSaver
from typing import TypedDict, Annotated
import operator, json, os
from openai import OpenAI

oai = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

class CostState(TypedDict):
    subscription_id: str
    resources: list[dict]
    candidates: Annotated[list, operator.add]
    applied: Annotated[list, operator.add]
    total_savings: float
    iteration: int

def scan_resources(state: CostState) -> dict:
    resources = azure_monitor_client.list_resources(state["subscription_id"])
    underutilized = [r for r in resources if r["avg_cpu_7d"] < 15 and r["avg_memory_7d"] < 30]
    return {"resources": underutilized, "iteration": state["iteration"] + 1}

def analyze_savings(state: CostState) -> dict:
    resp = oai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"""
            Analyze these underutilized Azure resources and recommend right-sizing.
            For each: resource_id, current_sku, recommended_sku, monthly_savings_usd, risk_level (LOW/MEDIUM/HIGH).
            Only include LOW or MEDIUM risk recommendations.
            Resources: {json.dumps(state["resources"])}
            Respond: {{"recommendations": [...]}}
        """}],
        response_format={"type": "json_object"}
    )
    candidates = json.loads(resp.choices[0].message.content)["recommendations"]
    return {"candidates": candidates, "total_savings": sum(c["monthly_savings_usd"] for c in candidates)}

def apply_changes(state: CostState) -> dict:
    applied = []
    for c in state["candidates"]:
        if c["risk_level"] == "LOW":
            azure_compute.resize_vm(c["resource_id"], c["recommended_sku"])
            applied.append(c)
    return {"applied": applied}

def check_threshold(state: CostState) -> str:
    return "loop" if state["total_savings"] > 500 and state["iteration"] < 5 else "done"

graph = StateGraph(CostState)
graph.add_node("scan", scan_resources)
graph.add_node("analyze", analyze_savings)
graph.add_node("apply", apply_changes)
graph.set_entry_point("scan")
graph.add_edge("scan", "analyze")
graph.add_edge("analyze", "apply")
graph.add_conditional_edges("apply", check_threshold, {"loop": "scan", "done": END})

optimizer = graph.compile(checkpointer=RedisSaver.from_conn_string(os.environ["REDIS_URL"]))
result = optimizer.invoke({
    "subscription_id": os.environ["AZURE_SUBSCRIPTION_ID"],
    "candidates": [], "applied": [], "total_savings": 0.0, "iteration": 0
})

Why this justifies LangGraph: The self-correcting re-scan is genuinely hard to express cleanly in plain tool calling without writing a custom state machine — which is exactly the under-engineering trap described earlier.

2. AutoGen 0.7.5 — Parallel Multi-Agent Collaboration

Version: 0.7.5 | pip install "autogen-agentchat>=0.7.5" "autogen-ext[openai,redis]"

Key additions in 0.7.5: linear memory via RedisMemory, fixed GraphFlow cycle detection, Anthropic thinking mode support, reasoning_effort parameter for GPT-5 models, improved Azure AI client streaming.

Use when: Multiple specialized agents run independent analysis simultaneously. Async, event-driven — agents can be deployed on separate containers with zero blocking between them.

Avoid when: The task is sequential and single-agent. Multi-agent coordination overhead only pays off when genuine parallelism exists.

Use case: Automated M&A due diligence — Legal, Financial, and Tech Audit agents work in parallel, then a Synthesis agent consolidates findings into an investment decision. Sequential execution would add hours of unnecessary latency to a time-sensitive deal process.

# requirements: autogen-agentchat>=0.7.5, autogen-ext[openai,redis]
import asyncio, os
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import SelectorGroupChat
from autogen_agentchat.conditions import TextMentionTermination
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_ext.memory.redis import RedisMemory  

model_client = OpenAIChatCompletionClient(model="gpt-4o", api_key=os.environ["OPENAI_API_KEY"])

legal_agent = AssistantAgent(
    name="LegalAgent",
    model_client=model_client,
    memory=[RedisMemory(redis_url=os.environ["REDIS_URL"], session_id="ma-legal")],
    system_message="""M&A legal counsel. Identify: IP gaps, change-of-control clauses,
    litigation exposure, employment liabilities.
    Output JSON: {risk_items, severity: BLOCKER|HIGH|MEDIUM|LOW, deal_impact}""",
    tools=[fetch_contract_repository, search_litigation_database]
)

financial_agent = AssistantAgent(
    name="FinancialAgent",
    model_client=model_client,
    memory=[RedisMemory(redis_url=os.environ["REDIS_URL"], session_id="ma-financial")],
    system_message="""M&A financial analyst. Identify: revenue quality, off-balance-sheet
    liabilities, working capital needs post-acquisition, EBITDA normalization.
    Output JSON: {financial_flags, normalized_ebitda, recommended_valuation_range}""",
    tools=[fetch_financial_statements, compute_dcf_model]
)

tech_audit_agent = AssistantAgent(
    name="TechAuditAgent",
    model_client=model_client,
    system_message="""Technical due diligence expert. Assess: tech debt score (1–10),
    security vulnerabilities, scalability ceiling, bus-factor risk.
    Output JSON: {tech_risks, estimated_remediation_cost_usd, integration_complexity}""",
    tools=[clone_and_analyze_repo, run_dependency_scan]
)

synthesis_agent = AssistantAgent(
    name="SynthesisAgent",
    model_client=model_client,
    system_message="""M&A deal lead. Wait for all domain agents to complete.
    Synthesize: PROCEED / PROCEED_WITH_CONDITIONS / ABORT.
    Include: top 5 risks, price adjustment recommendation, 90-day priorities.
    End your message with: ANALYSIS_COMPLETE""",
    tools=[generate_pdf_memo, notify_deal_team_slack]
)

async def run_ma_due_diligence(target: str):
    team = SelectorGroupChat(
        participants=[legal_agent, financial_agent, tech_audit_agent, synthesis_agent],
        model_client=model_client,
        termination_condition=TextMentionTermination("ANALYSIS_COMPLETE"),
        selector_prompt="""Run LegalAgent, FinancialAgent, TechAuditAgent first (order flexible,
        can run in parallel). Only select SynthesisAgent after all three have reported."""
    )
    async for msg in team.run_stream(task=f"Full M&A due diligence for: {target}"):
        print(f"[{msg.source}] {str(msg.content)[:120]}...")

asyncio.run(run_ma_due_diligence("TargetCorp Inc."))

3. Microsoft Agent Framework RC — Enterprise Production Agents

Version: Release Candidate, Feb 19, 2026 | pip install agent-framework --pre

The most significant shift in the Microsoft AI ecosystem right now. This framework unifies Semantic Kernel and AutoGen into a single SDK — both are entering maintenance mode as of this writing, and all new Microsoft investment flows into Agent Framework first.

RC signals a frozen, stable API surface with GA targeting Q1 2026. Core capabilities: persistent threads (Cosmos DB), Service Bus integration, MCP + A2A protocol support, multi-agent orchestration with handoff and group chat patterns, streaming checkpointing for long-running agents, full .NET and Python support.

Use when: Azure-first enterprise teams, production 24/7 background agents, regulated systems requiring full audit trails, or any team currently building on Semantic Kernel or AutoGen.

Avoid when: Greenfield Python-only stacks with no Azure dependency, or where GA stability is a hard requirement before adoption.

Use case: Autonomous DevOps deployment agent — monitors CI/CD completion events, validates health via Application Insights, promotes through dev → staging → prod automatically, pages on-call only when a gate fails. Runs continuously as a persistent, resumable agent.

# requirements: agent-framework --pre, azure-identity, azure-monitor-query, kubernetes
import os
from agent_framework import AgentClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
agent_client = AgentClient(
    endpoint=os.environ["AZURE_AI_FOUNDRY_ENDPOINT"],
    credential=credential
)

def check_app_insights(resource_id: str, environment: str, window_minutes: int = 5) -> dict:
    from azure.monitor.query import MetricsQueryClient
    from datetime import timedelta
    monitor = MetricsQueryClient(credential)
    result = monitor.query_resource(
        resource_id,
        metrics=["requests/failed", "requests/duration"],
        timespan=timedelta(minutes=window_minutes)
    )
    return {
        "environment": environment,
        "error_rate_percent": _calculate_error_rate(result),
        "p99_latency_ms": _calculate_p99(result),
        "gate_passed": _calculate_error_rate(result) < 1.0 and _calculate_p99(result) < 500
    }

def promote_deployment(service: str, image_tag: str, environment: str) -> dict:
    from kubernetes import client as k8s, config
    config.load_incluster_config()
    k8s.AppsV1Api().patch_namespaced_deployment(
        name=service, namespace=environment,
        body={"spec": {"template": {"spec": {"containers": [{
            "name": service,
            "image": f"{os.environ['ACR_REGISTRY']}/{service}:{image_tag}"
        }]}}}}
    )
    return {"status": "promoted", "service": service, "tag": image_tag, "environment": environment}

# Persistent thread — Cosmos DB preserves state across restarts
agent = agent_client.create_agent(
    model="gpt-4o",
    name="DeploymentOrchestrator",
    instructions="""Autonomous DevOps deployment agent.
    1. Validate health: error rate < 1%, p99 < 500ms, zero pod restarts
    2. Run smoke tests on the deployed environment
    3. All gates pass → promote to next environment
    4. Any gate fails → halt, capture full diagnostics, page on-call
    5. After prod deploy: monitor 15 minutes, auto-rollback if error rate > 2%
    Log every action: timestamp, metric values, decision, outcome.""",
    tools=[check_app_insights, promote_deployment, run_smoke_tests,
           rollback_deployment, page_oncall, log_deployment_event]
)
thread = agent_client.create_thread()

async def handle_pipeline_completion(event: dict):
    """Azure Event Grid webhook — fires on every pipeline completion"""
    agent_client.create_message(
        thread_id=thread.id, role="user",
        content=(
            f"Deployment complete — Service: {event['service_name']}, "
            f"Tag: {event['image_tag']}, Environment: {event['environment']}. "
            f"Begin validation and promotion workflow."
        )
    )
    return agent_client.create_and_process_run(thread_id=thread.id, agent_id=agent.id)

4. CrewAI 1.10.1 — Role-Based Sequential Pipelines

Version: 1.10.1 (March 3, 2026) | pip install crewai==1.10.1

⚠️ Note: v1.10.0 was yanked from PyPI due to an AMP runtime issue. Always pin to 1.10.1 directly. This release lazy loading in Memory module and resolves a concurrent multi-process LockException in production flows.

Use when: Clearly defined specialist roles, sequential task handoff, and fastest path from idea to working prototype. CrewAI is the most opinionated framework and the fastest to build with.

Avoid when: Fine-grained conditional edge control is needed, or throughput-sensitive production paths where abstraction overhead is measurable.

Use case: Automated system architecture pipeline — an Architect designs from requirements, a Reviewer stress-tests for production failures, a Documentation Lead produces the Architecture Decision Record for the engineering wiki.

# requirements: crewai==1.10.1
from crewai import Agent, Task, Crew, Process

architect = Agent(
    role="Principal Solutions Architect",
    goal="Design a scalable, cloud-native system architecture from requirements",
    backstory="10+ years on Azure and AWS. Expert in event-driven architecture, CQRS, and zero-trust.",
    verbose=True
)
reviewer = Agent(
    role="Staff Engineer — Technical Reviewer",
    goal="Identify scalability bottlenecks, security vulnerabilities, and operational risks",
    backstory="Former SRE. Has seen every production failure mode. Only approves designs that hold at 10x load.",
    verbose=True
)
doc_lead = Agent(
    role="Technical Documentation Lead",
    goal="Produce a complete Architecture Decision Record capturing every design trade-off",
    backstory="Undocumented architecture is a liability. Writes for the engineer joining 2 years later.",
    verbose=True
)

design_task = Task(
    description="""Design a complete architecture for: {requirements}
    Output JSON: components (name + responsibility), data_flows (source → destination),
    tech_choices (with justification), scaling_strategy per component, security_controls.""",
    expected_output="JSON architecture specification",
    agent=architect
)
review_task = Task(
    description="""Stress-test the architecture:
    1. 10x traffic spike — what breaks first?
    2. Single region failure — what is the blast radius?
    3. Compromised service account — what can an attacker reach?
    Return: {approved: YES/NO, blocking_issues: [], recommended_changes: []}""",
    expected_output="Review JSON with approval and findings",
    agent=reviewer,
    context=[design_task]
)
adr_task = Task(
    description="Write a complete ADR: Context, Decision, Consequences, Alternatives Considered, Risk Register. Format: Markdown.",
    expected_output="Complete ADR in Markdown",
    agent=doc_lead,
    context=[design_task, review_task]
)

crew = Crew(
    agents=[architect, reviewer, doc_lead],
    tasks=[design_task, review_task, adr_task],
    process=Process.sequential,
    verbose=True
)
result = crew.kickoff(inputs={
    "requirements": "Real-time fraud detection — 50K TPS, sub-100ms decisions, 99.99% uptime, multi-region active-active"
})
print(result.raw)

5. Haystack 2.x — Document Intelligence Pipelines

Version: 2.x | pip install haystack-ai

Use when: Document processing, extraction, or compliance analysis at scale is the core product — not general agentic behavior. Purpose-built for this and measurably outperforms general frameworks on document-centric workloads.

Avoid when: The problem is general agentic orchestration, multi-agent coordination, or any real-time interactive system.

Use case: Automated SOC 2 evidence collection — ingests all internal policy documents, maps clauses against Trust Service Criteria, produces a compliance gap report showing which controls are missing or non-conformant.

The pipeline maps extracted content against all six SOC 2 Trust Service Criteria (CC6, CC7, CC8, CC9, A1, C1), flags NOT_COVERED gaps with descriptions, and returns a structured compliance report with overall coverage percentage — no custom parsing layer required.

The Practical Decision Rule

If your AI system needs:

  Retries or self-correction    → LangGraph
  Multiple parallel specialists → AutoGen
  Enterprise Azure persistence  → Microsoft Agent Framework
  Sequential role-based tasks   → CrewAI
  Document extraction at scale  → Haystack
  None of the above             → No framework. Plain tool calling.

Framework Reference

Framework	Version	Primary Strength	Use When	Avoid When
Plain tool calling	—	Speed, simplicity, debuggability	Straight-line workflow	Never — if no loops exist
LangGraph	1.1.2	Cyclic graphs, checkpointing	Self-correcting loops, retries	No conditional branching
AutoGen	0.7.5	Parallel async agents, RedisMemory	Multiple specialists in parallel	Sequential single-agent tasks
Microsoft Agent Framework	RC Feb 2026	Enterprise persistence, SK + AutoGen unified	Azure production 24/7 agents	Python-only, no Azure stack
CrewAI	1.10.1	Role-based prototyping, fast iteration	Sequential delegation, fast prototyping	Fine-grained production control
Haystack	2.x	Document extraction pipelines	Document processing as core product	General agentic tasks

Production Lessons From Real Systems

After reviewing AI systems across multiple teams, a few patterns appear consistently:

1. Most workflows are simpler than they look.
The instinct when building with LLMs is to reach for orchestration layers. That instinct is usually wrong. Start with the simplest thing that could work, measure it, then add complexity only where the problem resists simplicity.

2. Agent frameworks pay off only when the workflow has the right shape.
Loops, parallel specialists, or long-running persistent state. If none of these exist, the framework is cost with no architectural return.

3. Debuggability matters more than clever architecture.
A system the team can debug at 2AM is worth more than an elegant multi-agent pipeline nobody fully understands. Production incidents don't wait for framework comprehension.

4. Hand-rolling framework primitives is the most expensive mistake.
Custom state machines, retry logic, and checkpointing written to avoid a framework dependency consistently cost more engineering time than learning the framework properly. Both failure modes described earlier confirm this.

5. Decide the architecture before writing the first line.
Draw the workflow. Identify the shape. Pick the tool. That decision should take 30 minutes, not six weeks of refactoring.

The Real Rule

If you can draw your workflow as a straight line — plain tool calling is your production architecture.
If that line needs to loop, branch, or coordinate parallel specialists — match the tool to the shape.

The best production AI systems are architecturally boring. One clean service, typed tool schemas, structured output, observable logs. No framework overhead unless the problem demands it.

Add complexity only when the problem resists simplicity. That's the whole framework.

What's Next in This Series

This post focused on orchestration — how to structure and run AI workflows in production.

But once orchestration is solved, most teams run into a different problem:

Their RAG system doesn’t actually work.

Not in demos — those look fine.
In production — it breaks.

Wrong answers. Missing context. Hallucinations with high confidence.

And in most cases, the root cause is not the vector database or the embedding model.

It’s the architecture around it.

The next article breaks down:

Why most RAG systems fail after launch
The common design mistakes teams repeat
And the production architecture that actually fixes it

If you’re building anything on top of retrieval, this is where things either scale — or quietly fail.

Are you running an agent framework in production — or did you strip one out and go back to basics?

What made that decision clear? Drop your stack and the reason in the comments. The real production stories are always more useful than the official docs.

Tags: #ai #llm #machinelearning #softwareengineering #devops

Version data sourced from official release channels: AutoGen 0.7.5 , Microsoft Agent Framework RC, CrewAI 1.10.1, LangGraph, Haystack

[Boost]

TheProdSDE — Wed, 18 Mar 2026 09:37:06 +0000

Stop Guessing Your LLM Replacement

TheProdSDE ・ Mar 14

#ai #llm #cloud #systemdesign

Stop Guessing Your LLM Replacement

TheProdSDE — Sat, 14 Mar 2026 11:32:29 +0000

A Practical Guide to Migrating GPT Apps Across Azure, AWS, and GCP

TL;DR — Most LLM migrations are not caused by model performance. They happen because of data residency laws, enterprise deployment requirements, or cloud standardisation decisions. This guide helps you narrow the search space to the right replacement candidates — not replace real testing.

Why Companies Actually Migrate LLMs

Engineers rarely wake up and decide to migrate their AI stack. Most migrations are triggered by business constraints, not technical ones.

Common Migration Scenarios

1️⃣ Expanding into Regions With Data Residency Laws

Your product currently runs on Azure OpenAI (gpt-4o-mini), but a new region requires:

EU sovereign cloud
Local data processing
Provider-specific compliance certifications

You may need to move to AWS Bedrock, Google Vertex AI, or Azure AI Foundry with open models — even though your application logic stays identical.

2️⃣ Enterprise Customers Want AI Inside Their Environment

This is extremely common in B2B SaaS. Enterprise customers often require:

Private cloud deployment
VPC-only access
On-prem inference
Sovereign cloud environments

Your API-based model suddenly needs to become Llama, Qwen, DeepSeek, or Mistral — running inside their infrastructure.

3️⃣ Corporate Cloud Standardisation

A classic scenario:

AI team → Azure
Platform team → AWS

Leadership decides: "All workloads must run on AWS."

Now your team must translate gpt-4o-mini into an AWS Bedrock equivalent — and the model catalog doesn't make that obvious.

The Problem: Model Names Don't Translate

Each vendor uses completely different naming schemes. There is no official cross-provider model map.

Vendor	Entry Model	Mid Model	Reasoning Model
OpenAI	`gpt-4o-mini`	`gpt-4o`	o-series
Anthropic	Haiku	Sonnet	Opus
Google	Flash	Pro	Pro reasoning
Meta	Scout	Maverick	Large variants
Qwen	Small (7B–14B)	72B–110B	235B Thinking
DeepSeek	V3 (non-thinking)	V3 standard	R1 (reasoning)

Common mistakes teams make

❌ Picking the cheapest model in the new catalog
❌ Picking the newest model by release date
❌ Picking the highest benchmark model regardless of tier

All three approaches can silently break production behaviour.

The Tier Model That Actually Works

Instead of comparing names, compare capability tiers. Every major provider follows the same five-tier structure.

Tier	Typical Use Cases	Latency	Relative Cost
Mini / Flash / Small	Chatbots, RAG, classification	Fastest	Lowest
Standard / Mid	Assistants, summarisation, coding	Medium	Moderate
Reasoning / Pro	Agents, planning, complex Q&A	Slower	Higher
Frontier / Flagship	Research workloads, safety-critical	Slowest	Highest

Once you know your current model's tier, finding candidates becomes systematic — not guesswork.

Tier 1 — Replacing `gpt-4o-mini`

Typical workloads: chat assistants, RAG pipelines, tool calling, lightweight coding

Candidates by cloud

Cloud	Replacement Candidates
Azure AI Foundry	Llama 4 Scout, Qwen3-8B/14B, DeepSeek V3, Claude Haiku
AWS Bedrock	Claude Haiku, Mistral Small
GCP Vertex AI	Gemini Flash-Lite, Gemini Flash

Behaviour differences at this tier

Model	Strengths	Watch out for
Claude Haiku	Reliable, low hallucination rate	~7× more expensive than `gpt-4o-mini`
Gemini Flash	Extremely fast, 1M token context	GCP-only; not available on Azure
Llama 4 Scout	Open-weight, 10M token context, Azure-hosted	Not a pure reasoning-tuned model
DeepSeek V3	Unusually strong reasoning (MMLU-Pro ~75.9, GPQA ~59.1) for this price tier	Direct API or Azure Foundry; no native AWS/GCP
Qwen3-8B/14B	Strong multilingual + math, Apache 2.0	Smaller context than Gemini/Llama

Tier 2 — Replacing `gpt-4o`

Typical workloads: document summarisation, coding assistance, enterprise chat assistants

Candidates by cloud

Cloud	Replacement Candidates
Azure AI Foundry	Claude Sonnet, Llama 4 Maverick, DeepSeek V3
AWS Bedrock	Claude Sonnet, Mistral Medium
GCP Vertex AI	Gemini Flash, Gemini 2.5 Pro

Benchmark reference (reasoning quality)

Model	MMLU	MMLU-Pro	GPQA-Diamond	Notes
Claude Sonnet 4.x	~88+	Strong	Competitive	Best SWE-bench coding score at this tier
Llama 4 Maverick	~85+	~80.5	~69.8	Beats GPT-4o on Meta's coding benchmarks
DeepSeek V3	88.5	75.9–81.2	59.1–68.4	Frontier-class at mid-tier pricing
Gemini Flash (GCP)	Strong	Competitive	~78% SWE-bench	GCP-only; fastest in this tier

DeepSeek V3 on Azure often outperforms gpt-4o on raw reasoning benchmarks at significantly lower cost. Treat it as a tier upgrade, not just a replacement.

Tier 3 — Replacing Reasoning Models (`gpt-4.1` / `gpt-5` / o-series)

Typical workloads: agent systems, research workflows, complex multi-step reasoning

Candidates by cloud

Cloud	Replacement Candidates
Azure AI Foundry	Claude Opus, DeepSeek R1, Qwen3-235B Thinking (via Foundry)
AWS Bedrock	Claude Opus
GCP Vertex AI	Gemini 2.5 Pro, Gemini 3.1 Pro

Reasoning benchmark reference (HLE + advanced)

Model	HLE Score	MMLU-Pro	GPQA-Diamond	AIME-25
Claude Opus 4.x	Top-tier (Anthropic reports #1 on HLE leaderboard)	~90+	Strong	Strong
Qwen3-235B-A22B Thinking	~18% (one of few published open-weight HLE scores)	~84.4%	~81%	~92%
DeepSeek R1	Not widely published	~81.2	~68.4	Strong
Gemini 2.5 / 3.1 Pro	Competitive	Strong	Strong	Strong

Qwen3-235B-A22B Thinking is currently one of the few open-weight models with a published Humanity's Last Exam score (~18%) — putting it in the same conversation as frontier closed models for reasoning-heavy tasks.

Architecture Pattern: Make LLMs Replaceable

The biggest mistake teams make is hard-coding a model into their architecture.

❌ The fragile pattern

Application → GPT-4o-mini (hardcoded)

Any migration requires touching application logic, service config, and prompt templates.

✅ The replaceable pattern

Benefits:

Vendor independence — swap providers via config
Easier model upgrades without app rewrites
Enables cost-optimised routing

Cost Optimisation: Intelligent Request Routing

Many production AI systems at scale route requests by task complexity rather than using one model for everything.

This pattern can reduce LLM costs by 60–90% in workloads with a mix of simple and complex requests.

Prompt Regression Testing — Non-Negotiable

Before committing to any model swap, run prompt regression tests on your real production prompts.

# Simple regression harness
test_prompts = load_production_samples(n=200)

results = []
for prompt in test_prompts:
    output_a = old_model.invoke(prompt)
    output_b = new_model.invoke(prompt)

    results.append({
        "prompt": prompt,
        "score_a": evaluate(output_a),   # correctness, format, hallucination
        "score_b": evaluate(output_b),
        "regression": score_b < score_a * 0.95
    })

regressions = [r for r in results if r["regression"]]
print(f"Regression rate: {len(regressions)/len(results)*100:.1f}%")

Check for:

Correctness — does the answer change?
Format compliance — does it still follow your output structure?
Hallucination rate — does it fabricate facts?
Latency — does it still meet your SLA?

⚠️ Important Disclaimer

The mappings in this guide are a starting point, not guaranteed drop-in replacements.

Models in the same tier can have meaningfully different behaviour on:

Your specific domain vocabulary
Your prompt style
Edge cases in your data

Every migration must include:

Prompt regression testing on real data
Human evaluation of sampled outputs
Shadow traffic validation (run both models in parallel, compare outputs)
Gradual rollout (5% → 25% → 100%)

Recommended Migration Workflow

Do not skip shadow traffic. It catches subtle regressions that prompt tests miss.

📋 LLM Migration Cheat Sheet

Mini Tier (`gpt-4o-mini` equivalent)

Azure → Llama 4 Scout, Qwen3-8B/14B, DeepSeek V3, Claude Haiku
AWS → Claude Haiku, Mistral Small
GCP → Gemini Flash-Lite, Gemini Flash

Standard Tier (`gpt-4o` equivalent)

Azure → Claude Sonnet, Llama 4 Maverick, DeepSeek V3
AWS → Claude Sonnet, Mistral Medium
GCP → Gemini Flash, Gemini 2.5 Pro

Reasoning Tier (o-series / `gpt-5` equivalent)

Azure → Claude Opus, DeepSeek R1, Qwen3-235B Thinking
AWS → Claude Opus
GCP → Gemini 2.5 Pro, Gemini 3.1 Pro

Final Takeaway

The LLM ecosystem is evolving too fast to depend on a single provider.

Design your systems so that:

GPT → Claude → Gemini → DeepSeek

…is a configuration change, not a system rewrite.

When that happens, migrations become boring infrastructure work.

And boring infrastructure is exactly what you want in production.

Found this useful? Follow TheProdSDE for more practical engineering guides on AI systems, cloud architecture, and developer tooling.

Tags: #ai #llm #cloud #azure #systemdesign

Worth your 10min of time if deciding which framework suits your use case better

TheProdSDE — Fri, 13 Mar 2026 14:48:04 +0000

LangGraph vs Semantic Kernel: Python AI Agents in 2026

TheProdSDE ・ Mar 11

#python #ai #agents #langchain

[Boost]

TheProdSDE — Fri, 13 Mar 2026 14:47:02 +0000

7 AI Prompts Senior Engineers Use to Build Production Code Faster

TheProdSDE ・ Mar 13

#ai #programming #productivity #software

7 AI Prompts Senior Engineers Use to Build Production Code Faster

TheProdSDE — Fri, 13 Mar 2026 13:15:10 +0000

AI coding tools are powerful — but most developers still use them like autocomplete.
The difference between average AI usage and 10x productivity often comes down to one thing:
how you prompt the system.

In a previous article, I explained why teams should stop treating AI like a magic architect and instead treat it like a fast junior engineer inside a well-designed system.

But that raises an important question:

How do experienced engineers actually prompt AI tools in real production workflows?

Below are 7 practical prompts senior engineers use to guide AI coding assistants toward reliable, production-quality output.

What You'll Learn

How senior engineers structure prompts for AI coding tools
Why constraints improve AI-generated code
Practical prompts you can reuse in real projects
How to avoid common AI development mistakes

Prompt 1 — Spec → Implementation

One of the most effective patterns is defining the specification first.

Instead of asking the AI to build a feature from scratch, you provide clear constraints and structure.

Example prompt:

You are a senior backend engineer.

Implement the following interface in TypeScript.

Constraints:
- Follow the existing repository pattern
- Do not introduce new dependencies
- Use the existing error handling system
- Write clean, maintainable code

Interface:
[PASTE INTERFACE HERE]

Why this works:

The engineer controls architecture
AI performs mechanical implementation
Code stays consistent with the existing system

Prompt 2 — Safe Refactoring

Refactoring is one of the safest and most valuable uses of AI coding tools.

Example prompt:

Refactor this service to extract the email logic into a NotificationService.

Constraints:
- Do not change public APIs
- Preserve existing behavior
- Maintain current tests
- Avoid introducing circular dependencies

Benefits:

Faster multi-file refactoring
Lower risk of manual mistakes
Improved code organization

AI becomes a high-speed refactoring assistant.

Prompt 3 — Test Generation

Writing test scaffolding is another task where AI shines.

Example prompt:

Generate Jest unit tests for this service.

Constraints:
- Cover success and failure cases
- Mock external dependencies
- Follow our existing test structure
- Focus on behavior rather than implementation details

AI can generate:

basic test structures
mocks
scenario coverage

You still review tests, but the initial draft appears instantly.

Prompt 4 — Code Review Assistant

AI can act as a secondary reviewer for pull requests.

Example prompt:

Review the following code as a senior engineer.

Focus on:
- architecture issues
- potential bugs
- performance concerns
- unnecessary complexity
- security risks

This helps engineers quickly surface:

hidden edge cases
inefficient patterns
potential race conditions

It does not replace human review, but it improves signal detection.

Prompt 5 — Documentation Generation

Documentation is often neglected in fast-moving teams.

AI can generate a first draft instantly.

Example prompt:

Generate documentation for this API endpoint.

Include:
- request format
- response format
- error conditions
- example usage
- explanation of business logic

Engineers then refine the output instead of writing everything from scratch.

Prompt 6 — Guided Multi-File Refactors

Large refactors across many files are tedious and error-prone.

AI can help coordinate them.

Example prompt:

Find all usages of sendEmail across the repository.

Refactor them to use NotificationService.

Constraints:
- Preserve existing behavior
- Do not modify unrelated logic
- Update imports if necessary

AI tools integrated with IDEs can perform these changes much faster than manual edits.

Prompt 7 — Architecture Sanity Check

One of the most useful prompts is asking AI to analyze architectural risks.

Example prompt:

Analyze this module for potential architecture issues.

Look for:
- tight coupling
- duplicated logic
- boundary violations
- hidden dependencies

This works surprisingly well when reviewing complex modules.

AI often catches patterns that are easy to miss during routine coding.

Why Constraints Make AI Better

A common mistake developers make is using vague prompts.

Example of a weak prompt:

Build authentication for my app

The AI must guess too much.

Better prompts:

define constraints
reference existing patterns
specify architecture boundaries

The clearer the prompt, the more reliable the output.

The Key Insight

AI coding tools are not magic architects.

They are execution engines.

Senior engineers do not ask AI to design systems.

They ask AI to implement clearly defined pieces of a system.

Final Thoughts

AI coding assistants are becoming a permanent part of the developer toolbox.

But the biggest productivity gains do not come from the tool itself.

They come from learning how to guide the tool effectively.

The engineers who benefit most from AI are not the ones who rely on it blindly.

They are the ones who combine strong engineering discipline with precise prompts.

And in that workflow, AI becomes less like autocomplete…

…and more like a very fast junior engineer.

Written by TheProdSDE — sharing insights on AI engineering, system design, and developer productivity.

Stop Treating AI Like Autocomplete — Design an AI-First Developer Workflow Instead

TheProdSDE — Thu, 12 Mar 2026 13:51:28 +0000

AI coding tools are everywhere — but most teams still use them like autocomplete.
The result? Faster code generation… and faster technical debt.

In 2026, almost every developer works with AI tools inside their IDE.

You type a function name.
Your AI assistant confidently generates 40 lines of code.

You accept it because the deadline is tight.

Two weeks later your team is debugging the same AI-generated code.

The issue isn't AI.

The issue is the workflow around it.

The teams getting massive productivity gains are not the ones simply using AI tools.

They are the ones engineering a workflow around AI.

💬 Quick question for you:

How are you currently using AI coding tools?

autocomplete replacement
debugging helper
architecture assistant
full feature generation

Drop your workflow in the comments — I'm curious how different teams are approaching this.

What You'll Learn

Why AI coding tools create hidden technical debt
The correct mental model for AI-assisted development
A practical AI-first development workflow
Real engineering patterns that work

* Rules for keeping humans in control

TL;DR

Most teams use AI coding tools like autocomplete.

That works for small tasks — but breaks down in real systems.

The better workflow:

Engineers design the architecture
AI generates implementation
Humans review the output
Tests validate behavior

AI shouldn't design your system.

It should accelerate implementation inside a system you already designed.

The Real Problem

AI coding assistants are now part of everyday development.

Developers use them to:

generate boilerplate
explain unfamiliar code
write tests
refactor functions
scaffold APIs

But problems appear when teams move from:

"AI helps write code"

"AI probably knows what the system should do."

That shift introduces hidden risk.

What AI Is Very Good At

Generating repetitive code
Translating intent into a first draft
Creating test scaffolding
Performing large refactors
Accelerating low-risk implementation

What AI Is Still Bad At

Understanding business rules
Making architectural trade-offs
Preserving system boundaries
Long-term maintainability
Handling ambiguity safely

AI doesn't remove engineering discipline.
It amplifies whatever discipline already exists in the team.

If the workflow is weak, AI accelerates bad architecture.

If the workflow is strong, AI accelerates productivity.

The Correct Mental Model

The best way to think about AI coding tools is simple:

Treat AI like a fast junior engineer inside your system.

A good junior engineer can:

implement well-defined tasks
write boilerplate
perform refactors
generate first drafts

But they still require:

clear requirements
architecture constraints
code review

AI works exactly the same way.

This workflow works especially well with AI-enabled IDEs like Cursor or coding assistants like GitHub Copilot.

A healthy AI development workflow looks like this:

Problem Definition
      ↓
Interface Design
      ↓
AI Implementation
      ↓
Human Code Review
      ↓
Testing & Validation
      ↓
Merge

Humans control architecture.

AI accelerates execution.

The Most Dangerous AI Workflow

Many teams accidentally follow this workflow:

Open IDE
Ask AI to build a feature
Accept generated code
Ship it

It feels fast.

But it introduces:

hidden coupling
duplicated logic
architecture violations
security risks

Speed without discipline is technical debt at machine speed.

A Practical AI-First Workflow

A better engineering model looks like this:

Define the problem clearly
Design interfaces first
Ask AI to implement inside constraints
Review like a pull request
Standardize repeatable prompts

Human-Owned Work

Engineers must still own:

system architecture
service boundaries
security decisions
data contracts
performance trade-offs
business rules

AI-Assisted Work

AI works best for:

boilerplate code
test scaffolding
repetitive refactors
documentation drafts
DTO and mapper generation
CRUD scaffolding

AI executes inside the system you designed.

Practical Pattern #1 — Spec → Design → AI Implementation

Weak prompt:

Build authentication for my app

Too vague.

Better workflow:

Step 1 — Lightweight spec

Feature: Email/password login

POST /api/auth/login
Request: { email, password }

Response:
{
  accessToken,
  refreshToken,
  user: { id, email, roles }
}

Step 2 — Define interface

export interface AuthService {
  login(email: string, password: string): Promise<LoginResult>;
  refresh(token: string): Promise<LoginResult>;
}

export interface LoginResult {
  accessToken: string;
  refreshToken: string;
  user: {
    id: string;
    email: string;
    roles: string[];
  };
}

Step 3 — AI implementation

Prompt:

Implement AuthService using UserRepository, bcrypt password checks, and the existing JWT helper.

Now:

You control the contract
AI handles mechanical work
Architecture remains stable

Practical Pattern #2 — Guided Refactors

Example: centralizing email logic behind a service.

Define abstraction first:

export interface NotificationService {
  sendWelcomeEmail(userId: string): Promise<void>;
  sendPasswordResetEmail(userId: string, resetToken: string): Promise<void>;
}

Then instruct AI:

Find direct sendEmail calls and route them through NotificationService.

Now the AI performs the mechanical refactor while architecture stays consistent.

Practical Pattern #3 — AI First Draft, Human Final Draft

AI can generate boilerplate quickly.

Example:

export function toUserResponse(entity: UserEntity): UserResponse {
  return {
    id: entity.id,
    email: entity.email,
    roles: entity.roles,
    createdAt: entity.createdAt.toISOString(),
    updatedAt: entity.updatedAt.toISOString(),
  };
}

Human refinement:

import { formatIsoDate } from "../utils/date";

export function toUserResponse(entity: UserEntity): UserResponse {
  return {
    id: entity.id,
    email: entity.email,
    roles: [...entity.roles],
    createdAt: formatIsoDate(entity.createdAt),
    updatedAt: formatIsoDate(entity.updatedAt),
  };
}

AI saves typing.

Engineers enforce consistency.

Practical Pattern #4 — AI as Test Author

Define test strategy first.

// Scenarios:
// valid credentials → success
// invalid email → AUTH_INVALID_CREDENTIALS
// invalid password → AUTH_INVALID_CREDENTIALS
// locked account → AUTH_ACCOUNT_LOCKED

Then prompt AI:

Generate Jest tests covering these scenarios.

You then review for:

missing edge cases
flaky tests
incorrect assumptions

The AI Development Rules I Follow

Simple rules that dramatically improve outcomes:

Never generate architecture with AI
Always define interfaces first
Constrain AI to specific files
Review AI code like junior code
Never merge AI code without tests
Standardize prompts for repeatable workflows

These guardrails keep velocity high without sacrificing quality.

Real Tools Supporting This Workflow

Modern tools supporting this approach include:

GitHub Copilot
Claude Code
Cursor

These tools are powerful — but the workflow around them matters more than the tool itself.

One Insight That Changes Everything

AI does not remove engineering responsibility.

It amplifies the engineering discipline already present in the team.

Bad workflows produce bad code faster.

Good workflows produce good code faster.

The difference is not the AI.

The difference is the system around it.

Before adopting AI agents across your team, make sure the workflow is clear.

Otherwise you are simply scaling chaos faster.

AI coding agents are now a standard part of software development.

The real competitive advantage is not using AI.

It is engineering workflows around AI.

If you treat AI like a magic senior engineer, you will be disappointed.

If you treat it like a fast junior inside a strong engineering system, you can ship faster without sacrificing code quality.

The next generation of software development is not AI-powered.

It is AI-orchestrated.

If you're experimenting with AI development workflows, I'd love to hear what has worked (or failed) for your team.

Drop your experiences in the comments 👇

Follow TheProdSDE for more content on:

AI engineering
system design
developer productivity

LangGraph vs Semantic Kernel: Python AI Agents in 2026

TheProdSDE — Wed, 11 Mar 2026 12:41:03 +0000

Why This Comparison Matters Right Now

Here's the honest reality of building Python AI agents in 2026: you have two genuinely good framework choices, and picking the wrong one for the wrong problem will cost you architecture refactors, not just a few hours of code changes.

LangGraph and Semantic Kernel have both crossed major milestones since most popular comparisons were written:

LangGraph hit v1.0 in October 2025 — with a formal stability commitment and no breaking changes until 2.0. (official announcement)
LangChain 1.0's create_agent now runs on the LangGraph runtime underneath, making LangGraph the execution engine of the LangChain ecosystem. (LangChain/LangGraph 1.0 blog)
Semantic Kernel shipped first-class MCP support for Python in v1.28.1 — SK can now act as both an MCP client and server natively in the SDK. (official SK dev blog)

If you're still reading comparisons that call LangGraph "unstable" or Semantic Kernel "too tied to .NET", you're reading old content.

This post is grounded in the official LangGraph docs, the official Semantic Kernel docs, and both framework changelogs. Let's go.

TL;DR: The One-Line Decision Rule

Problem Type	Framework
Stateful multi-step agent workflows	LangGraph
Plugin-based AI platforms & MCP interoperability	Semantic Kernel

That distinction explains every trade-off in this article.

Architecture: Two Very Different Mental Models

LangGraph — The Graph Runtime

LangGraph models your agent system as a stateful graph where you explicitly define state, nodes, and edges. Nodes are Python callables or subgraphs. Edges are transitions. State is a typed object that flows through the graph and gets updated at each step.

That's not an internal implementation detail — it's the primary abstraction you work with every day.

The official LangGraph v1 docs describe the framework around three core ideas: durable execution, controllability, and human-in-the-loop. Resuming a workflow from the last checkpoint after a crash, inserting a human review step, or branching into parallel sub-agents are first-class operations — not workarounds.

Since LangGraph v1, LangChain's create_agent lives on top of this runtime. The stack now has a clean separation:

Start with create_agent for standard tool-calling loops.
Drop down to raw LangGraph when you need explicit workflow topology.

Semantic Kernel — The Kernel-Plugin Middleware

Semantic Kernel starts from the Kernel abstraction, which holds AI services, plugins, and functions. Plugins are groups of functions exposed to the model and to agents, and can come from native Python code, prompt templates, or imported external schemas.

The official SK agent-functions docs state:

"Any Plugin available to an Agent is managed within its respective Kernel instance — this enables each Agent to access distinct functionalities based on its specific role."

Orchestration emerges from agents choosing functions and planners sequencing capability calls — rather than from a graph topology you define up front.

This makes Semantic Kernel feel more like AI middleware. You shape what your agent can do, then let function calling and the agent framework decide how to do it.

Architectural Difference — Quick Reference

Dimension	LangGraph	Semantic Kernel
Primary abstraction	Typed state graph (nodes + edges)	Kernel + plugins + agents
Workflow control	You define topology explicitly	Emerges from agent function calling
State management	First-class typed state + checkpointing	Externalized per service or plugin
Best mental model	Durable state machine for agents	AI middleware with composable capabilities

Code: The Same Agent in Both Frameworks

To make the architectural differences concrete, let's build the same agent in both: a multi-turn weather assistant with memory and a system prompt.

LangGraph — Weather Agent with Checkpointing

Pattern from the official LangGraph agents quickstart

# pip install -U langgraph "langchain[openai]"

from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import InMemorySaver
from langchain.chat_models import init_chat_model

# --- Tool: plain Python function ---
def get_weather(city: str) -> str:
    """Get the current weather for a given city."""
    # Replace with a real API call in production
    return f"It's sunny and 28°C in {city}."

# --- LLM ---
model = init_chat_model("openai:gpt-4o-mini", temperature=0)

# --- Checkpointer enables durable multi-turn memory ---
# Swap InMemorySaver for SqliteSaver or PostgresSaver in production
checkpointer = InMemorySaver()

# --- Compile graph agent ---
agent = create_react_agent(
    model=model,
    tools=[get_weather],
    prompt="You are a helpful weather assistant.",
    checkpointer=checkpointer,
)

# --- thread_id binds this conversation to a persistent checkpoint ---
config = {"configurable": {"thread_id": "user-session-1"}}

# Turn 1
response = agent.invoke(
    {"messages": [{"role": "user", "content": "What is the weather in Mumbai?"}]},
    config=config,
)
print(response["messages"][-1].content)

# Turn 2 — agent remembers context automatically via checkpointer
followup = agent.invoke(
    {"messages": [{"role": "user", "content": "How about Delhi?"}]},
    config=config,
)
print(followup["messages"][-1].content)

What is happening architecturally:

create_react_agent compiles a StateGraph with a model-tool loop under the hood.
The checkpointer persists state at every step; the same thread_id resumes from the last saved state automatically.
If the process crashes mid-run, restarting and invoking with the same thread_id picks up from the last checkpoint — durability is a runtime concern, not your concern.

Semantic Kernel — Weather Agent with Plugin

Pattern from the official SK agent-functions docs

# pip install semantic-kernel

import asyncio
from semantic_kernel import Kernel
from semantic_kernel.agents import ChatCompletionAgent
from semantic_kernel.connectors.ai.open_ai import (
    OpenAIChatCompletion,
    OpenAIChatPromptExecutionSettings,
)
from semantic_kernel.connectors.ai import FunctionChoiceBehavior
from semantic_kernel.functions import kernel_function
from semantic_kernel.contents import ChatHistory

# --- Plugin: class with @kernel_function decorators ---
class WeatherPlugin:
    @kernel_function(name="get_weather", description="Get the weather for a city.")
    def get_weather(self, city: str) -> str:
        # Replace with a real API call in production
        return f"It's sunny and 28°C in {city}."

# --- Kernel: holds services and plugins ---
kernel = Kernel()
kernel.add_service(OpenAIChatCompletion(ai_model_id="gpt-4o-mini"))

# --- Execution settings: enable auto function calling ---
settings = OpenAIChatPromptExecutionSettings()
settings.function_choice_behavior = FunctionChoiceBehavior.Auto()

# --- Register plugin ---
kernel.add_plugin(WeatherPlugin(), plugin_name="WeatherPlugin")

# --- Agent: kernel + instructions ---
agent = ChatCompletionAgent(
    kernel=kernel,
    name="WeatherAssistant",
    instructions="You are a helpful weather assistant.",
)

async def run_agent():
    # ChatHistory is your responsibility to maintain across turns
    history = ChatHistory()

    # Turn 1
    history.add_user_message("What is the weather in Mumbai?")
    async for message in agent.invoke(history):
        print(f"Agent: {message.content}")
        history.add_message(message)

    # Turn 2
    history.add_user_message("How about Delhi?")
    async for message in agent.invoke(history):
        print(f"Agent: {message.content}")
        history.add_message(message)

asyncio.run(run_agent())

What is happening architecturally:

The Kernel holds the AI service and plugins as a dependency container.
@kernel_function decorators make Python methods discoverable and invocable by the model automatically.
FunctionChoiceBehavior.Auto() tells the model to call functions when needed.
Memory is a ChatHistory object you manage and pass into each invocation. The runtime does not persist it for you.

The Most Revealing Difference in 6 Lines

# LangGraph — runtime owns durability
checkpointer = InMemorySaver()
config = {"configurable": {"thread_id": "session-1"}}
agent.invoke(messages, config)  # resumes from last checkpoint automatically

# Semantic Kernel — you own state
history = ChatHistory()
history.add_user_message("...")
agent.invoke(history)  # you pass and maintain state explicitly

In LangGraph, durability is a runtime concern. In Semantic Kernel, state management is your concern. Neither is wrong — they match different application models.

Protocol Support: MCP and A2A

This is where Semantic Kernel has made its most significant leap recently.

Semantic Kernel — Native MCP in the Python SDK

The official SK MCP announcement states:

"Python support for MCP has arrived... SK Python can act as both an MCP Host and an MCP Server, support multiple transport methods (stdio, SSE, WebSocket), chain multiple MCP servers together, and expose SK functions or agents as MCP servers."

That is not an adapter or community plugin. It is first-class SDK support from v1.28.1+. For teams building tools and agents that need to cross service boundaries via a standard protocol, this is a meaningful architectural upgrade.

LangGraph — Strong MCP at the Deployment Edge

LangGraph's MCP story is more about deployment than in-process integration. When deployed on the LangGraph Platform, every agent is automatically exposed as an MCP-accessible endpoint at /mcp with no extra code required. For self-hosted deployments, integration is available via the langchain-mcp-adapters package.

Bottom line:
SK is stronger when you want MCP semantics inside your Python process. LangGraph is stronger when you think of agents as deployed services that other clients consume via MCP.

Stability and Breaking Changes: The 2026 Reality

Here is what the official docs actually say now.

LangGraph v1 (October 2025): The official v1 release notes state that the core graph APIs and execution model are unchanged. The main migration note is deprecation of create_react_agent in langgraph.prebuilt in favour of LangChain's create_agent. The LangGraph 1.0 announcement explicitly commits to no breaking changes until 2.0.

Semantic Kernel 1.x: Most architectural disruption landed at 1.0 (namespace reorg, API renames, context variable changes). The H1 2025 SK roadmap and subsequent releases show an incremental, additive pattern with targeted fixes rather than structural breaks.

The old narrative of "LangGraph breaks every release" is no longer accurate. Both frameworks are now in a stability-first phase.

Updated Technical Ratings (March 2026)

Based on official docs and both frameworks' current stable releases:

Dimension	LangGraph	Semantic Kernel
Workflow control	⭐ 4.8 / 5	⭐ 4.0 / 5
Durable execution	⭐ 4.9 / 5	⭐ 4.1 / 5
Plugin/tool architecture	⭐ 4.2 / 5	⭐ 4.8 / 5
MCP interoperability	⭐ 3.9 / 5	⭐ 4.9 / 5
Flow debuggability	⭐ 4.7 / 5	⭐ 3.9 / 5
Start simple, scale complex	⭐ 4.8 / 5	⭐ 4.4 / 5
Python DX overall	⭐ 4.6 / 5	⭐ 4.5 / 5

The scores are intentionally close. Both are production-grade frameworks solving real problems well. The winner for your team is whichever abstraction maps better to how you think about the problem you are solving.

When to Choose Which

✅ Choose LangGraph when:

Your agent logic involves non-trivial branching, retries, human review, or approval steps that benefit from explicit graph topology.
You need durable execution — workflows that survive crashes, resume from checkpoints, and have auditable step history.
You are already invested in the LangChain ecosystem and want the clean create_agent → LangGraph stack with a clear upgrade path.
You want fine-grained observability into how execution moved through a workflow at the node level.

✅ Choose Semantic Kernel when:

You are building a platform or SDK where capabilities are composed as plugins and different agents consume different tool surfaces.
MCP or A2A interoperability is a core requirement and you want it natively in the Python SDK, not via adapters.
Your team already uses a DI/service-oriented architecture and the kernel-plugin model maps naturally to it.
You want lightweight deployment without a dedicated orchestration runtime and can manage state externally.

The One-Line Rule — Revisited

If your agent needs to behave like a durable state machine, use LangGraph.
If your agent needs to behave like a protocol-aware platform component, use Semantic Kernel.

That is the comparison most blog posts are not making. Hopefully this one was useful.

References

We Scaled from Azure App Service to Container Apps: Here's Why

TheProdSDE — Wed, 04 Feb 2026 17:28:43 +0000

If you’ve ever tried to deploy a full stack (Node backend, MCP server, ETL jobs, Angular SPA, and PostgreSQL) on Azure, you quickly hit a few hard limits at each tier. This is our story of rolling out exactly that stack and how we moved step‑by‑step from Azure App Service Free, to Basic, then to Azure Container Apps (ACA) while keeping Postgres on Azure DB for PostgreSQL.

1. Starting with App Service Free (and why it died fast)

We started where many devs do: Azure App Service Free tier (F1). We deployed:

Node.js backend API
Next: a WebSocket‑based MCP server
ETL job as an Azure WebJob
Angular SPA as static assets served by Node
PostgreSQL as Azure Database for PostgreSQL (B1ms + 32 GB storage)

The Free tier is fine for learning, but two hard limits killed it for any real‑world sharing:

No SLA and very limited CPU seconds.
No support for custom‑domain SSL; only *.azurewebsites.net with Azure‑managed cert.

Entra ID, modern browsers, and any third‑party integration now demand HTTPS with a custom domain.

App Service Free = no custom‑domain SSL unless you downgrade to plain HTTP, which is effectively unusable beyond local dev. about-azure

So for any external demo, client, or even corporate‑network access, Free tier is non‑starter.

2. Moving to App Service Basic (HTTPS gate, but one‑endpoint ceiling)

We upgraded to App Service Linux Basic tier (B1), which costs roughly ₹350–₹500/month in India, depending on region and billing cycle. learn.microsoft

This tier unlocked:

Dedicated compute (1 vCore, ~1.75 GB RAM).
Custom domains + App Service Managed Certificates (free, auto‑renewed TLS/SSL). codejack
WebJobs for background tasks (our daily ETL).

On Basic, we could:

Serve the Angular SPA from /
Expose the Node backend API on /api/*
Run the MCP server in the same Node process or via a virtual path. stackoverflow

However, one hard constraint emerged:

👉 Azure App Service exposes only one main HTTP endpoint per app. Everything fights for the same worker process, CPU, and memory.

This created three problems:

No isolation between backend, MCP server, and ETL.
ETL jobs (10–15 min) bloated the same instance that served the API.
Scaling is limited to a few instances (max 3 in Basic), all sharing the same codebase. learn.microsoft

So we had:

HTTPS + custom domain ✅
Cheap, simple PaaS ✅
True microservices isolation ❌

3. Why “one app Service Plan for backend + MCP + ETL” failed

Following the “one App Service Plan for all” pattern sounded clean: share the same plan to keep costs low and everything in one logical place.

But in practice:

All services share one HTTP endpoint, so /, /api, and MCP‑style endpoints live in the same process.
Long‑running ETL jobs prevent the app from being scaled‑to‑zero; you pay for the instance even when idle.
No independent scaling for backend vs MCP vs ETL.

This is where architecture becomes a hard constraint, not just a style choice.

4. PostgreSQL: always separate, always on Azure DB

Throughout this journey, we kept PostgreSQL as Azure Database for PostgreSQL Flexible Server (B1ms + 32 GB storage) on a separate deployment from our compute. azure.microsoft

Why? Because:

DB and app scale independently (compute vs storage, backup, replication).
Flexible Server supports managed backups, high availability, and autoscaling without tying it to App Service pricing tiers. azure.microsoft

Postgres cost (B1ms + 32 GB) sits roughly in ₹1,200–₹2,000/month for low‑to‑medium traffic. azure.microsoft

This is a universal pattern:

App Service / ACA / AKS should not host the DB directly.
Use Azure DB for PostgreSQL (or standalone VM) and wire it via connection string or Managed Identity when possible. docs.azure

5. ACA: the “microservices‑on‑budget” escape hatch

Once we hit the one‑endpoint bottleneck and wanted true isolation between backend, MCP, and ETL, we moved the compute to Azure Container Apps (ACA).

We:

Containerized the app (Node + MCP + SPA + ETL logic) into a single Docker image.
Pushed it to Azure Container Registry (ACR). azure.microsoft
Deployed the backend / MCP containers into Azure Container Apps and wired them to the same Azure DB for PostgreSQL. azure.microsoft

ACA became the dev‑test layer we wanted:

Microservices‑style isolation without full Kubernetes complexity.
Auto‑scale for ETL jobs (5–10 min bursts) while keeping cloud bills low by scaling down to near‑zero.
Per‑second billing with 180k vCPU‑seconds, 360k GiB‑seconds, and 2M requests per month free. azure.microsoft

We ran the backend with 0.25 vCPU + 0.5 GiB RAM and scaled up instances during ETL (5–10 min) only, then let them scale back. This kept the compute cost very low while gaining full control over scaling policies. azureway

For demo / internal‑use access, ACA’s default endpoint (*.azurecontainerapps.io) with built‑in HTTPS was sufficient; we did not need to bring a custom domain yet.

6. Why not ACI, why not just Nginx + self‑managed certs?

During the ACA phase, we evaluated:

Azure Container Instances (ACI) for the ETL / MCP workloads.
Nginx / reverse proxy in front of ACA or App Service with self‑managed SSL certs.

We ruled both out quickly:

ACI is great for short‑lived, one‑shot containers, but it lacks the managed app‑style primitives (health checks, scaling, ingress, jobs) we needed for backend + MCP + ETL. learn.microsoft
Self‑signed certs with Nginx/Grok cause SSL handshake failures in production when the proxy talks to a backend with a non‑trusted certificate. Workarounds like proxy_ssl_verify off are security anti‑patterns. explaintopic

Instead, we leaned into Azure managed TLS:

App Service: App Service Managed Certificates (free) for custom domains. azure.github
ACA: Use Azure Front Door or Azure Functions + App Gateway as the TLS‑terminating layer when you need a custom‑domain install in production. learn.microsoft

This keeps you from managing certs and instead uses Azure‑managed TLS wherever possible.

7. When we move to AKS (for production)

For Dev / Test, ACA + ACR + Azure DB for PostgreSQL is the sweet spot:

Microservices‑style architecture
Cost‑effective per‑second billing
No need to manage Kubernetes (yet)

For production, our plan is Azure Kubernetes Service (AKS) because:

Multi‑cluster, multi‑region HA with enterprise SLA.
Fine‑grained scaling, pod affinities, networking, observability.
RBAC, audit logs, and compliance story for regulated workloads. sedai

AKS is more expensive (control plane + worker nodes), but that’s the cost of enterprise‑grade ops, not just compute. azure.microsoft

So our ladder now looks like:

App Service Free – Learning, POC, but blocked by SSL.
App Service Basic – MVP that needs HTTPS + custom domain.
Azure Container Apps + ACR – Microservices on budget, dev‑test ready.
AKS + Azure DB for PostgreSQL – Production‑ready stack.

8. Cost snapshot (approx, India region)

These are rough monthly estimates for your stack (Node + MCP + ETL + SPA + PostgreSQL) on each tier:

Tier / Service	Cost (approx / month)	Notes
PostgreSQL (B1ms + 32 GB)	₹1,200–₹2,000	Constant across tiers azure.microsoft
App Service Linux (B1, Basic)	₹350–₹500	Single plan hosting backend + MCP + ETL job. learn.microsoft
Azure Container Apps (0.25 vCPU + 0.5 GiB)	₹100–₹500 (or <)	Often under cost when using free vCPU/GiB‑seconds. azure.microsoft
Azure Container Registry (Basic)	₹500–₹600	For 10 GB storage and basic usage. azure.microsoft
Azure Kubernetes Service (2‑node cluster)	₹4,000–₹6,000+	Node VMs dominate cost. azure.microsoft

From Free → App Service Basic → ACA → AKS, moving up is less about price and more about capabilities:

Isolation
True microservices
Production‑grade HA / SLA

9. Architecture pro‑/cons: Quick comparison

Service / Tier	Pros	Cons
App Service Free	Free, simple, no infra management. learn.microsoft	No custom‑domain SSL, limited CPU, zero SLA. learn.microsoft
App Service Basic	HTTPS + custom domain with free managed certs; low operational effort. azure.github	One HTTP endpoint per app, no true isolation, scaling limited. stackoverflow
Azure Container Apps	Microservices, per‑second billing, auto‑scale to zero, ETL‑burst friendly. azure.microsoft	No built‑in custom‑domain TLS; requires Front Door / App Gateway in front for production‑style domains. azure.microsoft
Azure Kubernetes Service (AKS)	Full Kubernetes, HA, RBAC, observability, enterprise‑grade ops. azure.microsoft	Higher cost, steeper learning curve, ops overhead. azure.microsoft
Azure DB for PostgreSQL	Managed DB, backups, HA, autoscale options. azure.microsoft	Separate billing; you must manage connection‑pooling patterns.

10. When to move up the ladder

Use this mental model:

Stay on Free only if you’re learning and not exposing anything to external users.
Move to App Service Basic as soon as you need HTTPS + custom domain (internal dashboards, client demos).
Move to ACA when you want true isolation between backend, MCP, ETL, or when you want per‑second, scale‑to‑zero billing.
Move to AKS when you need multi‑region HA, enterprise SLA, compliance, or mature Kubernetes for your stack.

Also, if you ever catch yourself trying to run everything in one App Service and hitting limits (one endpoint, ETL‑job‑blocking‑API, no scaling boundaries), that’s a clear signal to containerize and move to ACA or AKS.

11. Final takeaway

The Azure deployment ladder is not about chasing the cheapest tier forever. It’s about understanding what each tier unlocks:

Free tier → Proof of concept only.
Basic tier → HTTPS + custom domain, but monolithic; that’s the ceiling.
ACA → Microservices on a budget, with true isolation and flexible scaling.
AKS → Enterprise‑grade ops, HA, and Kubernetes.

In our stack:

PostgreSQL stays on Azure DB for PostgreSQL (B1ms + 32 GB) and interacts with all tiers.
App Service gets you from Free to Basic quickly.
ACA becomes the dev‑test layer for backend + MCP + ETL without over‑committing to AKS.
AKS is the production plan when you’re ready for full Kubernetes maturity.

If you’ve been wrestling with “do I put ETL in WebJobs or in a separate container?”, “how do I share the same DB with all my services?”, or “when is AKS actually worth it?”, this deployment ladder captures the real‑world constraints you’ll hit—and the next step to take.

Running PostgreSQL on AKS for non‑prod testing

Even though we keep PostgreSQL on Azure DB for PostgreSQL in production, you can also run it inside AKS for non‑production testing while using smaller node sizes to keep costs low. For example, you can:

Deploy PostgreSQL as a StatefulSet on low‑sized nodes (e.g., Standard_B2s) with modest CPU, RAM, and storage.
Use this cluster purely for feature, integration, and load‑testing of your backend, MCP, and ETL jobs.

This is useful if your goal is to:

Validate that your application stack behaves correctly under pressure.
Show demos where the entire platform (app + DB) lives on Kubernetes.

However, this setup comes with caveats:

You take on DB operations (backups, failover, patching, monitoring) yourself.
Non‑prod does not replace production‑style tests on Azure DB for PostgreSQL, which remains the recommended pattern for production workloads.

So you can use Postgres‑on‑AKS to prove your app can scale in a test environment, but you should keep managed DB for actual production, where HA, SLA, and operational safety matter.

References

Azure App Service plans overview learn.microsoft
App Service Managed Certificates GA azure.github
Azure Container Apps pricing (including free vCPU‑seconds / GiB‑seconds) azure.microsoft
Azure Database for PostgreSQL Flexible Server pricing & cost‑optimization guide docs.azure
Azure Container Registry pricing azure.microsoft
Azure Kubernetes Service pricing azure.microsoft