Polyglot AI Architecture: Keep Your .NET App, Add a Python RAG + Agent Service (gRPC, Qdrant, LangGraph, Langfuse — Real Code)

Your enterprise app is .NET. Your UI is ExtJS. Now product wants RAG, agents, and evals — and the entire AI ecosystem in 2026 ships in Python first. You don't want to rewrite anything. A polyglot architecture is the answer: keep .NET as your trusted system-of-record and security boundary, run the AI workload as a separate Python service, talk over a clean typed contract, and observe both sides end-to-end.

This is a real recipe — diagram, code, deployment, and operational playbook — for shipping that pattern in production. Full version with the architecture diagram is on the original post (link at the end).

The architecture at a glance

ExtJS web UI ──► .NET API Gateway ─REST/gRPC─► Python RAG/Agent service
                 (ASP.NET Core)                (FastAPI + LangChain + LangGraph)
                 auth · rate limit             ingestion · retrieval · orchestration
                 contract · security
                                                   │              │
                                                   ▼              ▼
                                             Qdrant vector DB   LLM provider
                                             (Docker)           (Azure OpenAI / OpenAI)

Observability spans every arrow:
  Langfuse       → LLM traces, prompts, evals, token cost
  OpenTelemetry  → end-to-end traces across .NET + Python + Qdrant + LLM

Five components, two languages, one observability fabric.

1. Why polyglot, and why now

Three forces pull against each other inside every enterprise stack in 2026:

1. Your business logic is in .NET. Years of domain models, validation, RBAC, audit trails, integrations. None of it needs to move.
2. Your UI is ExtJS / WinForms / WPF / Razor. It works. Rewriting it is a multi-quarter project that adds zero AI value.
3. The AI ecosystem ships Python first. LangChain, LangGraph, llama-index, vLLM, Ragas, DSPy — Python releases land months before .NET ports (if they exist at all).

Option	Outcome
Rewrite the AI stack in .NET (Semantic Kernel only)	Slow to adopt new techniques; smaller library surface; locked to .NET release cycles.
Rewrite the business app in Python	Throws away years of trusted code; ~100% new bugs; 12+ months.
Polyglot — keep .NET, add Python alongside	Best tool per layer; clean security boundary; both stacks evolve independently.

Polyglot is the only route that respects existing investment and lets you adopt the AI ecosystem at full speed. It means: single product, single front door; .NET owns the contract (authN/Z, rate-limiting, versioning, audit); Python owns the AI workload (RAG, embeddings, orchestration), never exposed to the internet; and the boundary is a typed contract — REST+OpenAPI or gRPC+.proto.

2. What each layer is for

• ExtJS UI — keep it. It's an HTTP client. Add a chat drawer that calls the existing .NET API. The fact that AI runs in Python is invisible to the browser.
• .NET API Gateway — the security boundary. Authentication, authorization (RBAC, row-level), rate limiting, contract versioning, audit logging, outbound gRPC to Python (Polly), PII redaction before the prompt leaves your network, quota/billing. The Python service is never exposed to the internet.
• Python RAG/Agent service — the workhorse. Ingestion (chunk → embed → upsert), retrieval (semantic + optional BM25 hybrid + rerank), prompt construction, pluggable LLM calls, LangGraph multi-agent orchestration, streaming, evals.
• Qdrant — the vector DB. Sub-100 ms ANN up to ~10M vectors on one node, filters, payload indexing, open source, identical in dev and prod. (pgvector / Weaviate / Pinecone are alternatives.)
• LLM provider — pluggable. Azure OpenAI for compliance, OpenAI direct for speed, Anthropic/Bedrock for Claude. Abstracted behind one interface so you swap providers without touching .NET.
• Observability — not optional. Langfuse for LLM-native tracing; OpenTelemetry for end-to-end distributed tracing.

3. The request flow, end to end

1. Browser → POST /api/v1/assistant/query with bearer token.
2. .NET Gateway validates the JWT, checks claims (tenant_id, roles), records the audit event, runs the rate-limit check, redacts PII, then proxies (gRPC) to Python.
3. Python RAG service embeds the query, searches Qdrant filtered by tenant_id, retrieves top-8 chunks, builds the prompt, calls the LLM with streaming, optionally invokes LangGraph agents, and streams tokens back.
4. Langfuse records the prompt, retrieved chunks (with scores), the LLM call (model, latency, tokens, cost), and any eval score.
5. OpenTelemetry records one trace: gateway-auth → rate-limit → grpc-call → embed → qdrant-search → llm-call → response-stream, all sharing one trace_id.
6. Total round trip in the reference build: 1.4 s to first token, 4.1 s to complete.

4. The .NET API Gateway (Program.cs)

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(o => {
        o.Authority = builder.Configuration["Auth:Authority"];
        o.Audience  = builder.Configuration["Auth:Audience"];
    });
builder.Services.AddAuthorization();

// Tenant-aware token-bucket rate limiting
builder.Services.AddRateLimiter(o => {
    o.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(ctx => {
        var tenant = ctx.User.FindFirst("tenant_id")?.Value ?? "anonymous";
        return RateLimitPartition.GetTokenBucketLimiter(tenant, _ => new() {
            TokenLimit = 100, TokensPerPeriod = 20,
            ReplenishmentPeriod = TimeSpan.FromSeconds(10), QueueLimit = 0,
        });
    });
});

// Typed gRPC client to Python, with retries + circuit breaker
builder.Services.AddGrpcClient<RagService.RagServiceClient>(o => {
    o.Address = new Uri(builder.Configuration["Rag:GrpcUrl"]!);
})
.AddPolicyHandler(HttpPolicyExtensions.HandleTransientHttpError()
    .WaitAndRetryAsync(3, n => TimeSpan.FromMilliseconds(200 * n)))
.AddPolicyHandler(HttpPolicyExtensions.HandleTransientHttpError()
    .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)));

// OpenTelemetry — traces propagate to Python via W3C headers automatically
builder.Services.AddOpenTelemetry().WithTracing(t => t
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("api-gateway"))
    .AddAspNetCoreInstrumentation().AddHttpClientInstrumentation()
    .AddGrpcClientInstrumentation().AddOtlpExporter());

var app = builder.Build();
app.UseAuthentication(); app.UseAuthorization(); app.UseRateLimiter();

app.MapPost("/api/v1/assistant/query", async (
    AssistantQueryRequest req, HttpContext ctx,
    RagService.RagServiceClient rag, ILogger<Program> log) =>
{
    var tenantId = ctx.User.FindFirst("tenant_id")?.Value;
    var userId   = ctx.User.FindFirst("sub")?.Value;
    if (tenantId is null || userId is null) return Results.Unauthorized();

    var sanitized = PiiRedactor.Redact(req.Query); // last place under your control
    var grpcReq = new QueryRequest {
        Query = sanitized, TenantId = tenantId, UserId = userId, TopK = req.TopK ?? 8,
    };
    var resp = await rag.QueryAsync(grpcReq, deadline: DateTime.UtcNow.AddSeconds(30));
    return Results.Ok(new AssistantQueryResponse(
        resp.Answer,
        resp.Citations.Select(c => new Citation(c.SourceId, c.Score)).ToList(),
        resp.TraceId));
}).RequireAuthorization();

app.Run();

Worth noticing: tenant-aware rate limiting keyed by tenant_id (not IP); automatic W3C traceparent propagation so FastAPI's OTel middleware continues the same trace; Polly retries + circuit breaker; and PII redaction before the wire.

4.2 The gRPC contract (rag.proto)

syntax = "proto3";
package rag.v1;

service RagService {
  rpc Query (QueryRequest)  returns (QueryResponse);
  rpc Ingest (IngestRequest) returns (IngestResponse);
  rpc StreamQuery (QueryRequest) returns (stream StreamToken);
}

message QueryRequest  { string query = 1; string tenant_id = 2; string user_id = 3; int32 top_k = 4; }
message QueryResponse { string answer = 1; repeated Citation citations = 2; string trace_id = 3; }
message Citation      { string source_id = 1; float score = 2; }
message StreamToken   { string text = 1; bool done = 2; }

Both sides generate code from this file. No type drift, no JSON-string guesswork.

5. The Python RAG / Agent service (FastAPI)

# app/main.py (abridged)
qdrant = QdrantClient(url=os.environ["QDRANT_URL"])
embeddings = AzureOpenAIEmbeddings(azure_deployment="text-embedding-3-small", api_version="2024-10-21")
llm = AzureChatOpenAI(azure_deployment="gpt-4o", api_version="2024-10-21", temperature=0, streaming=True)
vectorstore = Qdrant(client=qdrant, collection_name="docs", embeddings=embeddings)

app = FastAPI(lifespan=lifespan)
FastAPIInstrumentor.instrument_app(app)  # OTel auto-instrumentation

@app.post("/v1/query", response_model=QueryResponse)
async def query(req: QueryRequest) -> QueryResponse:
    with tracer.start_as_current_span("rag.query") as span:
        span.set_attribute("tenant_id", req.tenant_id)
        # Retrieval — Qdrant filtered by tenant for HARD isolation
        docs = await asyncio.to_thread(
            vectorstore.similarity_search_with_score, query=req.query, k=req.top_k,
            filter={"must": [{"key": "tenant_id", "match": {"value": req.tenant_id}}]})
        citations = [Citation(source_id=d.metadata["source_id"], score=float(s)) for d, s in docs]
        context = "\n\n".join(f"[{d.metadata['source_id']}]\n{d.page_content}" for d, _ in docs)
        chain = ({"context": lambda _: context, "query": RunnablePassthrough()}
                 | prompt | llm | StrOutputParser())
        answer = await chain.ainvoke(req.query, config={"callbacks": [lf]})
    return QueryResponse(answer=answer, citations=citations, trace_id=trace_id)

The tenant filter in retrieval is mandatory. Without it, your RAG service leaks documents across tenants.

5.2 The LangGraph agent — planner / tool-user / critic

# app/agent_graph.py (abridged)
def build_agent_graph(llm, vectorstore):
    g = StateGraph(AgentState)
    g.add_node("plan", plan)        # LLM -> JSON list of tool calls
    g.add_node("execute", execute)  # run sql_query / vector_search tools
    g.add_node("draft", draft)      # synthesise an answer, cite each fact
    g.add_node("critique", critique)# check accuracy / hallucination -> 'OK' or issues
    g.add_node("finalise", finalise)
    g.set_entry_point("plan")
    g.add_edge("plan", "execute"); g.add_edge("execute", "draft"); g.add_edge("draft", "critique")
    g.add_conditional_edges("critique",
        lambda s: "finalise" if "OK" in s["critique"] else "draft",
        {"finalise": "finalise", "draft": "draft"})
    g.add_edge("finalise", END)
    return g.compile()

The plan → execute → draft → critique → finalise loop is a classic agentic pattern; the conditional edge lets it iterate until the answer passes self-review (with a max-iteration guard in real code). Ingestion is just chunk → embed → upsert, with the same tenant_id on every chunk so retrieval filters enforce isolation.

6. Wiring it with Docker Compose

services:
  gateway:
    build: ./gateway
    ports: ["8080:8080"]              # ONLY the gateway is reachable from outside
    environment:
      Rag__GrpcUrl: "http://rag:50051"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
    depends_on: [rag, otel-collector]
  rag:
    build: ./rag
    expose: ["50051", "8000"]         # NOT published to host — internal network only
    environment:
      QDRANT_URL: "http://qdrant:6333"
      AZURE_OPENAI_API_KEY: "${AZURE_OPENAI_API_KEY}"
      LANGFUSE_SECRET_KEY: "${LANGFUSE_SECRET_KEY}"
    depends_on: [qdrant, otel-collector]
  qdrant:
    image: qdrant/qdrant:latest
    expose: ["6333"]
    volumes: ["qdrant-data:/qdrant/storage"]
volumes: { qdrant-data: {} }

Only the gateway is exposed; Qdrant, the RAG service, and the collector live on the internal Docker network with expose: (not ports:). In Kubernetes: one LoadBalancer for the gateway, ClusterIP for the rest, NetworkPolicies blocking all egress from the RAG namespace except to the LLM provider and Langfuse.

7. Observability — OTel + Langfuse answer different questions

OpenTelemetry — "where is the latency?" One trace for a query:

api-gateway: POST /api/v1/assistant/query        [1420 ms]
├── auth.validate-jwt                              [3 ms]
├── rate-limit.acquire                             [0.4 ms]
├── grpc.call rag.RagService/Query             [1410 ms]
│   └── rag-service: rag.query                  [1395 ms]
│       ├── rag.retrieve -> qdrant.search        [180 ms]
│       └── rag.generate -> azure-openai.chat   [1190 ms]

You see immediately: the LLM took ~85% of wall-clock, Qdrant ~12%, gateway single-digit ms.

Langfuse — "why did the model answer wrong?" The exact prompt (with retrieved chunks), the completion, tokens in/out, cost in USD, model version, and the citations with their scores. When a user says "the bot said our return policy is 60 days, it's 30," you open Langfuse, see that an outdated PDF chunk was retrieved at score 0.78 above the current policy at 0.74 — a concrete, reproducible bug. OTel tells you when something is slow; Langfuse tells you when something is wrong.

Evals — the closed loop

# tests/eval_rag.py (abridged) — run nightly in CI
scores = {
    "faithfulness":      faithfulness.score(results),
    "answer_relevancy":  answer_relevancy.score(results),
    "context_precision": context_precision.score(results),
}
if scores["faithfulness"] < 0.85:
    raise SystemExit("Faithfulness regression — block deploy")

Change a prompt, swap a model, or tweak chunking, and you catch quality regressions before they ship.

8. What this architecture buys you

Concern	How polyglot helps
Existing .NET investment	Zero rewrite; the gateway is plain ASP.NET Core
Existing ExtJS UI	Untouched; AI arrives as new endpoints
Velocity on AI features	Python ships at the speed of LangChain releases
Security & compliance	One auth boundary, one audit log, PII redaction at the edge
Multi-tenancy	Tenant filter enforced in Qdrant AND in .NET claims
LLM provider switch	One Python class; .NET never knew
Debugging	OTel for latency, Langfuse for quality, linked by trace_id
Scaling	Each service scales independently

9. When NOT to do this

• Greenfield AI-only product — go pure Python; the polyglot tax is pure overhead with no .NET legacy.
• Tiny app, single instance — use Semantic Kernel in-process and ship.
• No observability budget — Langfuse + OTel are not optional here.
• Hard real-time (<200 ms total) — the gRPC hop + LLM call combine to ~1 s minimum.

10. A 2-week integration plan

Week 1 — boundary: add the ASP.NET Core gateway endpoints (stubbed); wire ExtJS to a chat drawer; author rag.proto and generate both clients.
Week 2 — AI workload: stand up FastAPI + Qdrant in Compose; wire the LangChain RAG chain; add Langfuse + OTel with trace propagation; write 20 eval queries and run them nightly.

After two weeks you have a real, observable, production-shaped polyglot AI stack — and the rest is content, prompts, and evals, which is exactly where you want to spend time.

---

The mental model: two services, one trust boundary. The .NET side is your enterprise; the Python side is your AI lab. They meet at a contract narrow enough to maintain, typed enough to test, observable enough to debug. Keep the contract narrow, treat observability as a feature, and version everything (prompts, models, gRPC).

---

This is a condensed version. The full article — with the complete code, the full request-flow walkthrough, and the architecture diagram — is on PrepStack:

→ Polyglot Architecture for Enterprise AI — .NET API Gateway + Python RAG/Agent Service