DEV Community: claire nguyen

MCP in Production Reality vs the Spec

claire nguyen — Wed, 29 Apr 2026 05:58:18 +0000

Been building against MCP for the last four months and the gap between what vendors claim and what the spec actually supports is getting hard to ignore.

If you have not read the official roadmap yet, it is worth your time. The document published by AAIF in March lays things out clearly and honestly. The list of what is still missing is longer than many people in the ecosystem seem willing to admit.

Stateless Streaming Is Not Here Yet

Stateless Streamable HTTP is still marked as in progress. That has real consequences.

Today, if you want to scale horizontally, you are dealing with sticky sessions or putting a stateful proxy in front of your servers. This is not a small implementation detail. It directly affects reliability, cost, and operational complexity.

Every MCP native at scale pitch I have seen quietly works around this with a custom session layer. That may be practical for now, but it is not what people assume when they hear "stateless protocol."

Async Work Is Still DIY

The Tasks primitive for async and long running operations is also in progress.

In practice, this means any agent doing multi minute work is faking async. Most teams end up with polling endpoints, custom retry logic, and their own definitions of job state.

The problem is not just inconvenience. It is fragmentation. Each implementation behaves slightly differently, which makes interoperability harder before it even begins.

Discovery Is Still Manual

Server discovery is another gap that shows up quickly.

The idea of Server Cards exposed via .well known URLs is promising, but not available yet. Right now, you cannot know what an MCP server can do without connecting to it first.

The Registry preview from late 2025 helps, but it is not a replacement for protocol level discovery. You still end up writing glue code just to answer basic capability questions.

Enterprise Auth Is Not Ready

Authentication is where things feel especially incomplete for real world use.

Most implementations today rely on static client secrets. That works for prototypes, but does not align with how larger organizations manage identity and access.

The roadmap calls out SSO integrated cross app access as a priority. That is exactly what is needed. Until it lands, teams are building their own auth layers on top.

The Hidden Cost: Rewrites Later

Put all of this together and a pattern emerges.

If you are building serious MCP infrastructure today, you are not just implementing the spec. You are filling in gaps around session management, async orchestration, discovery, and authentication.

Those gaps come with a cost. Once these features land in the official spec, a lot of today's custom infrastructure will need to be reworked or replaced. Some abstractions will survive. Many will not.

If you are designing systems now, it is worth being explicit about where you are deviating from the spec and how hard it will be to unwind later.

About Those "Production Ready" Claims

This also makes it hard to take production ready MCP gateway claims at face value in April 2026.

There are usually two possibilities. Either the deployment is small enough that these issues have not surfaced yet, or the vendor has built proprietary extensions on top of MCP.

Neither is inherently wrong, but both are very different from what the marketing suggests.

The Good News

None of this is a knock on MCP itself.

The shape of the protocol feels right. The direction is solid. The roadmap is transparent about what is missing, which is more than can be said for many standards at this stage.

But the reality is simple. Production grade tooling is still catching up.

Are Execution-First Models Getting Underrated for Agent Workflows?

claire nguyen — Mon, 27 Apr 2026 04:50:33 +0000

A lot of model discussion still gravitates toward benchmark screenshots, clever chat demos, or long reasoning traces that look impressive at first glance. Those things are easy to share and easy to evaluate in isolation.

But once a model is actually embedded inside a product or an agent workflow, I am not convinced those are the most useful ways to think about performance anymore.

The question I keep coming back to is much simpler: how much useful work does the model actually get done per token, per step, and per retry?

Shifting the lens to execution

This is where models like Ling-2.6-1T start to stand out.

What makes it interesting is not just its size. It is the way it seems to be positioned. It feels much more execution-first. The focus appears to be on precise instruction following, handling long context without falling apart, fitting naturally into tool use, and maintaining tighter token discipline.

It is less about producing responses that look impressive and more about moving tasks forward efficiently.

That distinction matters more than it might seem.

Where real workflows break down

In real systems, the biggest pain points are rarely about whether a model seems reflective or articulate.

The issues tend to show up elsewhere:

Chains start to drift off task
Retries become expensive and unpredictable
Intermediate steps consume too many tokens
Tool calls become inconsistent or brittle
Multi-step workflows lose structure over time

Individually, these problems feel small. But at scale, they compound quickly. What starts as a minor inefficiency turns into real cost, latency, and operational friction.

In that environment, a model that is slightly more disciplined and slightly more direct can outperform one that is more “impressive” in a single turn.

The hidden cost of reasoning overhead

There is also a subtle cost to models that lean heavily into visible reasoning.

Long reasoning traces can be useful for debugging or transparency. But they also introduce overhead. More tokens, more latency, and more surface area for errors or drift.

If that extra reasoning does not translate into better execution across steps, it becomes hard to justify in production workflows.

Execution-first models, on the other hand, tend to optimize for forward progress. They aim to do the right thing with fewer steps, fewer tokens, and fewer retries.

That tradeoff is not always obvious in demos, but it becomes very clear in sustained usage.

Rethinking what “good” looks like

If the goal is to build reliable agent systems, the definition of a “good” model might need to shift.

It is less about maximum reasoning depth in a single interaction, and more about consistency across many interactions.

Can the model:

Read messy, real-world context without losing track?
Preserve task structure across multiple steps?
Call tools correctly and at the right time?
Avoid unnecessary verbosity and token waste?
Recover cleanly when something goes wrong?

These are not the most glamorous benchmarks, but they are the ones that determine whether a system actually works.

Open question

So I am curious how others are thinking about this.

If the real objective is to move multi-step work forward reliably, are we still overvaluing maximum reasoning depth?

And on the flip side, are we undervaluing execution-per-token as a core metric for agent workflows?

The Actual Cost of Self-Hosting Your LLM (Nobody Does This Math First)

claire nguyen — Thu, 23 Apr 2026 11:28:37 +0000

title: "The Actual Cost of Self-Hosting Your LLM (Nobody Does This Math First)"
published: true
tags: [infrastructure, llm, devops, sre]

canonical_url:

TL;DR: Self-hosting an LLM looks cheaper until you add up everything else. This post walks through the real numbers — compute, storage, networking, and the ops overhead you will absolutely pay. Know the full picture before you spin up that g5 instance.

The Decision That Seems Obvious

Your team is spending $4k/month on OpenAI. Someone runs the numbers on a g5.xlarge. "We could host our own model and cut costs by 70%." Sounds great on paper.

Then you actually do it.

I've watched this play out a few times. The initial estimate is real. The full picture is not. So let me walk you through what the bill actually looks like twelve months in.

Compute: The Number You Actually Know

A g5.xlarge runs about $1.006/hour on-demand. Running inference 24/7 for a mid-sized model, that's roughly $740/month per instance. For any real load, you'll need more than one.

Go with a g5.12xlarge for a serious 7B model? You're at $5.67/hour. That's $4,082/month before you've touched anything else.

Spot instances help. You can get 50–70% off. But spot instances get interrupted, and your inference service needs to handle that gracefully or users see errors. Not impossible, but that's engineering work.

# EKS node group config for spot GPU instances
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

managedNodeGroups:
  - name: gpu-spot
    instanceTypes: ["g5.xlarge", "g5.2xlarge"]
    spot: true
    minSize: 1
    maxSize: 10
    desiredCapacity: 2
    labels:
      workload: llm-inference
    taints:
      - key: nvidia.com/gpu
        value: "true"
        effect: NoSchedule
    iam:
      withAddonPolicies:
        autoScaler: true

The autoscaler handles scaling down. You still need logic to drain in-flight requests before a spot interruption kills the node. AWS gives you a two-minute warning, so your request timeout has to live inside that window.

Storage, Networking, and the Bits Everyone Forgets

Model weights are not small. Llama 3 8B in fp16 is about 16GB. Mixtral 8x7B hits 87GB. You're storing these in S3 or EFS and loading them on pod startup.

EFS for shared model weights runs roughly $0.30/GB/month. For a 50GB model, that's $15/month. Fine. But EFS read throughput matters at startup. Loading 50GB cold takes minutes per pod. Pre-warm your nodes or bake model weights into an EBS snapshot on your AMI.

Networking is where it gets sneaky. Data transfer out of AWS costs $0.09/GB after the first GB. If you're serving inference to users outside AWS — which is most teams — every response adds up. A million requests/month at 1KB average response is $90 in egress. At 10 million requests, that's $900, and nobody put it in the budget.

The Ops Tax Is Real

This is the one that bites hardest. You now own:

Model versioning and rollout
GPU driver updates
CUDA compatibility between driver and runtime
OOM debugging (GPU OOM behaves nothing like CPU OOM)
Observability for inference latency, token throughput, GPU utilisation
On-call when the model service falls over at 2am

That last one is the real cost. Human time. If your team hasn't run LLM inference at scale before, budget a few months of learning. GPU OOM crashes are cryptic. NCCL errors for multi-GPU setups will make you question your life choices.

Self-Hosted vs Managed API: The Honest Table

Factor	Self-Hosted	Managed API
Compute cost	Medium (but tunable)	High per token
Ops burden	High	Near zero
Latency control	Full	Limited
Model flexibility	Full	Vendor-locked
Data privacy	Full	Depends on vendor
Time to first token	Days to weeks setup	Minutes
Scaling complexity	You own it	Handled

No universally right answer here. High-volume batch workloads with infrastructure chops? Self-hosting wins. Need to move fast? Managed APIs are fine.

Some teams go hybrid: managed API for interactive use, self-hosted batch for cost-sensitive workloads. Tools like Bifrost help route between providers and track actual spend across both, which makes the hybrid model easier to reason about.

Trade-offs and Limitations

Self-hosting wins on cost only when GPU utilisation is high. A GPU sitting idle at 20% utilisation is expensive. You need to be hitting 60–70% to beat per-token API pricing.

Spot instances cut compute cost but add operational complexity. Async batch jobs can tolerate interruptions. Real-time inference is harder to design around.

Multi-GPU setups for larger models introduce inter-node networking costs and NCCL configuration pain. Worth it at scale. Not worth it for a team of five still figuring out their workload shape.

Model quantisation (int4, int8) can cut memory requirements dramatically. Running Llama 3 70B in int4 on two A10G GPUs is real. The quality trade-off is usually acceptable for most tasks. Always benchmark before committing to a quantisation level in production.

The Actual Recommendation

Run the full numbers. Not the compute number. The full number: compute, storage, networking, engineering time, on-call time.

If the math works and your team has the capacity, self-hosting is genuinely solid. You get control, data privacy, and cost efficiency at volume.

If you're not sure yet, start with a managed API. Get your usage patterns nailed down. Then revisit.

No worries either way. Both paths are valid. The mistake is assuming one is clearly better without doing the first.

ML Infrastructure Renaissance: What Everyone's Missing About GPU Orchestration

claire nguyen — Sun, 19 Apr 2026 20:25:34 +0000

What's Your Experience?

I'm curious to hear from other infrastructure engineers: what's the most overlooked challenge you're seeing in ML infrastructure today? Are there tools or patterns that have worked particularly well for your teams?

When Your LLM Provider Pulls the Rug: Lessons from Anthropic's OAuth Shutdown

claire nguyen — Sun, 19 Apr 2026 19:27:09 +0000

On April 4th, 2026, Anthropic revoked OAuth access for OpenClaw, killing over 135,000 integrations overnight. Developers who built workflows around Claude Pro/Max subscriptions woke up to broken pipelines and a 10-50x cost increase to stay on Anthropic via API.

This wasn't just another API change—it was a stark reminder that when your authentication and routing goes through a mechanism that your provider controls exclusively, they can pull the rug whenever their business priorities shift.

The Hidden Dependency Trap

We had two internal services that authenticated directly against Anthropic's API. When the OpenClaw situation happened, we weren't directly affected, but it prompted a thorough audit of our LLM infrastructure. What we found was concerning:

Hardcoded provider logic: Model names baked into agent configurations
Different auth patterns: Each provider has unique token refresh flows
Incompatible error handling: Different auth failure codes across providers
Cost lock-in: Subscription models vs API pricing created massive cost disparities

Swapping from Anthropic to OpenAI or Gemini wasn't just a model change—it required rewriting authentication layers, updating response parsing, and redeploying services.

The Gateway Pattern Solution

We moved to routing everything through a centralized gateway that manages provider credentials. Our application code now authenticates against our own gateway API keys, and the gateway handles per-provider authentication.

Benefits we gained:

Provider agnosticism: Swap models without touching application code
Centralized monitoring: Single point for health checks and metrics
Automatic failover: Gateway can reroute requests during outages
Cost optimization: Route to cheapest provider that meets quality thresholds

Implementation Strategy

Here's how we structured our gateway:

# Simplified gateway routing logic
class LLMGateway:
    def __init__(self):
        self.providers = {
            'anthropic': AnthropicProvider(),
            'openai': OpenAIProvider(),
            'gemini': GeminiProvider()
        }
        self.health_checker = HealthChecker()

    def route_request(self, request, fallback_strategy="cheapest_available"):
        # Check provider health and availability
        healthy_providers = self.health_checker.get_healthy_providers()

        # Apply routing strategy
        selected_provider = self._select_provider(
            healthy_providers, 
            fallback_strategy
        )

        return selected_provider.execute(request)

Key Takeaways

Abstract provider specifics: Don't let provider-specific logic leak into your application
Assume instability: Treat LLM providers as unreliable dependencies
Build for portability: Make swapping providers a config change, not a code change
Monitor everything: Health checks should be continuous, not reactive
Plan for failure: Have automatic fallback routes for every critical path

The Bigger Picture

This pattern extends beyond LLMs. Any third-party service that could change pricing, access models, or business priorities should be abstracted through a gateway. Whether it's payment processors, email services, or cloud storage—the principle remains the same.

The most resilient systems aren't the ones that never fail. They're the ones that fail gracefully and recover automatically.

By treating LLM providers as interchangeable components rather than foundational infrastructure, we build systems that can adapt to market changes without requiring massive rewrites.

What provider abstraction patterns are you using in your AI infrastructure? Share your experiences in the comments!