DEV Community: Signadot

Why the MCP Server Is Now a Critical Microservice

Signadot — Thu, 19 Feb 2026 15:55:36 +0000

Read this article on Signadot.

In my previous article on preparing CI/CD pipelines to ship production-ready agents, I argued that we cannot ship agents to production that are driven primarily by non-deterministic models. Instead, they must be built as robust workflows where the large language model (LLM) is introduced strategically at specific steps within a deterministic control flow.

Now we must examine the most critical node in that framework.

The Model Context Protocol (MCP) server facilitates interactions between the probabilistic LLM node and the deterministic microservices workflow. It acts as the translation layer connecting the reasoning engine to external data and tools.

The model is one half of the agent architecture. The MCP server is the other half. While model evaluations validate the reasoning engine, they cannot verify the system as a whole. Validation strategies relying on mocks fail to test the agent as a workflow.

Reliability of the end-to-end workflow is paramount when shipping agents to production. The MCP server is the critical node in this topology, acting as both sensory organ and effector arm. If it transmits ambiguous signals, the agent acts erratically. It hallucinates. It degrades user trust. It causes critical business errors.

The Architectural Shift From Contracts to Semantics

To understand the failure risks, we must examine how the MCP server alters service contracts.

Service-to-service communication is deterministic in standard microservices environments. Service A calls Service B using a strict REST or gRPC contract. The interaction is rigid. It is predictable. It is easily validated.

An agentic workflow inverts this.

The agent is a nondeterministic actor operating on probabilistic logic. It decides when to call a tool based on semantic context provided by the MCP server. The server exposes a world model rather than just an API endpoint.

This makes the MCP server a distinct type of microservice. It is a translation layer converting probabilistic intent into deterministic action. This responsibility manifests in three operations requiring rigorous engineering.

1. Defining Capability Boundaries

The MCP server defines agent capabilities through JSON-RPC tool definitions.

If the server exposes a schema with vague descriptions, the agent cannot formulate a valid execution plan. A human developer might read documentation to clarify an API field, but the agent relies solely on metadata exposed by the list_tools capability.

Consider a payment operations agent handling refunds. A fragile MCP implementation might expose a tool named refund_user to process a refund.

This lacks semantic density. The model does not know whether this applies to a full or partial refund or if it handles tax calculation. It is a black box.

A robust implementation defines the boundary with precision. It exposes process_prorated_subscription_refund. The description explicitly states that it calculates the remaining balance for the current billing cycle and issues a credit.

The reasoning chain breaks without this specificity.

2. Governing the Context Economy

The MCP server governs the context window. It must retrieve backend data and format it for LLM consumption.

This data engineering challenge requires differentiating between signal and noise.

Providing a raw 5 MB JSON dump dilutes agent attention. It wastes tokens and increases latency. Conversely, providing too little data causes the agent to hallucinate missing details.

The server must act as a transformation layer that optimizes raw data into context-ready snippets.

3. Executing Side Effects

The MCP server executes actions for the agent. When an agent triggers a deployment, the server is the execution mechanism.

A confused agent can trigger destructive loops if the server lacks idempotency or error handling. The server must implement safeguards preventing the model from erroneously retrying state-changing operations.

The Engineering Rigor Required for Production

Shipping agents to production requires due diligence exceeding standard microservice development. This is most visible in return state ambiguity.

A traditional API might return a 404 error code, which a client handles with logic. An MCP server faces a more complex challenge. It must return a natural language description or structured tool result explaining why the action failed.

If the server returns a generic stack trace, the agent may retry endlessly or invent a plausible but incorrect reason for failure. The error message becomes part of the prompt for the next conversation turn. It must be engineered as carefully as the system prompt.

Latency is also critical. Agents operate in a sequential thought loop. They reason. They call a tool. They wait. They reason again.

A slow server breaks the cognitive chain. High latency causes context timeouts, forcing the agent to abandon workflows. This leaves systems in inconsistent states.

Scaling Testing via Multitenancy

The nondeterministic nature of the client makes testing difficult. Traditional unit tests are insufficient.

Unit testing a Python function to ensure valid JSON output does not prove that an agent will understand how to use it. Mocks are equally ineffective. They decouple the test from real system behavior and create false confidence.

The only way to validate an MCP server is through rigorous end-to-end testing against real dependencies. However, spinning up full cluster replicas for every test is rarely feasible.

To validate an MCP server without the overhead of full environment replication, we treat the test run as a logical slice within a shared cluster. This life cycle relies on header based routing and session affinity:

Handshake and routing: The test harness initializes the agent with specific context metadata (such as a baggage header or a custom routing parameter) during the WebSocket or transport handshake. This signals the ingress controller or service mesh to route the persistent JSON-RPC session specifically to the candidate MCP server (the version under test), bypassing the stable production traffic.
Session isolation: Once connected, the agent operates within a strictly isolated session. While the underlying compute resources may be shared, the logical control flow is pinned to the candidate artifact. This ensures that the nondeterministic reasoning of the agent is exercising only the new code paths.
Shared downstream state: The candidate MCP server processes the agent’s intent but executes side effects against shared downstream dependencies such as staging databases or stable microservices. This eliminates the need for mocks, allowing the agent to interact with a realistic “world model” where API contracts and data schemas are genuine.

This architecture enables safe end-to-end semantic testing. The harness prompts the agent to perform an operation and verifies the state change against downstream microservices.

Isolation at the connection layer turns the test run into a private lane on a public highway. This enables full end-to-end validation of the MCP server without saturating testing infrastructure or introducing resource contention in shared staging environments.

Treat It Like Critical Infrastructure

Teams that are shipping advanced, customer-facing agents understand that robust MCP servers are critical infrastructure. We must recognize them as complex architectural nodes that directly affect agent reliability.

Model evals are critical but insufficient for production standards. Rigorous integration testing of agents with MCP servers is necessary.

An agent is only as effective as its tools. A fragile MCP server creates a fragile agent. Elevating the MCP server to a fully validated microservice is essential for advancing agent development from internal experiments to products that are ready for production.

Learn more about how to implement this testing workflow for your agents at Signadot.com

Your CI/CD Pipeline Is Not Ready To Ship AI Agents

Signadot — Wed, 18 Feb 2026 15:43:01 +0000

Read this blog on Signadot.

Let’s be honest with ourselves for a minute. If you look past the hype cycles, the viral Twitter demos and the astronomical valuation of foundation model companies, you will notice a distinct gap in the AI landscape.

We are incredibly early, and our infrastructure is failing us.

While every SaaS company has slapped a copilot sidebar onto its UI, actual autonomous agents are rare in the wild. I am referring to software that reliably executes complex and multistep tasks without human hand-holding. Most agents today are internal tools glued together by enthusiastic engineers to summarize Slack threads or query a SQL database. They live in the safe harbor of internal usage where a 20% failure rate is a quirky annoyance rather than a churn event.

Why aren’t these agents facing customers yet? It is not because the models lack intelligence. It is because our delivery pipelines lack rigor. Taking an agent from cool demo to production-grade reliability is an engineering nightmare that few have solved because traditional CI/CD pipelines simply were not designed for non-deterministic software.

We are learning the hard way that shipping agents is not an AI problem. It is a systems engineering problem. Specifically, it is a testing infrastructure problem.

The Death of ‘Prompt and Pray’

For the last year, the industry has been obsessed with frameworks that promised magic. You give the framework a goal and it figures out the rest. This was the “prompt and pray” era.

But as recent discussions in the engineering community highlight, specifically the insightful conversation around 12-Factor Agents, production reality is boringly deterministic. The developers actually shipping reliable agents are abandoning the idea of total autonomy. Instead, they are building robust and deterministic workflows where large language models (LLMs) are treated as fuzzy function calls injected at specific leverage points.

When teams start testing agents, they almost always start with evals.

The 12-Factor philosophy correctly argues that you must own your control flow. You cannot outsource your logic loop to a probabilistic model. If you do, you end up with a system that works 80% of the time and hallucinates itself into a corner the other 20%.

So we build the agent as a workflow. We treat the LLM as a component rather than the architect. But once we settle on this architecture, we run headfirst into a wall that traditional software engineering solved a decade ago but which AI has reopened. That wall is integration testing.

The Trap of Evals

When teams start testing agents, they almost always start with evals.

Evals are critical. You need frameworks to score your LLM outputs for relevance, toxicity and hallucinations. You need to know if your prompt changes caused a regression in reasoning.

However, in the context of shipping a product, evals are essentially unit tests. They test the logic of the node, but they do not test the integrity of the graph.

In a production environment, your agent is not chatting in a void. It is acting. It is calling tools. It is fetching data from a CRM, updating a ticket in Jira or triggering a deployment via an MCP (Model Context Protocol) server.

The reliability of your agent is not just defined by how well it writes text or code. It is defined by how consistently it handles the messy and structured data returned by these external dependencies.

The Integration Nightmare

This is where the platform engineering headache begins.

Imagine you have an agent designed to troubleshoot Kubernetes pod failures. To test this agent, you cannot just feed it a text prompt. You need to put it in an environment where it can do several things. It must call the Kubernetes API or an MCP server wrapping it. It must receive a JSON payload describing a CrashLoopBackOff. It must parse that payload. It must decide to check the logs. Finally, it must call the log service.

If the structure of that JSON payload changes, or if the latency of the log service spikes, or if the MCP server returns a slightly different error schema, your agent might break. It might hallucinate a solution because the input context did not match its training examples.

To test this reliably, you need integration testing. But integration testing for agents is significantly harder than for standard web apps.

Why Traditional Testing Tails

In traditional software development, we mock dependencies. We stub out the database and the third-party APIs.

But with LLM agents, the data is the control flow. If you mock the response from an MCP server, you are feeding the LLM a perfect and sanitized scenario. You are testing the happy path. But LLMs are most dangerous on the unhappy path.

You need to know how the agent reacts when the MCP server returns a 500 error, an empty list or a schema with missing fields. If you mock these interactions, you are writing the test to pass rather than to find bugs. You are not testing the agent’s ability to reason. You are testing your own ability to write mocks.

The alternative to mocking is usually a full staging environment where you spin up the agent, the MCP servers, the databases and the message queues.

But in a modern microservices architecture, spinning up a duplicate stack for every pull request is prohibitively expensive and slow. You cannot wait 45 minutes for a full environment provision just to test if a tweak to the system prompt handles a database error correctly.

The Need for Ephemeral Sandboxes

To ship production-grade agents, we need to rethink our CI/CD pipeline. We need infrastructure that allows us to perform high-fidelity integration testing early in the software development life cycle.

We need ephemeral sandboxes.

A platform engineer needs to provide a way for the AI developer to spin up a lightweight, isolated environment that contains:

The version of the agent being tested.
The specific MCP servers and microservices it depends on.
Access to real (or realistic) data stores.

Crucially, we do not need to duplicate the entire platform. We need a system that allows us to spin up the changed components while routing traffic intelligently to shared and stable baselines for the rest of the stack.

This approach solves the data fidelity problem. The agent interacts with real MCP servers running real logic. If the MCP server returns a complex JSON object, the agent has to ingest it. If the agent makes a state-changing call like restart pod, it actually hits the service or a sandboxed version of it. This ensures the loop is closed.

This is the only way to verify that the workflow holds up.

Shifting Left on Agentic Reliability

The future of AI agents is not just better models. It is better DevOps.

If we accept that production agents are just software with fuzzy logic, we must accept that they require the same rigor in integration testing as a payment gateway or a flight control system.

We are moving toward a world where the agent is just one microservice in a Kubernetes cluster. It communicates via MCP to other services. The challenge for platform engineers is to give developers the confidence to merge code.

That confidence does not come from a green checkmark on a prompt eval. It comes from seeing the agent navigate a live environment, query a live MCP server and execute a workflow successfully.

Conclusion

Building the agent is the easy part. Building the stack to reliably test the agent is where the battle is won or lost.

As we move from internal toys and controlled demos to customer-facing products, the teams that win will be those that can iterate fast without breaking things. They will be the teams that abandon the idea of “prompt and pray” and instead bring production fidelity to their pull request (PR) review. This requires a specific type of infrastructure focused on request-level isolation and ephemeral testing environments that work natively within Kubernetes.

Solving this infrastructure gap is our core mission at Signadot. We allow platform teams to create lightweight sandboxes to test agents against real dependencies without the complexity of full environments. If you are refining the architecture for your AI workflows, you can learn more about this testing pattern at signadot.com.

Ramp’s Inspect shows closed-loop AI agents are software’s future

Signadot — Thu, 12 Feb 2026 15:23:51 +0000

Read this blog on Signadot.

The recent release of the background coding agent Inspect by Ramp’s engineering team serves as a definitive proof point that closed-loop agentic systems are the future of software development. It has transformed coding agents into truly autonomous engineering partners, and it is fundamentally changing the way agents deliver software.

Whether teams use a custom cloud development environment (CDE) like Ramp or another approach, the signal is clear: Teams need to solve for this kind of autonomy or risk getting left behind. Modern engineers need access to coding agents that do not just generate code but also run it, verify the output, and iterate on the solution until it works.

This distinction represents a fundamental shift. The industry has been focused on optimizing the “brain” of agents, solving for context windows and reasoning. Ramp’s success validates that the “body” matters just as much.

The ability to interact with a runtime environment is what transforms code from a hypothesis into a solution. This verification loop separates truly autonomous coding agents from those that rely on humans to validate their work.

The open-loop bottleneck

Modern coding agents are impressive. They can plan complex refactors and generate thousands of lines of code. However, these agents typically operate in an open loop. They rely on the developer to act as the runtime environment. The agent proposes a solution. The human must compile, test, and interpret error messages or feed them back to the agent. The cognitive load of verification remains with the user.

This workflow caps developer velocity. The speed of the agent is irrelevant if the verification process is slow. We have optimized code generation to be near instantaneous, but verification remains bound by human bandwidth and linear CI pipelines.

Inspect demonstrates that closing that loop unlocks a new category of velocity. By giving the agent access to a sandbox to run builds and tests, the agent transitions from text generator to task completer. It hands off a verified solution rather than a draft.

The impact is measurable. Ramp reported vertical internal adoption charts. Within months, approximately 30% of all pull requests merged to its frontend and backend repositories were written by Inspect. This penetration suggests closed-loop agents are a step function change in productivity, not a marginal improvement.

The economics of curiosity

The value proposition of closed-loop agents is not just delivering code faster. It is about the parallelization of solution discovery.

In traditional workflows, exploring refactors or library upgrades is expensive. It requires context switching, stashing work and fighting dependency conflicts. Because experimentation costs are high, we experiment less. We stick to safe patterns to avoid the time sink of failure.

Background agents change the economics of curiosity. If an engineer can spin up 10 concurrent agent sessions to explore 10 architectural approaches, the cost of failure drops significantly.

Consider a team migrating a legacy component. Currently, this is a multiweek spike. In the new paradigm, a developer could instead task a fleet of agents to attempt the migration using different strategies. One agent might try a strangler fig pattern. Another might attempt a hard cutover. A third might focus on integration tests.

The developer then reviews results rather than typing code. The agents run in isolated sandboxes. They build, catch syntax errors, and run test suites until they achieve a green state. The developer wakes up to three potential pull requests verified against the CI pipeline and chooses the best one.

Verification beyond localhost

Ramp’s Inspect platform validates within a custom-built CDE. To ensure these environments start quickly despite their complexity, a sophisticated snapshotting system keeps images warm and ready to launch. Ramp was able to extend this CDE infrastructure to also support integration testing, a brilliant engineering feat that works well for its specific context.

However, for many organizations building complex, cloud native applications with high levels of dependencies, this approach faces significant hurdles. Often, the entire stack is too large to be spun up on a single virtual machine (VM) or devpod. In these scenarios, while CDEs remain excellent for replacing local development laptops, high-fidelity integration testing requires a different approach.

To enable true autonomy in these complex environments, we need a way to perform integration testing without replicating the entire world. We can connect agents directly to a shared baseline environment using existing Kubernetes infrastructure.

In this model, the agent deploys only the modified service to a lightweight sandbox. The infrastructure uses dynamic routing and context propagation to direct specific test traffic to that sandbox while fulfilling all other dependencies from a shared, stable baseline.

This approach gives coding agents the power to execute autonomous end-to-end testing, regardless of the stack’s size or complexity. It leverages the existing cluster to provide high-fidelity context. An agent can then run integration tests against real upstream and downstream services. It sees how the change interacts with the actual message queue schema and the latency of the live database.

This closes the loop with higher fidelity while lowering the infrastructure barrier. By testing against a shared cluster, the agent can catch integration regressions that might pass in a hermetic VM without requiring the platform team to build a custom orchestration engine to support it.

The future of software delivery

The release of Inspect is a clear signal of where software development is heading. The era of the human engineer as the sole verifier is ending. We are moving toward a world where agents operate as autonomous partners capable of exploring solutions and verifying their own work.

Ramp has proven that this workflow is not science fiction. It is working in production today and is driving massive efficiency gains. The question for the rest of the industry is not whether to adopt this workflow, but how.

Whether a team chooses to build a custom platform like Ramp or adopt an existing cloud native solution like Signadot to give their agents a runtime, the imperative is the same. We must provide our agents with a body. We must close the loop between generation and verification. Once we do, we unlock a level of velocity that will define the next generation of high-performing engineering teams.

Your infrastructure isn’t ready for agentic development at scale

Signadot — Thu, 05 Feb 2026 15:53:22 +0000

Read this blog on Signadot.

I have spent the last year watching the AI conversation shift from smart autocomplete to autonomous contribution. When I test tools like Claude Code or GitHub Copilot Workspace, I am no longer just seeing code suggestions. I am watching them solve tickets and refactor entire modules.

The promise is seductive. I imagine assigning a complex task and returning to merged work. But while these agents generate code in seconds, I have discovered that code verification is the new bottleneck.

For agents to be force multipliers, they cannot rely on humans to validate every step. If I have to debug every intermediate state, my productivity gains evaporate. To achieve 10 times the impact, we must transition to an agent-driven loop where humans provide intent while agents handle implementation and integration.

The code generation feedback loop crisis

Consider a scenario where an agent is tasked with updating a deprecated API endpoint in a user service. The agent parses the codebase, identifies the relevant files, and generates syntactically correct code. It may even generate a unit test that passes within the limited context of that specific repository.

However, problems emerge when code interacts with the broader system. A change might break a contract with a downstream payment gateway or an upstream authentication service. If the agent cannot see this failure, it assumes the task is complete and opens a pull request.

The burden then falls on human developers. They have to pull down the agent’s branch, spin up a local environment, or wait for a slow staging build to finish, only to discover the integration error. The developer pastes the error log back into the chat window and asks the agent to try again. This ping-pong effect destroys velocity.

Boris Cherny, creator of Claude Code, has noted the necessity of closed-loop systems for agents to be effective. An agent is only as capable as its ability to observe the consequences of its actions. Without a feedback loop that includes real runtime data, an agent is building in the dark.

In cloud native development, unit tests and mocks are insufficient for this feedback. In a microservices architecture, correctness is a function of the broader ecosystem.

Code that passes a unit test is merely a suggestion that it might work. True verification requires the code to run against real dependencies, real network latency, and real data schemas. For an agent to iterate autonomously, it needs access to runtime reality.

The requirement: Realistic runtime environments at scale

In a recent blog post, “Effective harnesses for long-running agents,” Anthropic’s engineering team argued that an agent’s performance is strictly limited by the quality of its harness. If the harness provides slow or inaccurate feedback, the agent cannot learn or correct itself.

This presents a massive infrastructure challenge for engineering leadership. In a large organization, you might deploy 100 autonomous agents to tackle backlog tasks simultaneously. To support this, you effectively need 100 distinct staging environments.

The traditional approach to this problem fails at scale. Spinning up full Kubernetes namespaces or ephemeral clusters for every task is cost-prohibitive and slow. Provisioning a full cluster with 50 or more microservices, databases, and message queues can take 15 minutes or more. This latency is fatal for an AI workflow. Large language models (LLMs) operate on a timescale of seconds.

We are left with a fundamental conflict. We need production-like fidelity to ensure reliability, but we cannot afford the production-level overhead for every agentic task. We need a way to verify code that is fast, cheap, and accurate.

The solution: Environment virtualization

The answer lies in decoupling the environment from the underlying infrastructure. This concept is known as environment virtualization.

Environment virtualization allows the creation of lightweight and ephemeral sandboxes within a shared Kubernetes cluster. In this model, a baseline environment runs the stable versions of all services. When an agent proposes a change to a specific service, such as the user service mentioned earlier, it does not clone the entire cluster. Instead, it spins up only the modified workload containing the agent’s new code as a shadow deployment.

The environment then utilizes dynamic traffic routing to create the illusion of a dedicated environment. It employs context propagation headers to route specific requests to the agent’s sandbox. If a request carries a specific routing key associated with the agent’s task, the service mesh or ingress controller directs that request to the shadow deployment. All other downstream calls fall back to the stable baseline services.

This architecture solves the agent-environment fit in three specific ways:

Speed: Because a single container or pod is launching, rather than a full cluster, sandboxes spin up in seconds.
Cost: The infrastructure footprint is minimal. You are not paying for idle databases or duplicate copies of stable services.
Fidelity: Agents test against real dependencies and valid data rather than stubs. The modified service interacts with the actual payment gateways and databases in the baseline.

The seamless verification workflow for AI agents

The mechanics of this verification loop rely on precise context propagation, typically handled through standard tracing headers like OpenTelemetry baggage.

When an agent works on a task, its environment is virtually mapped to the remote Kubernetes cluster. This setup supports conflict-free parallelism. Multiple agents can simultaneously work on the same microservice in different sandboxes without collision because routing is determined by unique headers attached to test traffic.

Here is the autonomous workflow for an agent refactoring a microservice:

Generation: The agent analyzes a ticket and generates a code fix with local static analysis. At this stage, the code is theoretical.
Instantiation: The agent triggers a sandbox via the Model Context Protocol (MCP) server. This deploys only the modified workload alongside the running baseline in seconds.
Verification: The agent runs integration tests against the cluster using a specific routing header. Requests route to the modified service while dependencies fall back to the baseline.
Feedback: If the change breaks a downstream contract, the baseline service returns a real runtime error (e.g., 400 Bad Request). The agent captures this actual exception rather than relying on a mock.
Iteration: The agent analyzes the error, refines the code to fix the integration failure, and updates the sandbox instantly. It runs the test again to confirm the fix works in the real environment.‍
Submission: Once tests pass, the agent submits a verified pull request (PR). The human reviewer receives a sandbox link to interact with the running code immediately, bypassing local setup.

Why engineering’s future is autonomous

As we scale the use of AI agents, the bottleneck moves from the keyboard to the infrastructure. If we treat agents as faster typists but force them to wait for slow legacy CI/CD pipelines, we gain nothing. We simply build a longer queue of unverified pull requests.

To move toward a truly autonomous engineering workforce, we must give agents the ability to see. They need to see how their code performs in the real world rather than just in a text editor. They need to experience the friction of deployment and the reality of network calls. This is Signadot’s approach.

Environment virtualization is shifting from a tool for developer experience to foundational infrastructure. By closing the loop, agents can do the messy and iterative work of integration. This leaves architects and engineers free to focus on system design, high-level intent, and the creative aspects of building software.

Traditional Code Review Is Dead. What Comes Next?

Signadot — Tue, 27 Jan 2026 18:22:10 +0000

Read this blog on Signadot.

I noticed a quiet shift in our engineering team recently that brought me to a broader realization about the future of software development: Code review has changed fundamentally.

It started with a pull request (PR). An engineer had used an agent to generate the entire change, iterating with it to define business logic, but ultimately relying on the agent to write the code. It was a substantial chunk of work. The code was syntactically perfect. It followed our linting rules. It even included unit tests that passed green.

The human reviewer, a senior engineer who is usually meticulous about architectural patterns and naming conventions, approved it almost immediately. The time between the PR opening and the approval was less than two minutes.

When I asked about the speed of the approval, they said they checked if the output was correct and moved on. They did not feel the need to parse every line of syntax because it was written by an agent. They spun up the deploy preview, clicked the buttons, verified the state changes and merged it.

This made sense, but it still took me by surprise. I realized that I was witnessing the silent death of traditional code review.

The Silent Death of the Code Review

For decades, the peer review process has been the primary quality gate in software engineering. Humans reading code written by other humans served two critical purposes:

It caught logic bugs that automated tests missed.
It maintained a shared mental model of the codebase across the team.

The assumption behind this process was that code is a scarce resource produced slowly. A human developer might write 50 to 100 lines of meaningful code in a day. Another human can reasonably review that volume while maintaining high cognitive focus.

But we are entering an era where code is becoming abundant and cheap. In fact, the precise goal of implementing coding agents is to generate code at a velocity and volume that by design makes it impossible for humans to keep up.

When an engineer sees a massive block of AI-generated code, the instinct is to offload the syntax-checking to the machine. If the linter is happy and the tests pass, the human assumes the code is valid. The rigorous line-by-line inspection vanishes.

The Problem: AI Trust and the Rubber Stamp

This shift leads to what I call the rubber stamp effect. We see a “lgtm” (looks good to me) approval on code that nobody actually read.

This creates a significant change to the risk profile. Human errors usually manifest as syntax errors or obvious logic gaps. AI errors are different. Large language models (LLMs) often hallucinate plausible but functionally incorrect code.

Traditional diff-based review tools are ill-equipped for this. A diff shows you what changed in the text file. It does not show you the emergent behavior of that change. When a human writes code, the diff is a representation of their intent. When an AI writes code, the diff is just a large volume of tokens that may or may not align with the prompt.

We are moving from a syntax-first culture to an outcome-first culture. The question is no longer “Did you write this correctly?” The question is “Does this do what we asked the agent for?”

Previews as the New Source of Truth

In this new world, where engineers are logic architects who offload the writing of code to agents, the most important artifact is not the code. It is the preview.

If we cannot rely on humans to read the code, we must rely on humans to verify the behavior. But to verify behavior, we need more than a diff. We need a destination. The code must be deployed to a live environment where it can be exercised.

While frontend previews have become standard, the critical gap — and the harder problem to solve — is the backend.

Consider a change to a payment processing microservice generated by an agent. The code might look syntactically correct. The logic flow seems correct. But does it handle the race condition when two requests hit the API simultaneously? Does the new database migration lock a critical table for too long?

You cannot see these problems in a text diff. You cannot even see them in a unit test mock. You can only see them when the code is running in a live, integrated environment.

A backend preview environment allows for true end-to-end verification. It allows a reviewer to execute real API calls against a real database instance. It transforms the review process from a passive reading exercise into an active verification session. We are not just checking whether the code compiles. We are checking whether the system behaves.

As AI agents write more code, the “review” phase of the software development life cycle must evolve into a “validation” phase. We are not reviewing the recipe. We are tasting the dish.

The Infrastructure Challenge: The Concurrency Explosion

However, this shift to outcome-based verification comes with a massive infrastructure challenge that most platform engineering teams are not ready for.

A human developer typically works linearly. They open a branch, write code, open a pull request, wait for review and merge. They might context switch between two tasks, but rarely more.

AI agents work in parallel. An agent tasked with fixing a bug might spin up 10 different strategies to solve it. It could open 10 parallel pull requests, each with a different implementation, and ask the human to select the best one.

This creates an explosion of concurrency.

Traditional CI/CD pipelines are built for linear human workflows. They assume a limited number of concurrent builds. If your AI agent opens 20 parallel sessions to test different hypotheses, you face two prohibitive problems: cost and contention.

First, you cannot have 20 full-scale staging environments spinning up on expensive cloud instances. Imagine spinning up a dedicated Kubernetes cluster and database for 20 variations of a single bug fix. The cloud costs would be astronomical.

Second, and perhaps worse, is the bottleneck of shared resources. Many pipelines rely on a single staging environment or limited testing slots. To avoid data collisions, these systems force PRs into a queue.

With existing human engineering teams, these queues are already a frustrating bottleneck. With multiple agents dumping 20 PRs into the pipe simultaneously, the queue becomes a deadlock. The alternative of running them all at once on shared infrastructure results in race conditions and flaky tests.

Scaling Development With Environment Virtualization

To scale agent-driven development, we cannot rely on infrastructure built for linear human pacing. We are talking about potentially hundreds of concurrent agents generating PRs in parallel, all of which need to be validated with previews. Cloning the entire stack for each one is not a viable option.

The solution is to multiplex these environments on shared infrastructure. Just as a single physical computer can host multiple virtual machines (VMs), a single Kubernetes cluster can multiplex thousands of lightweight, ephemeral environments.

By applying smart isolation techniques at the application layer, we can provide strict separation for each agent’s work without duplicating the underlying infrastructure. This allows us to spin up a dedicated sandbox for every change, ensuring agents can work in parallel and validate code end-to-end without stepping on each other’s toes or exploding cloud costs.

Conclusion

There is a clear shift happening in the way we review changes. As agents take over the writing of code, the review process naturally evolves from checking syntax to verifying behavior. The preview is no longer just a convenience. It is the only scalable way to validate the work that agents produce.

At Signdot, we are building for this future. We provide the orchestration layer that enables fleets of agents to work in parallel, generating and validating code end-to-end in a closed loop with instant, cost-effective previews.

The winners of the next era won’t be the teams with the best style guides, but those who can handle the parallelism of AI agents without exploding their cloud budgets or bringing their CI/CD pipelines to a grinding halt.

In an AI-first world, reading code is a luxury we can no longer afford. Verification is the new standard. If you cannot preview it, you cannot ship it.

Merging To Test Is Killing Your Microservices Velocity

Signadot — Mon, 19 Jan 2026 19:57:45 +0000

Read this blog on Signadot.

Frontend and data layers have evolved with branch-based previews and isolated environments. Why is the backend service layer stuck with shared staging?

If you are a platform engineer or an engineering leader, look at your current development pipeline. Is everything treated equally?

To me, it seems that there is a glaring discrepancy in the way we treat different parts of the stack.

When a frontend developer pushes code to a feature branch, tools like Vercel or Netlify immediately spin up a deploy preview. It is a unique URL, isolated from production, where they can click around and validate changes instantly.

When a database engineer needs to test a schema migration, modern platforms like Neon or PlanetScale allow them to branch the database. They get an isolated, copy-on-write clone of the production data to wreck and repair without affecting a single real user.

But what happens when a backend engineer pushes a change to one microservice in a mesh of 50?

Nothing.

There is a gaping hole in the middle of our cloud native stack. While frontend and data layers have evolved to embrace branch-based development, the backend service layer is stuck in the stone age of shared environments.

This isn’t just an annoyance. It is the primary bottleneck preventing teams from truly shifting left.

The Merge To Validate Anti-Pattern

In most distributed architectures, a developer working on a backend service cannot realistically run the entire platform on their laptop. It is too heavy.

So they rely on unit tests and mocks. But we all know that mocks are liars. They do not catch the contract drift between services or the latency issues that only appear over the network.

To get real validation, the developer has to merge their branch to the main trunk so it can be deployed to a shared staging environment.

This is where velocity goes to die.

The queue: Developers wait in line to deploy to staging.
The block: If one developer breaks staging, everyone is blocked.
The noise: Testing fails, but is it your code, or did someone else deploy a bad config to the auth-service five minutes ago?

We have normalized this dysfunction. We treat staging as a fragile, sacred monolith. But in an era where we want to deploy multiple times a day, merging to trunk just to see if your code works is backward. It is like pouring concrete before you have checked the blueprints.

The Solution: Service Branching

We need to bring the Vercel and Neon experience to the Kubernetes backend. We need service branching.

The goal is simple. Every git branch should result in a testable, isolated environment.

However, the physics of microservices makes this hard. You cannot duplicate a cluster with 100+ services for every single pull request. The cost and spin-up time would be prohibitive.

The solution is not duplication. It is isolation.

Imagine a base environment, your existing staging cluster, that runs the stable version of all your services. When a developer pushes a change to a specific service, the platform shouldn’t clone the cluster. It should simply spin up a lightweight sandbox containing only the modified service.

Smart routing does the rest:

Standard traffic flows through the stable baseline.
Test traffic is intercepted and rerouted only to the sandboxed service.
If the sandboxed service needs to call other services, it routes back into the stable baseline. This gives the developer the experience of a full, dedicated environment with a fraction of the infrastructure footprint.

The New Mental Model: Git Equals Environment

For this to work at scale, platform engineers need to provide a clean mental model that maps source code directly to infrastructure.

This is what the new standard looks like:

Trunk (main) corresponds to the baseline environment (staging). This is the source of truth. It represents the stable state of the world where all services are interacting as expected.
Feature branch (feat-xyz) corresponds to a sandbox environment. This is ephemeral. It lives only as long as the PR is open. It contains only the delta of the services that have changed in that specific branch.

When a developer opens a PR, they do not need to think about clusters or namespaces. They just get a dedicated playground that mirrors their branch perfectly.

The Holy Grail: The Full Virtual Stack

When you combine this service branching approach with the existing tools for frontend and database branching, you unlock something powerful: a full virtual stack per branch.

Imagine a workflow where a developer creates a branch, and magically, a complete, isolated environment materializes. To the developer, it feels like they have their own private copy of the entire company’s infrastructure.

This includes frontend, backend services and database schemas. They are all aligned to their specific code changes.

They can run end-to-end integration tests on their branch before merging. They can hand a URL to a product manager to demo the feature. They can validate complex migrations safely. It is a dedicated reality for their feature, created instantly and destroyed just as quickly.

Why This Matters: Speed and Quality at Scale

This model shifts the paradigm from serial blocking to massive parallelism.

Remove the bottleneck: Large engineering teams no longer have to queue up for staging. You can have 10, 50 or 100 developers and agents testing simultaneously without stepping on each other’s toes.
True shift left: Integration testing happens during development, not after the merge. You catch the bug when you write it, not three days later when the staging build fails.
More quality, faster: When testing is easy and isolated, people do more of it. We stop fearing deployments and start treating them as routine.

The result is a software delivery pipeline that is both significantly faster and more stable.

Closing the Gap

The technology to do this exists. The patterns are proven. It is time for platform teams to stop managing static environments and start managing dynamic, ephemeral workflows.

If you are looking to implement this service branching layer to complete your testing strategy, this is exactly what Signadot was built for. Signadot provides the orchestration layer that brings request-based isolation to Kubernetes.

Stop merging to test. Start branching to validate.

Agentic Coding Tools Are Accelerating Output, Not Velocity

Signadot — Mon, 12 Jan 2026 16:14:19 +0000

Read this article on Signadot.

We are at a pivotal moment in software development. In just a few years, LLMs have gone from impressive chatbots to full-blown coding agents built directly into your IDE.

This has the entire industry racing to accelerate developer velocity with AI. With 84% of developers now using AI tools, and companies like Google stating that 25% or more of their code is now generated by AI, the mandate is clear.

So far, this hasn’t translated into significantly more code being generated, but that is already changing. Models and tooling are evolving rapidly, with agents now able to work in parallel on longer coding tasks

Right now, this workflow is still maturing. We are in a transition phase where tools are helpful but often require heavy verification, limiting the actual output of code that makes it to PR. But as the tools continue to get better, the per-developer output of code is only going to increase.

And that is the goal. Teams are adopting these tools now with the expectation that they will continue to improve developer velocity, ultimately enabling the same teams to write 5x or 10x more code.

But the future state where that goal is achieved presents a critical challenge. The resulting explosion in code output will inevitably make existing CI/CD bottlenecks worse in direct proportion to the coding efficiencies gained.

The Invisible Queue

Picture the emerging workflow. A developer runs three Cursor sessions at once. One refactors the auth service. Another adds a feature to the notification system. A third optimizes database queries. By the end of the day she has three PRs ready to publish. Multiply that by a hundred developers and you get a sense of the scale of code that will be hitting legacy CI/CD pipelines.

Without modernizing validation infrastructure in parallel to developer tooling, the acceleration in code generation will never translate to an equal acceleration in productivity. Instead, it will create a merge crisis. All those PRs will continue to pile up at the exact same chokepoint: validation.

Code review becomes the new bottleneck. Even with AI-assisted reviews, humans still need to sign off. If teams achieve their goal of 5x developer output, that translates to hundreds of PRs a week needing eyeballs, testing, and approval. The queue grows. PRs age. Conflicts multiply.

But the real chaos happens post-merge. All that code lands in staging and staging breaks, because nobody could actually test how their microservices changes interact until they're all deployed together. Now 100 developers are blocked, staring at a broken staging environment, waiting for someone to untangle the mess.

The Only Way Forward

There's only one solution, and it's counterintuitive: test more, but earlier.

Not unit tests. Not mocks. Real integration tests. End-to-end tests. Running against actual services. But here's the key: run them at the granularity of every code change, during local development, before the PR even exists.

Think about it differently. Right now, that Cursor session is generating code. It writes a new API endpoint, updates three service calls, modifies a database schema. The developer reviews the code, thinks it looks good, publishes the PR. Then the waiting starts. Code review queue. CI pipeline. Manual testing in staging. Maybe it works. Maybe it doesn't. Either way, you find out hours or days later.

What if instead, while Cursor is writing that code, it's also spinning up a lightweight test environment, deploying those changes, running integration tests, and validating everything works, all before the developer even switches back to that tab?

That's not science fiction. That's how development needs to work in the agentic era.

Shifting Left, For Real This Time

We've been talking about "shift left" for years. This time it's not optional. It's critical for survival.

The paradigm: every unit of code produced by an AI agent gets tested end-to-end during local development. Not after merge. Not during the PR review. While being written.

When this works, something magical happens. The PR that lands for review isn't a leap of faith. It's already validated. The reviewer sees: "15 integration tests passed. 3 end-to-end flows validated. Contract tests clean." Code review becomes fast. Confidence is high. Merges happen quickly. Staging stays green.

This is where Signadot comes in.

Making It Real

The solution isn't better code review tools or faster CI pipelines. It's parallelizing validation itself.

Parallelize Validation

To unblock the queue, we must enable full integration testing during local development.

Signadot allows developers to use the existing staging environment as a shared baseline. By isolating requests rather than duplicating infrastructure, every developer gets a sandboxed virtual private staging environment.

This allows 100+ developers to test in parallel on the same cluster without impacting each other. No queue. No contention. No waiting for staging to be free.

Agent-Native Infrastructure via MCP

Signadot integrates directly into the AI coding workflow via the Signadot MCP (Model Context Protocol) Server.

Tools like Cursor and Claude Code can now "speak" infrastructure. They use plain English commands to instantiate sandboxes, configure routing, and manage resources directly from the agent interface.

The agent doesn't need you to switch contexts. It handles infrastructure the same way it handles code.

Bi-Directional Tunneling

Once the sandbox is requested, Signadot establishes a secure, bi-directional tunnel between the agent's local environment and the remote Kubernetes cluster.

Requests triggered from the staging UI in the cluster are routed to your local machine. Simultaneously, your locally running service directly hits remote dependencies—database, auth, other microservices—in the cluster. You're testing against the real stack, not mocks.

AI-Driven Test Generation via Traffic Capture

As you interact with your local service, Signadot captures the actual request/response traffic flows. The coding agent analyzes this captured traffic to automatically generate functional API tests, ensuring your changes don't break existing contracts.

No manual test writing. The agent sees how your service actually behaves and writes tests that validate those behaviors.

Autonomous Debugging Loop

Test failed? The agent reads the live logs streamed directly from the Signadot sandbox to pinpoint the error. It corrects the code and reruns the tests instantly in the same sandbox.

This closes the feedback loop without human intervention. The agent iterates until tests pass, all during local development.

When you publish the PR, you attach the sandbox link. Your reviewer sees exactly what was tested, how it performed, and what edge cases were validated. The PR goes from "needs investigation" to "LGTM" in minutes.

The Real Unlock

Agentic coding tools promise 10x code generation and companies are racing to achieve that goal. But without granular pre-merge testing, we will remain stuck with 1x shipping velocity.

The companies that will win are not the ones generating the most code. They are the ones who can validate and ship that code as fast as it is written. That means testing at the same granularity as coding. Every change. Every branch. Every experiment.

The bottleneck is moving. The question is whether your testing infrastructure is moving with it.

At Signadot, we're building the platform that makes agentic coding actually productive. Because code in a PR queue isn't value. Code in production is.

It’s Time To Kill Staging: The Case for Testing in Production

Signadot — Thu, 04 Dec 2025 16:02:20 +0000

Read this blog on Signadot.

Learn how testing in production with the right guardrails can eliminate bottlenecks, reduce costs and help your team ship more reliable code faster.

Staging has always been a necessary evil. New approaches to isolation and on-demand sandboxes have finally made it just plain evil.

For decades, the staging environment has been a fixture of software development. And for just as long, it has been hated by developers everywhere. It’s the proverbial traffic jam that every developer is forced to sit in just to get their work validated.

Yet staging is no longer necessary; its time has come and gone. Newer isolation methods enable developers to test safely in live environments, providing fast, high-fidelity feedback that is impossible to achieve in a staging environment.

It is time to kill your staging environment.

The Problem We All Know

Staging environments make sense, in theory. We must test code in a production-like environment before we ship to production. Anything else would be madness.

But the cure has become its own disease. In any organization with more than a handful of microservices, the staging environment inevitably becomes a wasteland of developer pain and burned cash.

It’s a bottleneck: When 50 developers merge code, staging becomes a shared queue. Tests fail not because of bad code, but because another developer deployed a conflicting change.
It’s not production: We call it “production-like,” but it never has the same data scale, traffic patterns or identity and access management (IAM) policies. This fidelity gap is where dangerous bugs hide.
It kills velocity: Commit code, wait for CI, wait for a deploy slot, run a 40-minute test suite. This multihour cycle destroys flow state.
Nobody maintains it: Teams treat it as a dumping ground for unstable builds, further diverging from production.

We have accepted this broken workflow for 20 years. We believed it was the only way.

From Environment Isolation To Request Isolation

Staging exists because of the assumption that testing must be isolated at the environment level. To test a new version of the payment service, you must deploy it to an environment that also contains a cart service, user service and auth service.

This assumption is outdated and obsolete.

The new model is request-level isolation. Instead of cloning an entire environment, you spin up only the service you’re changing. This model is enabled by Kubernetes-native platforms that provide on-demand sandboxes for every request.

Here is how it works:

A new service version is spun up in an on-demand, isolated “sandbox.”
When a test request is sent (tagged with a unique header), it is routed to the sandboxed service.
As that service calls its dependencies, those calls are routed back to the stable, baseline services in production.
The test request remains isolated as it travels through the stack, while all other traffic flows normally.

With this approach, you get high-fidelity testing (real dependencies, real network policies) without the downsides of shared environments (no collisions, no queues, dramatically lower cost).

Request-level isolation can be implemented into a traditional local > staging > production deployment flow to improve it, eliminating contention and long waits for a CI pipeline. But its real power lies in bypassing the need for staging altogether, enabling testing in production.

The Safety Model

Testing in production sounds dangerous. It’s not when you have the right guardrails.

Strict data isolation is critical. The biggest concern is that a sandboxed service could corrupt data in other services. The solution is simple. The same routing header that isolates test traffic also routes database operations to separate test databases. For example, when a test request flows through the system, each service recognizes the test context and directs all database writes to isolated test data stores, completely separate from production databases. Test users interact only with test data. Production data remains untouched.
Multitenancy provides the foundation. Virtual private network (VPN) restrictions ensure test traffic originates only from authorized internal networks. Audit logs track every sandbox session for compliance.
Request routing provides blast radius control. Your sandbox is isolated at the request level. Your colleagues’ work is unaffected. Production traffic flows normally.
Progressive rollout remains essential. Sandboxes handle preproduction validation, but you still use canary deployments, feature flags and observability to safely roll out to real users.

Answering the Hard Questions

Testing in production is the logical evolution of the movement to shift testing left. But making such a foundational change to your CI/CD pipeline naturally brings up some critical questions:

“How do you guarantee test traffic doesn’t corrupt production data?” Test writes are isolated. The isolation header redirects all database writes to ephemeral, nonproduction data stores that are destroyed after the test. Production data is never touched.
“What about the blast radius? How do you stop a bad test from DDoSing a downstream service?” Sandboxes are “shadow” deployments built with guardrails like circuit breakers and network policies. A buggy test with runaway network requests is automatically throttled and contained, preventing it from overwhelming baseline services or affecting other users.
“This sounds fine for simple APIs, but what about Kafka or gRPC?” The isolation model is protocol-agnostic. The isolation header is propagated over gRPC or as a Kafka message header. A sandboxed consumer, for example, reads from the main topic but only processes messages with its unique sandbox ID.
“What about compliance and audit requirements?” This model is more auditable. Every sandbox is tied to a specific user and a pull request/dev session. All test traffic is explicitly tagged with a sandbox ID and user identity, creating a granular audit log that is far superior to a shared staging environment.

Addressing these and making the shift to testing in production will inevitably involve some upfront engineering investment. However, the return on investment for that work is not just the savings from eliminating the direct infrastructure costs of your staging. It’s also a better developer experience, faster product delivery and fewer opportunities are lost to competitors that can iterate and ship faster than you.

It’s Time To Let Go

Eliminating staging may sound like a pipe dream, but several prominent cloud native teams like DoorDash and Uber have already made the shift left to testing in production. Driven by their highly complex microservice stacks and a need to get better testing fidelity, they are also realizing huge infrastructure cost savings.

Teams like these deprecating their staging environments to test in production represent a broader trend: the rejection of approximation in favor of reality. Staging environments are artifacts of an era when duplicating infrastructure was a harder problem to solve than coordinating humans around shared resources.

That era is ending.

The future isn’t about building better approximations of production or optimizing your CI pipeline. It’s about adopting an entirely new paradigm. The teams taking this step aren’t just moving faster and cutting costs. They’re also shipping more reliable code.

It’s time to kill your staging environment.

Kubernetes Isn't Your AI Bottleneck - It's Your Secret Weapon

Signadot — Mon, 17 Nov 2025 15:24:20 +0000

Read this blog on Signadot.

Let’s get one thing straight: In the AI era, the only thing that matters is experimenting faster than your competition. Generative coding assistants are enabling rapid iteration and users are flocking to whichever platform offers them the latest, most automated features.

But for many established companies, there’s a huge, complex anchor slowing you down: Kubernetes (K8s).

You adopted it to scale, and now it feels like a velocity tax. It’s the stumbling block that gets in the way of rapid, AI-assisted coding converting into actual product iteration. You watch agile, AI-first teams shipping 10 times faster on simpler platforms, and you worry that your “mature” stack is leaving you behind.

I’ll be blunt: If you’re a five-person team trying to find product-market fit, this criticism is 100% correct. You shouldn’t be near K8s. Your constraint is discovery, not delivery. Stop worrying about infrastructure.

But this article isn’t for you.

This is for the teams in growth mode and beyond who feel that exact pain. The teams that need to ship fast to compete but are trapped by their own scale.

I’m here to tell you Kubernetes isn’t your stumbling block. If you use it right, it’s the superpower that will let you win the AI integration race.

The Bottleneck Has Moved

Let’s be real about what’s happening. Tools like Cursor, Claude and the whole fleet of AI coding tools are ridiculously good at generating code, to the extent that they are even enabling a new class of code contributors. The “blank page” problem is vanishing. I can ask an agent to “refactor this Python service to use a new gRPC endpoint and add retry logic,” and it will generate a 90% correct pull request (PR) in 30 seconds.

The bottleneck is no longer writing code. It’s validating.

My engineers’ job is shifting from “typist” to “editor-in-chief.” Their day is less about writing code line by line and more about asking:

“The agent gave me three valid-looking approaches. Which one is actually better?”
“This PR looks right, but did it silently break one of the 15 downstream services that depend on it?”
“How can I test this end to end without spending two days mocking dependencies?”

The new currency for competitive advantage is speed of experimentation. The team that can validate and ship 10 AI-generated experiments while the other team is still setting up their environment is going to win. Period.

And this is where K8s, the platform everyone loves to hate, becomes your secret weapon.

K8s Is the Ultimate Experimentation Platform

The old complaint about K8s was, “It’s too complex! I don’t want to run 20+ microservices on my laptop just to test a one-line change.”

That argument is dead. If you’re still doing that, you’re using it wrong.

Tools (like my own startup, Signadot) can help. No one runs the whole stack locally anymore. You connect your local machine to the cluster, or better yet, you get an isolated “sandbox” inside the cluster for every single PR.

This is the game-changer that unlocks rapid experimentation at enterprise scale.

When an engineer gets a PR from an AI agent, they shouldn’t be testing it on their MacBook. They should have a workflow that, with one click, gives them:

An ephemeral sandbox: K8s is brilliant at this. It spins up a lightweight, isolated environment containing just the new version of their service.
Request-level isolation: This new “sandbox” environment intelligently routes only the developer’s test requests to their new code. All other traffic flows to the stable “baseline” services.

A preview URL: The developer gets a unique URL to test their AI-generated feature against the entire production-like stack, without colliding with anyone else or breaking staging.

Now, that engineer can review three different AI-generated PRs in an hour. They can run a full suite of end-to-end (E2E) tests against each one. They’re not validating code in a vacuum; they’re validating outcomes in a real-world environment.

Stop Validating Code and Start Validating Hypotheses

This brings me to my last point: testing.

What about running all those tests? K8s gives you the ultimate building blocks. Sure, you could just use a managed CI provider, but that’s like building your factory inside someone else’s warehouse. You’re stuck with their rules, their limitations and their pricing. You’ll inevitably outgrow it. K8s is about owning the factory itself. It gives you the control to build a custom validation platform that is tailored precisely to your company’s workflow. You’re not renting a generic tool; you’re building a durable, competitive asset.

Your whole validation pipeline runs on the same platform as your application.

An agent generates a PR.
CI kicks in, builds a container and deploys it to a K8s sandbox.
A series of Kubernetes Jobs are triggered, hammering that sandbox’s preview URL with E2E tests, load tests and contract tests.
The engineer gets a report: “Approach A passed all tests. Approach B failed the latency test under load.”

This is how you build an “experimentation engine,” not a “CI/CD pipeline.” You’re creating a factory for validating hypotheses at scale. The human’s job is to manage the factory, not turn the wrenches.

So yes, K8s is complex. It’s a beast. But in an era where code is generated instantly, the battleground has shifted from creation to validation. And the only platform built to handle that level of high-concurrency, isolated, on-demand experimentation at scale is Kubernetes.

If you’re in growth mode, speed of innovation is all that matters. Stop arguing about YAML and start building your experimentation engine.

The Great AWS Outage: The $11 Billion Argument for Kubernetes

Signadot — Mon, 10 Nov 2025 18:45:51 +0000

Image by Erik McLean from Unsplash. Read this blog on Signadot.

Well, that was a rough week.

If you’re in tech, you know exactly which week I’m talking about. The great AWS us-east-1 outage of October 2025. Where were you when the alerts started firing?

I was in the middle of a demo, and suddenly, nothing worked. Not our app, not our auth, not our CI/CD pipelines. It was a digital ghost town. For eight hours, a massive chunk of the internet — from streaming services and e-commerce sites to, terrifyingly, financial and healthcare platforms simply vanished.

In the days and weeks since, the postmortems have rolled in. Publications like Tom’s Guide chronicled the massive, cascading impact, while Forbes has been tallying the cost. The current estimate? Over $11 billion in lost revenue and market value.

Eleven. Billion. Dollars.

The Siren Song of the Single Cloud

For a decade, the public cloud has been an unbelievable accelerator. We traded capital expenditure for operational expenditure, and in return, we got compute, storage and a rich ecosystem of managed services on demand. It was a fantastic deal.

But we also got comfortable. We built our entire systems, our businesses, around one provider’s proprietary APIs. We built on top of DynamoDB, used Lambda for functions and wired everything together with identity and access management (IAM). It was fast, it was powerful, and it was a ticking time bomb.

The October outage wasn’t just a service failure; it was the failure of the control plane. When the identity services went down, the whole house of cards collapsed. It exposed the fundamental flaw in our thinking: We built incredibly resilient applications on top of a single, massive point of failure.

The Forbes article pointed out that Wall Street’s new favorite phrase is “multicloud.” And they’re not wrong. The companies that had active-active setups across AWS and Google Cloud Platform (GCP), for example, were the ones tweeting, “We’re experiencing minor issues” instead of “We are completely dead.”

But for most of us, “just go multicloud” is a terrible, simplistic and wildly expensive piece of advice.

It’s easy to jump on the bandwagon and blame AWS. But let’s be honest, engineering at that scale is impossibly hard. Failures will happen.

Now that the dust has settled, we need to ask ourselves the real, uncomfortable question: Why were we so fragile?

Why ‘Multicloud’ Is a Trap (Usually)

If your answer to the outage is to have your team also learn the ins and outs of Google‘s IAM, Azure’s Active Directory and all their distinct managed database services … good luck.

True multicloud is hard. It’s not just running a few virtual machines (VMs) in two places. It’s:

Different APIs: The way you provision a load balancer in AWS is completely different from how you do it in GCP.
Different services: There is no 1:1 equivalent for every managed service. You end up building for the lowest common denominator, or worse, building two (or three) entirely separate stacks.
Different tooling: Your scripts are useless in Azure. Your entire CI/CD and observability stack may need to be duplicated or rearchitected.

This approach doesn’t just double your infrastructure cost; it doubles your cognitive load. You’re shipping features, managing reliability and fighting fires across two completely alien ecosystems.

For years, this complexity was the price of admission for true resilience. Most of us, rightly, decided not to pay it.

But what if the price of admission just dropped to zero? What if the platform we’ve been adopting for other reasons was the key all along?

Kubernetes as the Great Abstraction Layer

This is where Kubernetes (K8s) changes the game.

For many, K8s is just a “container orchestrator.” It’s what you use to run your microservices. But that description misses the forest for the trees.

Kubernetes is a consistent, cloud-agnostic API for your entire application.

Think about it. A Kubernetes Deployment.yaml looks identical whether you’re submitting it to a cluster running on Elastic Kubernetes Service (AWS), Google Kubernetes Engine or Azure Kubernetes Service. A Service object abstracts away the underlying cloud load balancer. A PersistentVolumeClaim abstracts away the underlying storage class (Amazon Elastic Block Store, Google Persistent Disk, etc.).

K8s is the abstraction layer we’ve been missing. It’s the “operating system” for the cloud.

When your application only speaks Kubernetes, you are no longer locked into a cloud provider’s proprietary APIs. You are locked into an open source standard that runs everywhere.

This makes the multicloud dream a practical reality:

True portability: A container image is a container image. Your app, packaged as a container, will run identically on your laptop, in an AWS us-east-1 cluster and in a GCP europe-west-2 cluster.
Infrastructure as data: Your application’s entire desired state is just a set of YAML or JSON files. Pointing your Argo CD or Flux (GitOps) pipeline to a new, empty cluster in a different region — or on a different cloud — is trivial.
Federation and failover: With modern tooling, you can manage a fleet of clusters as one logical unit. Service meshes (like Linkerd or Istio) can automatically route traffic away from a failing region or cloud provider, often with no human intervention.

Adopting Kubernetes isn’t just about container orchestration. It’s a strategic business decision to buy back your freedom. It’s how you save billions of dollars, not just by surviving an outage, but by not having to build and maintain N different versions of your platform.

Beyond Resilience: The Real Win Is Velocity

Here’s the part that most “multicloud” think pieces are missing. Focusing on Kubernetes purely for disaster recovery is like buying an F1 car to go grocery shopping.

The real, day-to-day magic of Kubernetes is what it does for your developer productivity.

We are living in a new, AI-native world. We have copilots and agents generating enormous amounts of code. The bottleneck in software is no longer writing code; it’s testing and validating it.

How can you be sure your AI-generated (or junior dev-generated) change doesn’t break one of the 50 other microservices it has to talk to?

The old way was to have a shared staging environment. A single, brittle, always-broken “God” environment that everyone was terrified to touch. It was a permanent bottleneck. But, used the right way, Kubernetes can be a velocity supercharger.

With its native concepts of namespaces, resource quotas and network policies, Kubernetes is an excellent multitenant platform. This multitenancy unlocks a far more powerful and scalable model than simply spinning up complete, isolated copies of your entire stack for every pull request — a strategy that quickly becomes unfeasible with dozens or hundreds of microservices.

Imagine this more advanced approach:

A developer opens a pull request (PR) with a change to a single microservice.
A CI/CD pipeline instantly spins up only that changed service.
Using a service mesh (like Linkerd or Istio) and context-aware routing, the platform creates a “virtual” test environment.
When a developer or an automated end-to-end test sends a request to this environment (e.g., by adding a special HTTP header), the mesh intelligently routes that request.
Requests for the changed service go to the new version. Requests for all other (unchanged) services are routed to the stable, shared baseline stack.
The developer gets a high-fidelity, isolated test against the full stack, but without the massive overhead of duplicating it.
Once the PR is merged, only that single, lightweight namespace is destroyed.

This is the holy grail of CI/CD. It gives teams the confidence to merge and deploy 50+ times a day. And it’s something that is simply not feasible, financially or technically, on a traditional VM-based architecture.

Don’t Focus on the Last Outage, Prep for the Next Decade

The AWS outage was a painful, expensive lesson in the fragility of concentration. Yes, Kubernetes is the technical solution that enables the multiregion and multicloud resilience that Wall Street is now demanding.

But that’s just the beginning.

Don’t adopt K8s just to survive the next provider outage. Adopt it to build a platform that is resilient to the bottlenecks in your own development process. Adopt it so your teams can ship faster, safer and with more confidence.

This is the vision that excites me. At Signadot, we see Kubernetes as the ultimate foundation for developer productivity — a platform that lets every developer get an isolated, high-fidelity test environment on demand, even in this new AI-driven world of constant, rapid change. (You can see more about this approach in our docs.)

The future of software is fast, distributed and complex. Stop building castles on a single provider’s sand. It’s time to build on rock.

Developer-First Traffic Management: See Live Traffic & Override APIs Instantly

Signadot — Thu, 06 Nov 2025 15:46:06 +0000

Image by Guerilla Buzz from Unsplash

Read this article on Signadot.

What if developers could see live traffic flowing in a complex cluster, and then instantly mock or override any API in that microservices stack for testing?

Service meshes are the undisputed titans of modern infrastructure, giving platform engineers unprecedented control over production traffic. But they were built for platform engineers, not developers. When you hand the keys to that complex engine to a developer who just wants to take a test drive, you get friction, slowdowns, and a development cycle that feels more like navigating a bureaucracy than building innovative software.

This isn't a critique of the meshes themselves; it's a critique of how they're used. We’ve learned a hard lesson: tools built for the operational rigor of production are often the wrong tools for the creative chaos of development.

This post focuses on what we built to solve this problem: a new set of developer-first traffic management tools that puts the developer back in the driver's seat. In a follow-up post (Part 2), we'll dive deep into how we built it.

The Developer's Dilemma: A Day in the Life

Let's ground this in a concrete example. Imagine you're a developer working on a large e-commerce application, tasked with adding a feature to the checkout-service. To test it properly, you need it to interact with the real payment-service and inventory-service running in a shared staging environment.

Testing this change the "old way" presented a terrible choice: attempt to run the entire stack locally (a fantasy), or redeploy the entire service for every minor code change (a nightmare). We've previously solved this core problem with Sandboxes: a way to create an isolated environment that intelligently routes all traffic for your checkout-service to your local workstation, while all other service-to-service traffic remains unaffected.

This is a massive leap forward. But as developers started using this power, they wanted to do more. The "all-or-nothing" routing of an entire service was great, but it didn't solve for more granular, complex scenarios. Developers still needed to:

Deeply inspect live traffic: Understand the actual request and response payloads flowing between services, not just what the (often stale) documentation claims.
Reroute a single API call: Make a granular change to just one API endpoint (e.g., POST /checkout/new-feature) to test a new function, while letting the stable cluster version handle all other calls (e.g., GET /checkout/history).
Mock specific network calls: Test failure modes by mocking a specific response (e.g., what happens when the payment-service returns a 503?) without writing a ton of custom mocking code.
Debug and reproduce specific cases: Isolate a single problematic API call to reproduce a bug, without having to change the entire running service.

Technically, most of these things are possible with a service mesh. But as we quickly learned, "possible" is not the same as "practical."

Where the Mesh Falls Short for Developers

This is where the dream collides with reality. To implement that "simple" granular route, a developer must:

Become a Mesh Expert: They have to understand the specific, complex CRDs of their mesh, like Istio's VirtualService, DestinationRule, or the Kubernetes Gateway API constructs like HTTPRoute. They need to learn a platform engineer-centric YAML schema that was designed for cluster-wide operations, not for a developer's temporary workflow.
Suffer High Cognitive Load: Their focus shifts from "How does my feature work?" to "What's the right incantation of YAML to make the network do what I want?" Even with newer standards like the Kubernetes Gateway API, it's still a leaky abstraction that forces developers to understand and manage Kubernetes-native configuration, which is not their primary job.
Risk Breaking Things: In many setups, developers might be editing shared mesh configuration files. A small typo in an Istio VirtualService could inadvertently disrupt traffic for the entire team working in that shared environment. The blast radius for a simple mistake is far too large.

Using mesh-native tools for simple dev tasks felt like using a sledgehammer to crack a nut. We needed a different approach.

Unleashing Developer Superpowers: What We Built

We built a new, programmable layer that sits above the mesh, designed specifically as a developer-first toolkit. It provides two core capabilities to start, enabling a complete, frictionless workflow. This isn't an exhaustive list, but rather the first two powerful tools we chose to build.

1. Real-Time Traffic Discovery: "What's Actually Happening?"

Before you can change an API, you have to understand it. Stale documentation and out-of-date OpenAPI specs don't cut it. Our new layer lets a developer safely tap into a sandboxed version of a service on the cluster and see the real-time flow.

With a simple policy, a developer can get a "read-only" stream of all request and response payloads for a given service in their sandbox. This provides a safe, passive view into the live behavior of the system, letting them work with the ground truth, not just documentation.

2. API-Level Sandboxing: "Surgical Override for Your Local Code"

Once a developer understands the traffic, they can act. This is the killer feature. Instead of redeploying an entire service, the developer runs a single command:

signadot local override --sandbox ... -w ... --p=80 --to=localhost:8000

This command connects their local process (running on localhost:8000) to a specific sandbox's workload port. Here's the magic: our proxy first sends the request to the local process. If that process returns the sd-override header set to true, its response is used (the override is successful). If it doesn't return that header, the proxy seamlessly falls back, using the response from the original workload in the sandbox.

This simple mechanism becomes incredibly powerful when coupled with a new workload type we call "virtual." Unlike "fork" or "local" workloads, a "virtual" workload isn't a separate copy of a service; it's just a pointer to the stable, "baseline" version.

By using the override command against a "virtual" workload, a developer can use their sandbox's routing key (an identifier for their request) to test a single API call against the stable baseline service, but have it served by their local code. This means in just two commands, they can surgically override one API call from the baseline cluster service with code running in their local IDE, all without touching any YAML.

Conclusion: A New Framework for Developer-First Networking

Our journey taught us a critical lesson, and it's the key takeaway we want to leave you with: a vast number of powerful capabilities already exist in our infrastructure, but they aren't exposed at the right level of abstraction. When we build the right developer-first layer to expose them, we can supercharge the developer workflow.

In the modern stack, where teams are building with AI and moving faster than ever, the ability to safely inspect and manipulate live traffic isn't just a convenience; it's a developer superpower. The tools built for networking and operations, like service meshes, are powerful but operate at the wrong abstraction level for this kind of development velocity. The key to unlocking developer productivity is to partner with platform engineers and use a simple, programmable, developer-first layer on top of the mesh to safely expose its power.

The capabilities we've shown today, live traffic discovery and API-level sandboxing, are just the beginning. We've built a general "middleware" framework that allows many more traffic capabilities to be exposed to developers safely. We're on a path to build out a full plugin system, one that empowers platform teams to write and deploy their own custom middlewares, tailored to their organization's specific needs.

Coming in Part 2: Now that you've seen what this layer does, you're probably wondering how it works. In Part 2, we'll dive into the architecture: how we built a high-performance middleware proxy, and integrated it all with different service meshes.

Using Sandboxes with Apollo GraphQL

Signadot — Mon, 03 Nov 2025 15:35:41 +0000

Read this tutorial on Signadot. See git repo here.

Introduction

Introducing schema changes in a federated GraphQL architecture can be tricky. You need a way to safely test and validate new functionality without disrupting baseline traffic or impacting other developers. This is especially important in fast-moving teams where multiple changes can happen in parallel.

In this post, we’ll walk through how to combine Apollo GraphQL with Signadot Sandboxes to create fully isolated environments for testing schema updates tied to pull requests.

Baseline Environment

Before we dive into using Sandboxes with Apollo GraphQL, let’s set up a simple baseline environment.

For this guide, we’ll use the Apollo Supergraph Demo and deploy it to a local Minikube cluster, running side-by-side with the Signadot Operator. This will give us a clean, realistic playground to test how Sandboxes and Apollo can work together.

The demo relies on Apollo Managed Federation and the Apollo Router, but the same approach can be adapted to other Apollo deployments as well. Think of this as a starting point you can evolve to fit your own setup.

Prerequisites

kubectl and minikube installed
An Apollo Studio account
The rover CLI installed and authenticated
A Signadot account with the operator installed in the cluster
The signadot CLI installed and authenticated

Steps

We’ll start by creating a new graph in Apollo Studio. Once the graph is created, export the following environment variables using the values returned by Apollo:

export APOLLO_KEY=<generated-key>
export APOLLO_GRAPH_NAME=<graph-name>
export APOLLO_BASELINE_VARIANT=<graph-variant>

With those values set, you’re ready to prepare your environment. Run the following commands in your terminal:

# Clone the signadot/examples repo
mkdir -p ~/git/signadot/
cd ~/git/signadot/
git clone https://github.com/signadot/examples.git


# Clone the apollographql/supergraph-demo repo
mkdir -p ~/git/third-party/apollographql
cd ~/git/third-party/apollographql
git clone https://github.com/apollographql/supergraph-demo.git


# Build demo images
cd ~/git/signadot/examples/apollographql-tutorial
eval $(minikube docker-env)
./build.sh


# Create and setup all the subgraphs in Apollo Studio
./setup-apollo.sh


# Create the namespace and router secret
kubectl create ns apollographql-demo
kubectl -n apollographql-demo create secret generic router \
    --from-literal=APOLLO_KEY=${APOLLO_KEY} \
    --from-literal=APOLLO_GRAPH_NAME=${APOLLO_GRAPH_NAME} \
    --from-literal=APOLLO_BASELINE_VARIANT=${APOLLO_BASELINE_VARIANT}


# Deploy baseline services + the Apollo Router
kubectl apply -n apollographql-demo -f ./k8s

Validation

At this point, your federated graph should be active and visible in Apollo Studio.You can open the Studio console to confirm that all subgraphs have been successfully registered and are part of the graph.

You can also explore the router’s GraphQL Playground locally using Signadot:

1) Start the local connection:

signadot local connect

2) Open the router playground in your browser: http://router.apollographql-demo.svc:4000/‍
3) Paste and run the following GraphQL query in the editor:

query{
  allProducts {
    id
    createdBy {
      name
      email
    }
  }
}

Using Sandboxes

To safely introduce schema changes without impacting the baseline environment, we’ll leverage Apollo Variants.

The idea is simple: each sandbox corresponds to a dedicated Apollo variant, ensuring that schema changes are fully isolated and don’t interfere with the baseline graph.

In a typical setup, variant creation happens automatically in the CI pipeline, usually triggered by a pull request. To support this, we provide a helper script called clone-variant.sh, which uses the rover CLI to fetch all subgraph schemas from the baseline supergraph and publish them into a new PR variant. From there, any specific changes from the PR can be applied on top.

In this demo, we include an alternative version of the users subgraph, which introduces a few new queries and a phoneNumber field on the User entity (you can find the code here). These changes are what we’ll be sandboxing.

To continue with the example, let’s manually create a PR variant, as if this were happening in the context of PR #1234:

export PR_GRAPH_VARIANT="pr-1234"

cd ~/git/signadot/examples/apollographql-tutorial

# Create the PR variant based on the baseline
./clone-variant.sh ${APOLLO_GRAPH_NAME} ${APOLLO_BASELINE_VARIANT} ${PR_GRAPH_VARIANT}

# Apply the schema changes introduced in the PR
rover subgraph publish "${APOLLO_GRAPH_NAME}@${PR_GRAPH_VARIANT}" \
  --schema ./subgraphs/users/users.graphql \
  --name users \
  --routing-url http://users.apollographql-demo.svc:4000/graphql

Now let’s create a sandbox that forks both the router (to point it to the PR variant) and the users service (to use the modified image with the schema changes), replace <cluster-name> with the corresponding cluster name:

signadot sandbox apply -f - <<EOF
name: apollographql-users
spec:
  cluster: <cluster-name>
  forks:
    # Fork the router pointing it to the PR variant
    - forkOf:
        kind: Deployment
        namespace: apollographql-demo
        name: router
      customizations:
        env:
          - name: APOLLO_GRAPH_VARIANT
            value: "${PR_GRAPH_VARIANT}"

    # Fork the users workload pointing it to the new version of the service
    - forkOf:
        kind: Deployment
        namespace: apollographql-demo
        name: users
      customizations:
        images:
          - image: signadot/apollographql-demo-users-fg:latest
EOF

With this sandbox in place, any request sent using the associated routing key will resolve against the forked users service and the router configured with the PR variant, giving you an isolated environment to safely test the schema changes.

Validation

If you open the router playground again, this time under the context of the sandbox you just created, you’ll see the new phoneNumber field available on the User type, along with the allUsers and user queries.

In this demo, we’re using the Signadot browser extension to easily route traffic through the sandbox.

If you run the same query against the baseline environment, you’ll get an error since the allUsers query (and the new phoneNumber field) don’t exist in that version of the graph.

How it Works

Once the sandbox is created, the request flow changes slightly so that requests carrying the sandbox routing key are routed through the forked workloads instead of the baseline ones.

Here’s how it works step by step:

Requests enter through the DevMesh Proxy in front of the Apollo Router.If the request includes the corresponding routing key, it’s automatically routed to the sandbox fork of the router (in this example, the PR #1234 environment).
The sandboxed router is configured to use the PR variant we created earlier.This means it exposes the updated schema, including the new fields and queries introduced in the forked users subgraph.
To resolve queries, the router uses the original routing endpoints defined in Apollo.However, since the request still carries the routing key, the DevMesh proxy forwards the request to the forked users service instead of the baseline version.
The response is then returned through the same path, giving the client a fully isolated experience of the new schema, all without affecting production traffic.

Alternatives and Improvements

Single Router Mode

In this article we explored integrations using the built-in capabilities of Apollo, where each Router instance can only point to a single graph variant at a time.

Because of this limitation, every sandbox that includes a schema change must also spin up its own Router instance. While this works well operationally, it is sub-optimal from a resource perspective.

Ideally, we’d be able to handle multiple schema variants within a single Router. In that model, the router would dynamically resolve, load, and cache variants on a per-request basis, trading off increased memory usage for reduced infrastructure footprint.

This more advanced (and therefore more complex) setup is technically possible through a native Rust plugin in the Router, which can load and cache multiple supergraph services and dispatch requests to the appropriate variant at runtime.

Though this approach require custom development and add operational complexity, it can reduce the number of running router instances if many short-lived sandboxes are in play.

Variant Reconciler

In the solution described in this tutorial, the PR variant is created at CI time and remains unchanged afterward. However, in a dynamic development environment where multiple changes are happening in parallel, this can lead to schema drift between the baseline and the PR variants.

For example, imagine we’re working on PR #1234. After submitting the PR and creating the variant and sandbox, another PR, say #1233, is merged, introducing changes to the products service. The baseline now includes those updates, but the PR #1234 variant is still based on the old state and doesn’t reflect the new schema.

To address this issue, you can introduce a variant reconciler process. This process would detect changes introduced to the baseline variant and automatically propagate them to existing PR variants, ensuring they stay in sync with the latest baseline. By running this reconciler at the cluster level, you can maintain consistency across all active sandboxes without manual intervention.

Conclusion

Using Signadot Sandboxes with Apollo GraphQL provides a clean, flexible way to test schema changes in isolation without interfering with baseline traffic or blocking other teams. By combining Apollo Variants with sandbox routing, each pull request can get its own dedicated environment, enabling faster iteration and safer releases.

We also explored potential improvements, such as reducing resource overhead by handling multiple variants in a single router, or implementing a variant reconciler to keep PR variants in sync with the baseline as it evolves. These enhancements can make the workflow even more efficient at scale.