DEV Community: Eyal Bukchin

Testing AI-Generated Code in a Shared Kubernetes Environment

Eyal Bukchin — Tue, 13 Oct 2026 06:40:48 +0000

Written by Arsh Sharma. Originally published on metalbear.com.

Research on AI productivity have been finding out that while AI coding tools measurably increase code output, they don't produce a proportional increase in code merged to production. The gap is the feedback loop, specifically in how long it takes to find out whether generated code actually works in a real environment. In our previous post, we looked at how remocal testing — running your service locally while connected to all of its dependencies in a real Kubernetes cluster — closes that loop for individual developers. But closing it for a team is a different problem entirely: if multiple developers and/or AI agents are all running local services against the same staging cluster, what prevents them from interfering with each other?

The shared environment problem

When an AI agent tests generated code against staging, it interacts with other services in the cluster. It sends requests, consumes messages from queues, and reads and writes to databases. In a shared cluster, those interactions can collide.

A few specific scenarios:

Traffic collisions: Two developers both have a locally running instance of the same service to test out code generated by their AI agents. A request comes into the cluster, which they both want redirected to their locally running process to test their code. But which local instance gets it?
Queue interference: An AI agent generates a queue consumer and starts processing messages from the shared queue. Those messages were meant for other consumers.
Database corruption: AI-generated code runs a migration or executes writes against the shared staging database. Everyone's test data is now affected.

The knee-jerk response is to give each developer or AI agent their own isolated environment. But as discussed in the previous post, that trades one problem for another: high infrastructure costs, and environment drift.

mirrord takes a different approach. Rather than isolating at the environment level, it isolates at the interaction level, filtering which traffic reaches your local process, splitting queues so each developer gets their own copy of messages, and branching databases so changes don't affect shared data. This lets the shared staging cluster stay fully operational for everyone, while each developer's local testing stays isolated to their own interactions.

Let’s now look at some mirrord features that make testing AI-generated code in a shared Kubernetes cluster safe.

HTTP traffic filtering

By default, when mirrord steals incoming traffic from the cluster to your local process, it steals all of it. Every request that is meant to hit the target service in the cluster gets forwarded to your local instance. Without a way to filter traffic, stealing all incoming traffic to a local process means the service in the cluster stops receiving requests, breaking the environment for everyone else who depends on it.

mirrord's HTTP traffic filtering lets you route only requests that match specific criteria (a header, a path, or other filters) to your local process. For example, you can configure mirrord to only forward requests containing a header like baggage: mirrord-session=arsh to your locally running instance. Every other request continues to go to the live service in the cluster.

The result is that multiple developers and their AI agents can simultaneously test different changes against the same cluster without worrying about any traffic getting routed to the wrong place.

For the AI agent, this is particularly useful since it can now send requests repeatedly, observe responses, adjust the generated code, and iterate. All of this while interacting with real dependencies but without touching traffic that belongs to anyone else.

Queue splitting

Most modern applications include a message queue like Kafka, SQS, RabbitMQ, or something similar. When an AI agent generates code for a consumer service, testing it against the real cluster creates an immediate conflict because the agent's locally running consumer competes for messages with the existing consumer in the cluster.

If the generated code is wrong, which it often is on the first iteration, it processes or drops messages other services were expecting to receive. Depending on the system, this can cause cascading failures across your staging environment.

mirrord's queue splitting handles this by creating an isolated temporary queue for the local process. The original consumer in the cluster continues processing from the main queue as normal. The locally running service receives its own copy of those messages in a separate queue that only it sees.

This means the AI agent can run, fail, adjust, and re-run without affecting the shared queue or any other consumers. It gets real messages from the cluster to test against, but those messages are exclusively its own.

Database branching

Database interactions are where AI-generated code most commonly breaks in ways that are hard to catch from reading the code alone. An AI agent might generate a query that doesn't match the actual schema, a migration that conflicts with existing indexes, or write logic that violates data constraints. These only surface when the code runs against a real database.

But testing against the shared staging database directly is risky. A bad migration, an accidental mass update, or a schema change can break the environment for everyone.

mirrord's database branching creates an isolated branch of the database for the local process. The branch mirrors the schema and data of the real staging database, so tests run against realistic conditions. But writes, migrations, and schema changes only affect that branch, while the shared database is untouched. This lets AI agents test the full range of database interactions without any risk to the rest of the team.

Additional safeguards

HTTP filtering, queue splitting, and database branching cover the most common collision points. But mirrord includes a few more capabilities that help make testing AI-generated code safer:

mirrord Policies let platform teams define guardrails at the cluster level. For example, you can enforce that locally running AI-generated code can never write to shared databases by default, or restrict the namespaces where services can be targeted for stealing. Policies apply across the team so you're not relying on each developer to configure isolation correctly every time.
Preview environments allow AI agents to deploy code generated for a service as an isolated pod inside the shared staging cluster. This pod runs alongside the stable version of the service and only traffic explicitly routed to it reaches the new pod and everything else continues hitting the live service as normal. This lets you see and share a working version of the changes with stakeholders like product managers, QA, or sales before anything is merged, without needing a separate environment or disrupting anyone else.
mirrord for CI extends the remocal testing model to CI pipelines. Instead of provisioning new environments or relying on mocks for CI tests, your pipeline can run tests directly against the same shared staging cluster. This further reduces your cloud bill and the number of environments you have to manage for testing code.

"It'll never work"

Sharing a staging cluster across a full team (and AI agents on top of that) makes people itch with anxiety. The most common objection we hear to the shared environment approach is: there’s no way an entire team can safely work on the same cluster at the same time. What if one AI agent’s generated code does something that breaks the cluster for everyone else? As a response to that personal dev environments for each AI agent seem like the safest bet.

But in practice, these personal environments don’t hold up because they drift from production, data goes stale, and infrastructure costs compound. And when AI agents are generating and iterating on code faster than any individual developer, the operational overhead of maintaining an environment per agent becomes completely impractical.

The only model that scales in this case is a shared staging cluster where isolation happens at the interaction level, not the infrastructure level. One cluster, equipped with traffic filtering, queue splitting, and database branching, can support an entire team of developers and AI agents simultaneously, without interference. Each person's testing stays contained to their own interactions. And the environment remains operational for everyone else to keep working.

This is what mirrord is built for. Large organizations like monday.com, zooplus, and SurveyMonkey already have hundreds of developers using mirrord concurrently, sharing a Kubernetes environment in their organization every day. And as AI agents start running at higher velocity, generating complete features in minutes and iterating on tests in real time, having this infrastructure in place is what will allow you to truly see the benefits from using AI coding tools.

How Our Engineering Team Uses AI, Part II: Meat Proxies

Eyal Bukchin — Thu, 08 Oct 2026 10:33:43 +0000

Originally published on metalbear.com.

In January we published How Our Engineering Team Uses AI. The tl;dr was that AI was useful around the edges of our work (getting oriented in unfamiliar code, exploring approaches, writing scripts) but not much use at the center of it, because mirrord is a low-level Rust tool with an unusual architecture, and general models couldn't really deal with it. We wrapped up the post by calling AI "a powerful tool around the edges of software development."

Eight months later, we asked the engineering team the same questions again. AI has since moved from the edges to the center: most of the engineers who replied barely write code by hand anymore. Someone still has to read all that code, and so the part of the job that grew the most is review, not only of pull requests, but also the inner loop of reading and correcting (or re-prompting) every chunk of code the agent writes.

I'm not really writing any code, I just review the slop and try to tame the beast.

We heard you like tools

In Part I, tools converged on ChatGPT, Gemini, Claude Code and Cursor. We still don’t mandate any specific tooling, and now there’s even less standardization among what the team uses. Claude Code and Codex are the most common agents, used through the CLI, the desktop apps, or an editor integration. Beyond that, it's a long tail. Some examples:

Claude inside Emacs, through agent-shell, an Emacs package for running coding agents inside the editor.
Amp, a coding agent and development environment that can work in your local checkout from the terminal or on remote machines it spins up per task.
Orca, a desktop app for running several agents in parallel, each in its own git worktree.
pi, a minimal open-source agent harness that runs Codex, Claude and Copilot models from one place and is extended with plugins and your own markdown files.
Graphify, which turns a codebase into a graph agents can query, used to make cheaper Codex models more accurate on our code.
Gemma running locally through llama.cpp, for small chores like commits and rebases.
Zed keeps showing up as well, both for its agent integrations and as a fast, light editor for when a human needs to edit or clean up what the agent wrote.

Cost has entered the equation too. By one engineer's estimate, Fable uses about ten times as much of a usage allowance as Opus, and for their work the quality gain was moderate at best, so they stay on Opus. Another engineer tried to save tokens by having a strong model write a detailed implementation guide and handing it to a cheaper model to write the code, since generating output costs more than reading input (alas, the cheaper models wrote code bad enough to make the savings pointless, so this experiment failed).

Where it helps

People now use agents for pretty much everything: planning, breaking tickets into subtasks, researching bugs, exploring unfamiliar parts of the codebase, writing code (both on the edges and in the core of the product), writing and running tests, pre-reviewing PRs before asking a human to look at it, diagnosing and fixing CI failures, and investigating production incidents. Some use cases in the general consensus:

Finding things by description rather than by name. When you're working with e.g. an unfamiliar Kubernetes data model, you often don't know what the thing you're looking for is called, so you can't grep for it, but you can describe it. Agents are very good at this kind of search.
Code outside the product. Build tooling, bash scripts, and throwaway tests for checking locally how our infrastructure behaves in an edge case (like how a queue handles some unusual sequence of events). The stakes are low, since a mistake breaks a build step or a local experiment rather than something a customer runs. Our xtask build scripts were described as something "no human hand has ever touched."
Status updates. The age-old chore of keeping Linear issues and status posts current can finally be outsourced.
Questions about past decisions. Why a component was designed the way it was, or what was decided about a feature, is often faster to ask an agent than to track down the person who made the call.
Collecting and analyzing messy data. That's how we found our flakiest tests and measured how our CI changes affected run durations.
Work a script should be doing. For repetitive tasks nobody has automated yet, having an agent do the work each time is slower and more expensive than a script would be, but we’re lazy.

Where it still struggles

Rust quality remains a recurring complaint. Agent-written Rust compiles, but it tends to be much longer than it needs to be, with unnecessary .to_owned(), .clone() and .collect() calls that someone then has to go through and remove. Agents are also eager to write tests, and love splitting code into tiny functions so they can add tiny tests for them. Even the engineers who have moved furthest toward AI still evaluate every line and the overall structure of a change themselves, and they often find it wanting.

Agents also tend to reinvent things. Ask one how to solve a problem and it will often come up with its own solution instead of finding the library or well-known approach that already solves it, so its research is a starting point rather than an answer. So most engineers prefer to do the initial research for a new area by hand. The point is for the engineer to understand the problem space deeply enough to make good calls on everything built on top of it.

Writing the PR description/commits makes me look at the problem from a higher level, which ensures I at the very least understand the solution and can judge it appropriately.

The strongest opinions were about writing style. Several engineers write every commit message, PR description and code comment themselves, because agents produce too much of it and too little of it makes sense. Those who let agents draft docs or RFCs usually end up rewriting most of it, because, as one put it, "AI generated text reads like word pulp." But the best argument for doing it by hand is that writing the PR description forces you to look at the problem from a higher level, which can help ensure you understood the solution well enough.

AI generated text reads like word pulp.

We mostly just review now

Asked how their usage had changed since January, one engineer described much more parallelism and much more fatigue:

My week used to be 20-40% reviews, 60-80% code gen, now it's 95% reviews, it's so tiring.

Others described something close to burnout. A task now takes more rounds of reviewing and correcting before it's actually finished, so finishing feels further away than it used to, and with several agents running, you're in that state on several tasks at once. Some refuse the parallelism altogether, running a single agent thread with a narrow purpose and thinking carefully about what comes back.

i only use 1 thread at a time with a very specific purpose and think about things carefully. all those agent orchestrators and stuff are too much for me, it feels like the fastest way to get burnout

The most difficult part is keeping control over quality and maintaining an understanding of how the system works. Even the engineers who no longer write code by hand still go through several review passes after the code works, until they understand and have tested every line. Everyone works hard to not become meat proxies.

We’re trying to find solutions, though, and right now we’re hoping we can successfully automate the first pass. Greptile reviews our pull requests automatically (we’re currently evaluating it against Qodo, using AI to run the evaluation), and many of us have an agent review our own PR before a person sees it.

Building for the agent that will maintain it

There were also changes in how people design things. Some of us now build with the assumption that an agent will maintain the code, which in practice means thorough logging, so that an agent debugging it later has something to read (a courtesy I’m sure many humans would have also appreciated). Some write personal skills that encode their repeated rules and fixes.

That has a company-level version. The January post showed the CLAUDE.md in our mirrord repo, which back then included a long architecture walkthrough. It has since become an AGENTS.md, with CLAUDE.md as a link to it, and it got shorter: the walkthrough was deemed unnecessary and cut, leaving a one-line description of each core component plus the commands for building, testing and linting. It now sits alongside a company-wide one. Everyone at MetalBear imports one shared context repository from their personal ~/.claude/CLAUDE.md, which describes our teams, the tools we use and the known quirks of each, so every agent starts from the same background. The repository is private, but the pattern is simple enough to copy.

One engineer's experiment with agents on a schedule

One of our engineers has been experimenting with agents that run on a schedule rather than on request. One agent triages CI every night. Another picks one small, open bug from Linear every morning and opens a PR to fix it. Tiny but non-critical is a category of work that people never get around to and agents don't mind.

The same engineer also built a support bot that our solutions engineers use to draft answers, which a human then checks and sends. After it confidently described an unverified configuration as "confirmed working in a past similar case," it got two standing rules: search past resolutions before reasoning from scratch, and end every answer with what it is based on.

So what changed?

In January, AI was a tool around the edges of our work. Now, thanks to models improving, our processes maturing, and simply the time it takes workflow changes to propagate, AI writes most of the code for many of the engineers who answered. The work that grew is making sure a person understood that code before it ships. It is certainly more tiring, and hopefully our next post on this will be about how we solved that.

Build vs. Buy Your Kubernetes Development Environment

Eyal Bukchin — Wed, 07 Oct 2026 10:43:26 +0000

Written by Arsh Sharma. Originally published on metalbear.com.

Every conversation about buying a dev tool now includes a brief silence in which someone mentally scopes building it instead. The instinct isn't new, engineers have always been one whiteboard session away from replacing any product with a quarterly project. But Claude Code has compressed that quarterly project into a weekend.

And the truth is, with AI, a capable platform engineer probably could build a working prototype in a sprint: something that connects a developer's code (running on a laptop, in an agent's sandbox, or in CI) and the remote staging cluster, letting them test against real services without deploying every single time. The mistake is thinking that sprint gets you the finished product. Noting the obvious bias we have as a vendor, here's what we've learned from building and maintaining a Kubernetes development platform: what the other 90% of the work looks like, what it actually costs, and how to decide whether buying or building is the right call.

What is mirrord?

mirrord is a Kubernetes development platform that lets developers and AI coding agents test code in a production-like environment before deploying it. Traditionally, giving developers a realistic environment to test in means one of two things: spinning up an ephemeral dev/test environment for each of them or having everyone wait their turn to deploy to staging. Both are slow because every iteration means building images and deploying. Ephemeral environments also get expensive fast since you're duplicating infrastructure for every developer.

mirrord takes a different approach: all developers in the organization share a single staging cluster, and each runs their service wherever they're working, be it locally on their machine, in a CI runner, or in an agent's sandbox. mirrord then proxies incoming and outgoing traffic, environment variables, and files between the local process and its counterpart deployed in the cluster. The running service behaves as if it were deployed since it receives real requests and talks to the cluster's real services, databases, and queues. Because of this, the code behaves as it would in the context of the entire application without actually deploying it. And because everyone shares a cluster that already exists, there's nothing ephemeral to spin up and no duplicated infrastructure to pay for.

At first glance, building what mirrord does might look like a sprint's worth of work. A platform engineer can stitch together a proof of concept with kubectl port-forward and some scripting in a week or two: run the new version of a service beside the deployed one, tag requests with a header, and route only the tagged requests to it. The demo works, the new code is testable against real dependencies without deploying it, and paying for a tool starts to feel absurd when your own team just built the core of one in a sprint.

LaunchDarkly wrote about this exact pattern with feature flags: a feature flag is just an if statement, so building a feature management platform looks trivial until you realize that every capability that makes it actually useful turns out to be its own project, requiring you to invest more and more engineering time into it. Our version of "it's just an if statement" is "it's just a proxy." And in both cases, what people miss is that the demo is not the finished product.

What the other 90% actually costs

Here's what that hidden 90% of work after the initial prototype consists of.

Sharing the cluster is the hard part

We've written extensively about the benefits of shared environments over dedicated ephemeral ones in the past, so we'll assume here that the tool you're building targets a shared cluster, usually staging, like mirrord does.

The tool has two jobs. The first is letting a developer quickly test their locally running code against that production-like cluster, and this is probably what your proof of concept will cover. The second job is letting every developer do this at the same time, against the same cluster, without stepping on each other. This is where the scope explodes, especially because developers aren't the only users anymore. With every developer now launching multiple agents and pointing them to the same cluster to verify the code they just generated, a hundred engineers can easily mean several hundred sessions sharing the cluster.

The conflict everyone scopes for is traffic: two developers can't both receive all the requests hitting a service, so you obviously need HTTP filtering that routes each developer's requests to the right session, and you could say that part is fairly easy to build as well. What usually goes unscoped is everything else that collides. Two local processes compete for the same queue messages, so you need queue-splitting logic that routes each message to the right session. They write to the same database, so you need a way to give each developer an isolated copy. This is why mirrord has features like queue splitting, database branching, and mirrord Policies that make sharing a cluster among thousands of developers possible. Each is a significant engineering effort on its own, and none of them will be in your v1.

Your R&D org pays the real bill

Say your five-person platform team spends six months building this. The visible cost is five salaries for six months plus some ongoing maintenance cost, and depending on your budget, that might look fine. But there's a bigger number. Every day the tool isn't finished is another day your hundred-person R&D org keeps working with the slow dev loop you set out to fix (here's how to put a number on what that costs). The meter isn't just running on the team building the tool, it's running on everyone waiting for it.

Finished isn't adopted

Internal devtools only pay off if engineers actually use them, and the adoption bar is high: the tool should be easy to set up, easy to use, and stable enough that developers trust it with their daily workflow. That trust is hard to earn in the first place because developers don't rush to try untested internal tools. And it's even harder to earn back: a developer who tried v1, hit a rough edge, and went back to their old workflow isn't returning for v2 just because you tell them that things got better. If the experience is clunky, engineers are less likely to file feature requests and more likely to just quietly work around the tool and avoid using it.

This creates an awkward loop for an in-house built tool. You don't reach that quality bar with just a good design document and a prompt. You reach it by iterating against real developer feedback repeatedly, which means that even after you're "done" building, you're not really done. You still need to see what features your team needs that are missing, what parts aren't stable, and constantly build towards them. But that iteration needs developers actually using the tool, and the rough early versions are exactly what teaches them to stay away. It's also the stage AI doesn't really speed up, because the bottleneck is developers hitting friction and telling you about it, not writing the code faster.

Long term support

The work doesn't end once the tool is built, because the ground underneath it never stops moving: Kubernetes ships three releases a year, service mesh upgrades can change how traffic flows, and new protocols show up in your stack. Each of these lands on your team as a support request, and each one forces the same choice, either sink more engineering time into the in-house tool or leave the gap unsupported.

What makes this a losing game internally is who you're doing it for. Every one of those investments serves the same few hundred engineers at your company, so each round of maintenance gets harder to justify against everything else on the platform team's backlog. That exact same work is easy for us to justify, because a fix or a newly supported protocol ships to tens of thousands of engineers across all the companies using mirrord.

Why vibe coding this isn't as easy as it seems

AI has changed the math around the build vs. buy argument: an agent can scaffold a proof of concept in an afternoon instead of a sprint, so building is cheaper than ever. The first half of that is true, and it actually makes the trap worse, because the demo now arrives even faster while the 90% below the waterline still remains a hard problem to solve.

We have some direct experience here. mirrord is a low-level systems tool built in Rust, our engineering team uses AI every day while building it, and we've written candidly about how that actually goes. AI is genuinely useful for things like getting oriented in unfamiliar code, exploring approaches, generating scripts to debug things. But all these require the code to already exist. Vibe coding a tool like this from scratch is a totally different experience. With no existing codebase to ground the model in, no domain experts to review the output, and months of sustained work instead of a single scoped session, you're trusting the model exactly where it's the weakest: short on context, unsupervised, and running for months.

Even with engineers who know the architecture reviewing every line, and CLAUDE.md files we maintain specifically to feed models that context, we've had very few cases of AI shipping a real feature in mirrord without heavy manual intervention.

The TCO comparison

Here's what the two options actually cost. The build numbers come from what we've seen teams scope once they get past the proof of concept.

	Build internally	Buy
Time to a usable v1	6+ months	~1 week
Project maturity at launch	Your engineers discover every failure mode first	Years of hardening across various customers (4+ years and 20,000+ users, in mirrord's case)
Shared-environment support	Another 3-6 months of engineering work.	Already supported through features like DB branching, queue splitting, and traffic filtering
Ongoing maintenance	Your roadmap, indefinitely	The vendor's roadmap, no maintenance on your side
When it breaks	Your team gets paged, your developers wait	A dedicated team that has probably seen the failure mode before

A decision checklist

Questions worth answering with your team before committing either way:

Can you staff it like a product? Not just 3–5 systems engineers for the six month, but permanent ownership afterward including being on-call and supporting upgrades.
Does your v1 scope include more than the happy path? If the plan doesn't mention queue splitting, database isolation, or mesh compatibility, the plan covers only the 10% above the waterline.
What does an outage cost you? If the answer is "every developer is blocked," your internal tool needs production-grade reliability from day one.
Will engineers actually adopt it? If the experience isn't seamless, engineers work around it, and you end up maintaining a tool nobody uses.
Does the plan account for AI agents? Coding agents that test their own changes multiply concurrent sessions and need the same isolation guarantees as developers. Scoping for humans only means rebuilding for agents later.
Do you have a genuinely unusual constraint? Regulatory requirements that no vendor can meet are real reasons to build.

If you answered yes to most of these, building may honestly be the right call for you. For everyone else, the math points the other way.

Where mirrord fits

mirrord lets developers and AI agents run their code on a laptop, in a CI runner, or in an agent's sandbox while connected to a real cluster. Incoming traffic, outgoing calls, environment variables, and files all behave as if the process were running inside the cluster, with no image builds or deployments in the loop. And because a mirrord session is just a CLI command, an agent can run the code it just wrote against the real cluster and verify it actually works, instead of guessing or making assumptions.

The shared-environment features that make up the bulk of the iceberg (queue splitting, database branching, traffic filtering) are already built, which is how teams like monday.com and National Australia Bank (NAB) run it for hundreds of engineers against a single shared environment.

But that's not only what mirrord does. Since this is our main product we've built tons of features on top of it:

Preview Environments: Every pull request spins up a thin preview environment where only the work-in-progress service is duplicated in the cluster as an isolated pod. Reviewers like product managers, QA, or designers get a URL to see real behavior against real services before anything is merged.
mirrord for CI: The same model applied to pipelines. Your PR's code runs in the CI runner but is tested against the real cluster, so there's no ephemeral environment to spin up in CI and no images to build and deploy, which saves teams 20–25 minutes per CI run, with concurrent runs kept isolated from each other and from developers using the same cluster.
Admin Dashboard: A web interface for the platform team running mirrord, with real-time visibility into mirrord sessions, users, targets, CI pipelines, and adoption trends across the organization. Allows platform teams who buy the tools to show adoption, usage, and ROI to stakeholders.

Whichever way your team decides, price the whole iceberg and not just the part above the water.

Frequently asked build vs. buy questions

FAQ

Should I build or buy a Kubernetes development environment?

It depends on whether you can staff it like a product, not just a project. A capable platform team can prototype the core behavior you need in a sprint, but a production-grade version that supports your whole org with features like sharing a cluster across hundreds of developers without collisions, and getting maintained as Kubernetes and your stack evolve, usually takes 6+ months and permanent ongoing ownership. If you have that staffing and a genuinely unusual constraint a vendor can't meet, building can be the right call. For most teams, the math favors buying.

Does AI make it easier to build internal tools in-house?

AI makes the demo faster to reach, not the finished product. AI coding agents are genuinely useful once a codebase already exists: getting oriented, exploring approaches, writing debug scripts. But building a Kubernetes development platform from scratch means months of sustained work with no existing codebase to ground the model and no domain experts reviewing every line, which is exactly the setup AI handles worst. It also doesn't speed up the part of the work that matters most for adoption: iterating on real developer feedback.

What does it cost to maintain an in-house Kubernetes dev tool?

Beyond the initial build, ongoing maintenance means keeping up with three Kubernetes releases a year, service mesh upgrades that can change how traffic flows, and new protocols showing up in your stack, each landing on your platform team as a support request. That work only serves the few hundred engineers at your company, so it competes against everything else on the team's backlog indefinitely. A vendor absorbs that same maintenance across tens of thousands of engineers, which is why it's easier for them to justify.

How does mirrord handle multiple developers sharing one staging cluster?

mirrord uses features purpose-built for this: HTTP traffic filtering routes each developer's requests to their session, queue splitting routes each queue message to the right session, and database branching gives each developer an isolated copy of the data they touch. These are already in use for thousands of developers across companies like monday.com and NAB, rather than something a team has to build and harden on its own.

Auto-verifying your AI-SRE's fixes (Part II): HolmesGPT end-to-end on a real cluster

Eyal Bukchin — Wed, 24 Jun 2026 12:18:12 +0000

Part II of two. See Part I for the recipe.

In Part I we've discussed how you can plug mirrord into your AI-SRE so it can autonomously test its fix in the real cluster. In this post, we'll show this running end-to-end on a real GKE cluster against HolmesGPT, the only OSS AI-SRE we found that is self-hostable. We planted two bugs to test both possible verdicts: PASS when the patched run clears the alert's SLO without any regressions, REJECT when it doesn't (the patched run still violates the alert's SLO, or cleared the SLO but added a regression).

Setup: HolmesGPT investigates an alert and returns a markdown report. A small Claude wrapper turns the report into a code patch. The verifier runs that patch under mirrord exec, compares against an unpatched baseline, and returns a verdict tied to the alert's actual SLO.

What's HolmesGPT? A prebuilt LLM-backed agent tailored for on-call work: it can run kubectl commands, fetch logs from the cluster, and read the runbooks you've already written (in Notion, Confluence, or markdown files) to ground its investigation. Without it, you'd be building this scaffolding around Claude yourself.

The demo cluster

A Python service called checkout handles HTTP requests; on each request it calls a pricing service for the item price. A loadgen pod hits checkout continuously to keep the system warm. Prometheus scrapes checkout's metrics; Alertmanager fires when SLOs are violated. Everything sits in one namespace.

HolmesGPT runs as a pod in the cluster, subscribed to Alertmanager. When an alert fires, HolmesGPT investigates and hands its markdown report off to a separate verifier pod. The verifier runs the bridge (the Claude call that turns the report into a code patch), then uses mirrord exec to run the patched code inside the verifier pod with deploy/checkout's network identity, env, and mounts. When the patched copy calls pricing, it hits the real pricing pod.

Scenario 1: error-rate alert (PASS)

A recent change introduced a request format checkout() doesn't handle: every request for an item_id ending in -3 raises ValueError and returns 500. Loadgen sends a mix; ~10% of requests fail. A Prometheus rule fires CheckoutErrorRateHigh once the error rate climbs over 5%.

Ask HolmesGPT to investigate. This is the call HolmesGPT runs in-cluster when Alertmanager fires:

holmes investigate alertmanager \
  --alertmanager-url http://localhost:9093 \
  --alertmanager-alertname CheckoutErrorRateHigh \
  --model 'anthropic/claude-sonnet-4-20250514'

HolmesGPT investigated for ~30 seconds (pulled pod descriptions, fetched logs, walked the service config) and concluded:

Root Cause: Application logic error causing 500 responses for specific item (item-3)

Error Details: Error rate: 20.09% (above 5% SLO). Specific error: ValueError: unsupported catalog shape for item_id=item-3. Pattern: repeated failures for item-3 checkout requests returning HTTP 500.

Remediation: Fix application code to handle item-3 catalog shape or add proper validation.

Clean diagnosis: HolmesGPT pulled the exception out of the logs and attributed it correctly.

Bridge to a code patch. HolmesGPT outputs a markdown report, not a git diff. The bridging step is a small Claude call running in the verifier pod that takes the report plus the service's source and returns a code patch. We asked it to faithfully implement HolmesGPT's recommendation: handle the item-3 case (Claude chose to return zero) rather than raising. Claude produced the minimal edit to checkout.py.

Verify. Two runs (baseline and patched), 100 requests each. The verifier compares the alert's SLO condition against the patched run, plus a regression watchlist on the other signals. The "baseline" here is the verifier's own load test on the unpatched code, same load and same downstream as the patched run, not the live error rate HolmesGPT observed. That's why the number below differs from the 20.09% in HolmesGPT's report.

SLO under verification: error_rate > 5% (alert fires while this holds)

	Baseline	Patched	SLO threshold	Status
Error rate	10.00%	0.00%	5.00%	✅ satisfies

Regression watchlist:

Signal	Baseline	Patched	Change
p50 latency (ms)	469.2	470.5	+0.3%
p99 latency (ms)	505.7	531.8	+5.2%

Verdict: PASS. Alert condition error_rate > 5% no longer satisfied (10% → 0%). Regression watchlist clean.

Scenario 2: latency alert (REJECT)

The error-rate bug was easy: a literal exception in the logs with an obvious fix. What does the loop look like for a less log-visible bug? We swapped in the second planted defect: fetch_price() has no client-side timeout, and pricing has a long tail: 1 in 10 calls takes ~1.5s. p99 drifts to ~2 seconds, well above the 300ms SLO. CheckoutP99High fires.

Ask HolmesGPT:

holmes investigate alertmanager \
  --alertmanager-url http://localhost:9093 \
  --alertmanager-alertname CheckoutP99High \
  --model 'anthropic/claude-sonnet-4-20250514'

Root Cause Analysis: Latency caused by synchronous dependency calls (each checkout request triggers pricing API calls), no apparent caching, potential bottleneck in pricing service.

Recommendations:

Implement caching for pricing data to reduce API calls

Add async processing for non-critical pricing lookups

Monitor pricing service performance and scaling

Consider request batching to pricing service

Caching is a reasonable optimization. It's also generally a good idea. The question is whether it resolves this specific alert.

Bridge: Of the four recommendations, only caching is a single code change; the others are runtime tuning or architectural refactors. Claude implemented it as an in-memory TTL cache around fetch_price().

Verify.

SLO under verification: p99 latency (ms) > 300

	Baseline	Patched	SLO threshold	Status
p99 latency (ms)	2162.5	2010.3	300.0	❌ violates

Regression watchlist:

Signal	Baseline	Patched	Change
p50 latency (ms)	529.8	0.9	−99.8%
Error rate	0.00%	0.00%	no change

Verdict: REJECT. Alert would still be firing post-merge. SLO p99_ms > 300.0 (baseline 2162, patched 2010).

p50 dropped 99.8% (cache hits return instantly), but the 99th-percentile request is still a cache miss hitting the 1.5s tail. The alert-aware classifier says the thing the on-call needs to hear: the alert that fired would still fire on this code; SLO threshold 300ms; patched value 2010ms.

The fix HolmesGPT didn't propose: a short client-side timeout on fetch_price() to fail-fast under the SLO instead of waiting out the 1.5s tail. We re-ran the bridge with a hand-crafted patch doing that; verifier returned PASS at patched p99 ~330ms. The point isn't that HolmesGPT missed the right fix, it's that the verifier surfaced the gap before the wrong patch shipped.

What the two scenarios show

The loop closes against a real OSS AI-SRE you can install yourself. HolmesGPT investigates, the bridge turns its report into a patch, mirrord runs it against the live cluster, the classifier returns a verdict tied to the actual alert. No integration code from us.

PASS and REJECT both have to be useful or the verifier becomes decorative. Scenario 1 signs off on a real fix. Scenario 2 stops one that looks good (p50 −99.8%) but doesn't clear the alert. Without both, on-call learns to ignore the verifier and starts merging on faith.

Run it yourself

The sample checkout/pricing services, Prometheus rules, verifier code, and bridge script are small (~500 lines of Python plus the ~80-line bridge). The code lives at github.com/metalbear-co/holmes-mirrord-verifier. You'll need the mirrord operator installed in your cluster.

If you haven't read Part I yet, it has the recipe: the pattern, the verification code, and how to wire it into the AI-SRE you're already running.

mirrord: https://metalbear.com/mirrord
Questions: hi@metalbear.com · https://metalbear.com/slack

Auto-verifying your AI-SRE's fixes against your real cluster, with mirrord

Eyal Bukchin — Thu, 11 Jun 2026 11:02:50 +0000

Part I of two. Part II will be published next week, and will show the same loop end-to-end against HolmesGPT on a real cluster.

You're rolling out an AI-SRE (your own homegrown in-house AI loop, or one of the new products: Resolve AI, incident.io's Investigator, Datadog Bits). When an alert fires, it proposes a fix: a timeout here, a retry there, a config bump. The suggestion looks plausible. Do you merge it?

Today, you read the diff, eyeball it against your mental model of the service, and decide, or you deploy to staging and wait. The bugs an AI-SRE gets paged on (an unhandled exception under a rare request shape, a downstream that times out for some requests, a webhook that races with itself, a slow query that only surfaces at scale) live in the gap between local and production. Unit tests and mocks don't reach there. Staging does, but staging is slow (a deploy on every iteration) and shared (you're competing with everyone else trying to reproduce their own bugs).

This is where mirrord can help. mirrord runs a separate copy of your service as if it were a pod in your cluster, wired into the same real downstreams and upstreams without a deploy. This guide shows how to wire that into your AI-SRE so every suggested fix is verified against the real cluster before it reaches a human, automatically, in no time, with no SDK or code to import.

What's an AI-SRE?

A new category of tools that take the first investigative pass when an alert fires, before the on-call human is even awake. The AI-SRE pulls together the context a human would have spent the first ten minutes assembling (recent deploys, related logs, metric anomalies, runbooks, similar past incidents) and proposes a remediation. Most just produce suggestions; a human is still the one acting on them. A few are starting to auto-execute low-risk actions (pod restarts, config flag flips, even small auto-merged code patches).

AI-SREs come in three flavors: standalone (Resolve AI, NeuBird, Cleric), AI bolted onto existing incident or observability tools (incident.io's Investigator, Datadog Bits, Rootly's Copilot, etc.), and homegrown LLM loops, which are the most common shape.

Below are some examples of what an AI-SRE actually outputs. Every one of these is a fix that can't be verified without real cluster state, which is why this guide exists:

Alert	AI-SRE's suggestion	What verification has to prove
Spike of `KeyError: 'address'` in checkout handler	Default the field to `None` and guard the deref	The previously-failing requests now return 2xx, no new errors introduced
Checkout p99 latency above SLO	Add `timeout=300ms` to the pricing-service call	The slow outliers get cut off; p99 actually drops
5xx burst after deploy	Roll back commit `abc123`, or guard the new code path with a feature flag	The error rate stops climbing on real production traffic
Webhook handler creating duplicate orders	Check idempotency key in Redis before insert	A retried webhook produces one row, not two
Kafka consumer lag growing	Bump `max.poll.records`, add a backoff on a flaky downstream	Lag actually drains, downstream stays healthy

None of these can be verified with unit tests or mocks.

Meet `mirrord exec`

mirrord exec runs a local process as if it were a pod in your cluster. Its outbound calls resolve through the target pod's network context, so when your patched code calls payments or inventory or your database, it hits the real in-cluster service. mirrord also routes incoming traffic to the local process when you ask it to. Environment variables, mounted ConfigMaps and Secrets, and the target's filesystem are inherited by default. This is what allows your AI-SRE to test the code fix it came up with instantly and without having to deploy to staging or production. The deployed pod is never touched.

Where mirrord fits

What your AI-SRE already owns

This guide assumes three things about your AI-SRE: it (a) ingests an alert, (b) produces a candidate fix (a code diff, or a recommendation that needs a small bridge step to actually turn it into code, covered in Part II), and (c) has some check for whether the bug is fixed (a repro script, a load test, the alerting query, or a known-good test case).

mirrord doesn't replace any of these. It runs your patched service under mirrord exec, so (c) can hit the real cluster, without a deploy.

How to set it up

Prerequisites

Two things to install:

The mirrord operator in your cluster (install docs), one helm command.
The mirrord CLI wherever the AI-SRE will run the verification step: typically the AI-SRE's own worker, or a sidecar container alongside it (install docs).

Automated verification with `mirrord exec`

When the AI-SRE produces a candidate fix, run a short verification block from inside its own hook:

Check out a working copy of the service code (git clone)
Apply the AI-SRE's patch to it (git apply patch.diff, or whatever your AI-SRE delivers)
Run that patched copy under mirrord exec.

mirrord makes the local process behave as if it were the target pod (same network identity, same downstreams, same env), but the deployed service is untouched.

Run it twice, once with the patch, once without. Both runs see the same live downstreams and upstreams, so the only variable is the patch.

You'll want mirrord in steal mode with a header filter, so the verifier diverts only its own tagged requests to the candidate and leaves production traffic alone. The mirrord config (write once, reuse per run):

// .mirrord/verifier.json
{
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "^X-Verifier-Run: {{ key }}$",
          "ports": [8080]
        }
      }
    }
  }
}

{{ key }} is a mirrord template variable that resolves to whatever you pass via mirrord exec --key, so each verification run can tag itself uniquely without templating the config file.

If your check is a repro script that exits non-zero when the bug is present:

RUN_ID=$(uuidgen)

# Start the candidate; mirrord will steal only requests tagged with $RUN_ID.
mirrord exec --config-file .mirrord/verifier.json --key "$RUN_ID" \
  --target deploy/<your-service> -- ./service/run-server.sh &

# Drive the repro at the real service URL, tagged with the header.
# Other traffic to <your-service> keeps hitting the deployed pods.
./repro.sh https://<your-service>.svc.cluster.local --header "X-Verifier-Run: $RUN_ID"

If your check is metric-based (latency or error rate, the common case for AI-SRE alerts), same shape, swap the repro for a load test:

RUN_ID=$(uuidgen)

mirrord exec --config-file .mirrord/verifier.json --key "$RUN_ID" \
  --target deploy/<your-service> -- ./service/run-server.sh &

./loadtest.sh https://<your-service>.svc.cluster.local --header "X-Verifier-Run: $RUN_ID" > patched-metrics.json

Run that block once on the unpatched code and once on the patched one, diff baseline-metrics.json vs patched-metrics.json, and you have a verdict. The comparison is shaped by the alert: did the SLO condition the alert fires on still hold on the patched run? Plus a sanity check that the other signals didn't regress past tolerance. For an error_rate > 5% alert, check the patched error rate is now under 5%, and that p50/p99 didn't grow past your tolerance. For a p99 > 300ms alert, check p99 is now under 300ms, and that error rate didn't climb. (If your AI-SRE outputs a written report rather than ready-to-apply diffs, you'll also want a small bridge step; Part II walks through one.)

An immediate verdict, against your real cluster. The verifier can also reject. Sometimes the AI's fix moves the metric the alert fires on without moving it enough to clear the SLO, or improves one signal and regresses another. You want to find that out here, not in production.

Wiring it into your AI-SRE

The integration is light. Most AI-SREs already run a hook when they finalize a suggestion (the part that posts the PR comment or the Slack message). In that hook:

Apply the generated patch to a checked-out copy of the service.
Run the verification block above. If it doesn't pass, post the comparison back to the AI-SRE so it can try again (per the diagram). The on-call human keeps working the original alert; they just don't review a fix that didn't survive verification.
If it passes, post the verdict and the diff. A human (or your auto-merge policy) takes it from there.

One caveat on the REJECT-and-retry loop. Today's vendor AI-SREs (HolmesGPT, Resolve AI, incident.io's Investigator) are single-shot. To actually close the loop, put the retry in the small wrapper that turns the AI-SRE's recommendation into a code patch (covered in Part II): feed it the failed run's verdict and ask for another approach. Homegrown AI-SREs can put the retry wherever.

In a noisy incident where your AI proposes ten patches, you spend ten quick verification runs and only the survivors reach a human.

Bonus: hand humans a live preview (mirrord Enterprise)

When verification passes and a human still wants to click around before merging, mirrord preview environments (an Enterprise-tier operator feature) give them an isolated pod from the patched image, reachable at the same service URL with a session header.

mirrord preview start --key fix-$INCIDENT_ID \
  --image registry.example.com/<your-service>:fix-$INCIDENT_ID \
  --target deployment/<your-service> --ttl 30

curl -H 'baggage: mirrord-session=fix-$INCIDENT_ID' \
  https://<your-service>.prod.svc/...

Other traffic keeps hitting the unpatched pods; the header is what routes you to the candidate. After the TTL the operator tears it down. See the preview environments docs for details.

That's the recipe. Part II (coming next week) shows the loop running end-to-end against HolmesGPT, an OSS AI-SRE that ships standalone. Two planted bugs in a real GKE cluster: one the loop signs off on, one the loop correctly rejects.

Why not just stage a copy of the service?

You could run a copy of checkout in a separate namespace, point it at the real pricing, and call that verification. The reasons mirrord exec is a better fit specifically for AI-SRE auto-verification:

Identity matches prod. mirrord exec inherits the target pod's ServiceAccount, OIDC token, and service-mesh identity. A staged copy gets its own. The bugs AI-SREs handle often involve auth, mTLS, or NetworkPolicy interactions where identity is the variable; if your verification runs as a different principal than prod, you can pass on a fix that prod would still reject.
Env, secrets, mounts inherited automatically. mirrord pulls the target pod's environment variables, mounted ConfigMaps and Secrets, and filesystem. Staged copies need a parallel set kept in sync; drift = false verdict. AI-SRE patches frequently touch env-shaped behavior (a missing flag, a stale config), which is exactly the drift class you want verification to catch.
Seconds, not minutes. mirrord exec runs from a checked-out working copy in a few seconds. The staged-namespace approach is build image, push, kubectl apply, wait-ready, port-forward, several minutes per try. In a noisy incident where the AI tries ten patches, that's the difference between "all ten verified in under five minutes" and "one verification per coffee."
Parallel runs are free. Each mirrord exec is its own process. Ten candidate patches can verify concurrently. The staged-copy approach shares a Deployment, Service, and Ingress, so concurrent runs collide unless you also templatize per-run namespaces.

Try it

Setup against any cluster you can kubectl into:

# install the CLI
brew install metalbear-co/mirrord/mirrord

# install the operator in your cluster (Prerequisites above)

# pick any deployment and confirm the loop works
mirrord exec --target deploy/<your-service> -- printenv | head -10

If that prints env vars from the target pod (not your local shell's), the verification primitive is live. Wire your AI-SRE's post-suggestion hook to the pattern in Automated verification above and let the next alert prove itself. Then read Part II (coming next week) for the full walkthrough against HolmesGPT.

mirrord: https://metalbear.com/mirrord
Questions: hi@metalbear.com · https://metalbear.com/slack

Six Claude Code Skills That Close the AI Agent Feedback Loop

Eyal Bukchin — Wed, 13 May 2026 16:56:14 +0000

AI agents write code that compiles, runs locally, and breaks the first time it touches your Kubernetes cluster. The cluster is full of state the model never sees: the env vars on the running pod, the schema in your real Postgres, the headers your upstream auth-service sends, the topics your consumer subscribes to. Without that context, the code an agent writes for your live infrastructure is informed guessing, whether you're shipping a new feature or fixing a regression.

mirrord closes that gap. It runs a local process as if it were a real pod inside your cluster: real env vars, real DNS, real network, optionally real inbound traffic. A real example: Daylight Security pairs Cursor with mirrord for daily development. Their team cut their typical edit-test cycle from 5–8 minutes to about 5 seconds. The reason isn't faster CPUs; it's that the agent now operates against the real cluster the way a senior engineer would, instead of guessing from logs.

We recently shipped six Agent Skills that teach AI agents how and when to use mirrord. The whole bundle installs in one command.

# Claude Code
/plugin marketplace add metalbear-co/skills

# Any Agent Skills consumer
npx skills add metalbear-co/skills

Here's what each skill does, with a concrete prompt that triggers it.

1. `mirrord-quickstart`

Zero-to-first-session for engineers (and agents) who have never used mirrord. Detects your OS, walks through CLI install or VS Code / IntelliJ setup, finds your target pod in the cluster, runs your first session. Your local process can now reach every service, database, and queue in the cluster.

Try: "I'm new to mirrord, help me run my Node app against my staging cluster."

The agent installs mirrord, lists targets in your namespace, picks a likely match, and runs mirrord exec --target … -- node server.js. No copy-paste from docs.

2. `mirrord-config`

Generates and validates mirrord.json, which tells mirrord what to do and where to do it. mirrord's config surface is wide: traffic stealing vs mirroring, filesystem modes, env injection, target selection, database and queue behavior. The skill turns "I want X behavior" into valid config without you opening the docs.

Try: "Steal traffic from pod/api-server, but only requests carrying my baggage header so I don't break anyone else's session."

The agent writes the right config, validates it against the schema, and explains what it does. The interesting part: the skill covers the full mirrord.json surface (target selection, traffic modes, env injection, file system hooks), not just filters. Filtered steal is one of the things that lets multiple developers share one cluster without colliding, but it's only one of the patterns mirrord-config knows how to set up.

3. `mirrord-operator`

Sets up the mirrord Operator for teams. Mirroring traffic from a pod is concurrency-safe out of the box; you only need the operator when multiple developers want to steal the same pod's traffic with different filters, share branched databases, or split a Kafka topic. The operator brokers session boundaries, RBAC, and the routing rules that make those interactions work without collisions.

Try: "Install the operator on our EKS cluster and configure RBAC so only the dev group can use it."

monday.com runs 350+ engineers on a single shared staging cluster this way. The operator is what makes that scale work: concurrent filtered steal so multiple devs share one pod, queue splitting so they share one SQS topic, DB branching so they share one database, RBAC so they don't touch workloads they shouldn't, and the rest of the routing rules that let 350 developers work on the same cluster at the same time.

4. `mirrord-ci`

Run integration tests in CI in isolation against your staging cluster, instead of spinning up an ephemeral test environment for each PR. The service under test runs in the CI runner with mirrord; mirrord steals the cluster traffic destined for it and routes it to your build, so test traffic follows the same path it would in production, with only that one service swapped. That catches the integration bugs mocks miss, with one shared staging cluster instead of one ephemeral cluster per PR.

Try: "Set up GitHub Actions to run our integration tests against the staging cluster."

The agent writes the workflow, injects your kubeconfig from a secret, sets MIRRORD_CI_API_KEY, and wires mirrord ci start around your service and mirrord ci stop in the cleanup hook.

5. `mirrord-db-branching`

Per-developer database branches. Copy-on-write Postgres (or any supported DB), so two engineers can develop against "the same" database without stepping on each other's writes.

Try: "Give me an isolated DB branch off the staging Postgres for this feature."

The agent provisions the branch via the operator, points your local process at the branch, and tears down when the session ends. No more "who deleted the test users?" Slack threads.

6. `mirrord-kafka`

Kafka queue splitting. Each developer gets a slice of the topic that only they consume, while the original consumer keeps running in the cluster. Lets you run a real Kafka workload locally without intercepting messages other people care about.

Try: "Set up queue splitting on the orders.created topic for my local consumer."

The agent configures the operator's Kafka splitter, gives your local process a per-developer consumer group, and confirms message routing.

Install

# Claude Code
/plugin marketplace add metalbear-co/skills

# Any Agent Skills consumer
npx skills add metalbear-co/skills

Repo: github.com/metalbear-co/skills. Issues and PRs welcome; we ship updates fast.

mirrord as an alternative to Telepresence

Eyal Bukchin — Fri, 11 Aug 2023 09:50:22 +0000

A question that comes up often from those already familiar with local Kubernetes development is how mirrord compares to Telepresence. The idea at the base of both products is indeed similar: instead of deploying your new code to the cloud and testing it there, connect it to the cloud from your local machine. By shifting left on cloud testing this way you utilize your cloud environment much more effectively and speed up your development process. The people at Ambassador Labs wrote about it at length.

However, the technical approach at the base of mirrord is very different from that of Telepresence, which translates into significant differences in usability, compatibility, and performance. What Telepresence does is install an operator in your cluster, then connect you to the cluster via VPN (either your entire development machine or a containerized subset). On the other hand, this is what happens when you run a local process with mirrord:

mirrord starts a pod (called the mirrord agent) on the same network namespace as your target. This pod is cleaned up automatically at the end of execution.
mirrord then injects itself into your process, overrides low-level functions and relays them to the mirrord agent, who then executes them on the target in the cluster and sends back the results. For example, when your process tries to read a file, mirrord intercepts that function call and instead asks the agent to read that file from the remote target. mirrord does this for everything - traffic, file operations, environment variables - so your process behaves as if it was executed within the Kubernetes cluster, and with the entire context of your target.

It’s pretty heavy lifting compared to just starting a VPN - mirrord has to hook and reimplement a lot of functions for everything to work smoothly across different frameworks. But there’s a payoff:

By providing your local process with the entire remote context of the pod you chose to target, mirrord natively supports complex flows. For example, if when your process receives a request it queries a database - since mirrord provides it with the pod’s environment variables and files, it’s going to have the necessary credentials, and since mirrord intercepts the outgoing requests and sends it out from the remote pod instead, it won’t be interrupted by any network rules blocking access from outside of the cluster.
mirrord doesn’t need root access on your machine. All it does is override the functions of a running process; the rest of your machine remains untouched
You don’t have to install anything on your cluster - mirrord uses the Kubernetes API, so all you need is a configured kubeconfig. mirrord does create a pod when it runs, but it cleans itself up at the end of execution.
As you might have guessed, mirrord supports traffic mirroring. It can still intercept traffic like Telepresence does, but if you want to leave the remote process completely untouched, mirrord can also just duplicate its traffic and send a copy to you. The original requests are handled by the remote service, so people accessing it are completely unaware that you’re using mirrord to debug it.
As opposed to Telepresence which works at the network level, mirrord works at the pod level. This means it also supports pods that aren’t exposed through a service, and gives your local code access to anything the pod can access, including components outside the cluster. It also natively supports environments with service mesh like Linkerd or Istio.
Because the mirrord client works at the process level, you can easily configure exactly what’s executed remotely and what stays local. For example, you could make all the components of the remote cluster accessible to your local process, but have it write to the database locally; or have it read files from the local file system, except a few specific files that you want to get from the remote pod.
mirrord is a tool for developers, and as such, in addition to a CLI tool, it also comes as an IDE plugin for both VS Code and the IntelliJ IDEs. This is the most popular way to use mirrord, where it naturally fits into the existing development workflow.

Most importantly though, all of these things are supported out of the box and with no required changes to your system. You don’t have to set up mounts, manually ingest environment variables, or create headless services. Just run your process with mirrord, and everything works. It’s this lack of friction that really makes the vision of local Kubernetes development possible - a true shift left of the cloud, where running your code in Kubernetes isn’t the thing you’re saving for last after all the unit tests and local setups, but rather the first thing you do when you write new code.

Usage is simple and intuitive. In the CLI:

And in the IDE - enable mirrord, run or debug your code, and select a target. For example, in VSCode:

Try it out!

If you want to take mirrord for a spin, check out the quick start guide. We’d love to hear about your experience or just general thoughts - chat us up on our Discord or open an issue or discussion on GitHub.

Approaches in Cloud Development Ergonomics

Eyal Bukchin — Tue, 15 Nov 2022 15:47:30 +0000

The advent of microservice architectures and cloud-native has taken some pretty severe tolls on developer ergonomics. It feels as if the tools cloud developers have at their disposal haven’t evolved fast enough to keep up with the rapid progress in infrastructure, and this tool debt is readily apparent in the day-to-day of the modern developer.

The clearest example of this is that it’s now really hard to just run your application. Not the one microservice you’re currently working on, but your actual entire application, which you ultimately deploy to the cloud and serves your customers. You used to be able to fire up your monolith right there on your laptop from within your IDE (crash, change some code, run it again), but architectures today have become resource-intensive, reliant on third-party services, and mostly just plain convoluted to the point where local execution is no longer a viable option.

When we talk about something as elementary as running your application, it naturally applies to many aspects of the developer’s day-to-day: debugging, testing, even just writing new code to see if it works. Therefore, solutions that make your application even a little easier to run provide significant value, just by affecting so many routine activities.
In this article, we’re going to discuss today’s dominant approaches to making microservice applications easier for developers to run, and we’re also going to make the case for our own thing - namely, a single shared environment in the cloud that the organization maintains, and that developers can plug in and out of non-intrusively as they develop their microservices.

Contemporary Cloud Development Workflows

In the frictionless microservice utopia, the development workflow goes something along these lines:

Microservices are independent and new functionality can be developed one microservice at a time
Microservices talk to each other via well-defined contracts which can be tested in isolation
Databases, queues, and third-party services consumed by a microservice also expose a consistent and well-defined contract, and as such can be mocked for testing purposes

In other words, a service can be tested on its own, and then deployed to the production environment where it would play nicely with all the other components.

However, let's assume you’re a microservice developer who doesn’t work in an organization where everyone designs perfect contracts and writes rigorous tests (or they do, but the third-party managed services you rely on don’t). When writing new code, you probably make as much progress as you can within the scope of the one or two microservices you’re working on at the moment, but then you do one or more of the following:

Wrangle your entire application - microservices, databases and all - to run on your local machine
Deploy your new code to a personal persistent/ephemeral development environment in the cloud
Deploy your new code to a shared staging environment and test it there

Run Locally

This approach works great when it’s feasible, which is usually at a very early stage in the life of the application where it’s still small and tenable. There’s some tooling that lets you extend this honeymoon phase by letting you do it more easily, like docker-compose, Skaffold, or Tilt. However, at a certain point, even if you’ve written whatever scriptage is needed to actually configure and run the latest stable version of all of your components together, you’re going to hit some sort of ceiling: if you’ve got a large database, or some CPU-heavy computations, or you’re relying on some managed service that can’t be containerized, this approach soon becomes untenable.

Pros:

Convenient development on your local machine
No reliance on network or external services

Cons:

Difficult/impossible for large applications (ones with e.g. many microservices, a large database, high bandwidth usage)
Doesn’t support hard-to-containerize components, like managed services

Personal Cloud Environment

With Infrastructure as Code at its current state of maturity, it’s now easier than ever to replicate microservice environments in the cloud. This unlocked a new approach of having a personal production-like cloud environment for every developer, which they can use freely and in isolation. It comes in two flavors - persistent environments, or ephemeral environments created on demand with products like Okteto or Bunnyshell (also sometimes called Environment as a Service)¹. This approach overcomes the resource limitations of the local environment but substitutes them for some new difficulties:

For persistent environments, cost (determined by the size of the environment times the number of developers) becomes a factor. As the company grows, so do its architecture and the number of developers, leading to soaring cloud bills
To generate state, database seed files are used. These need to be maintained so they don’t become stale
You have to work against a cloud environment, which isn’t always fun (though some tools make it easier, like GitHub Codespaces or Gitpod)
Same as with running your application locally, having hard-to-containerize components in your architecture like managed services can make this approach unfeasible.

Pros:

Not capped by the hardware limitations of your development machine
Better simulation of your production environment

Cons:

No production-like database state
Local changes have to be synced to the cloud
Doesn’t support hard-to-containerize components, like managed services

Shared Cloud Environment

A lot of organizations have already put in the effort of setting up a single, shared production-like environment for testing - the staging environment. It’s persistent and has already been set up manually to include all components, including third-party services, databases with full state (some companies even automatically stream data from production), and all the microservices. In terms of simulating production, it’s hard to do much better.

However, since this environment is shared by multiple developers, some measures are usually in place to keep it stable - for example, you can only actually deploy new code through the CI, often meaning you have to wait for a code review and your entire suite of automated tests to run (not to mention the deployment script itself) before it actually gets there. Even then, the tests aren’t perfect, and sometimes you end up deploying broken code and corrupting the environment, much to the chagrin of your fellow developers.

This is where shared environment tools like Telepresence and CodeZero can help. They assume you're only working on one or two microservices anyway, and running them locally is not an issue. These tools let you connect your local service to the staging environment, replacing the service currently running in the cluster, without deployment. The code you're working on runs locally, and its dependencies run in the cloud. Not only can you keep working on your local environment, you can iterate on testing against your cloud environment without waiting for lengthy CI and deployment processes.

Pros:

Leverage a single shared environment, making it feasible for you to manually set up database state and managed services (since you only have to do it once, for one environment)
You can still work locally within your IDE
Code changes happen locally, so iterations are fast (no need to deploy to the cloud)

Cons:

One developer can break the environment for the others
If the environment is corrupted, it’s hard to recover

The Sell

So local setups are easy to use but don’t support large deployments; personal cloud environments support large deployments but don’t really reflect production, don’t support components that can’t be containerized, and cost a lot; and shared environments reflect production for cheap, but don’t really support concurrency without sacrificing stability.

mirrord lets you have all of these cakes and eat them too. We essentially took the shared environment approach and tacked on isolation. We do this by:

Letting developers mirror incoming traffic rather than just intercept it. This lets multiple developers work on the same service in the cloud, while the service is still running and handling requests.
Giving developers fine-grained control over what happens locally and what happens remotely. This way, for example, incoming traffic mirroring and file reads can happen remotely, but outgoing traffic and file writes can be local. Future versions of mirrord will include more advanced modes of configuration, letting users filter by e.g. hostname or protocol.
In future versions, supporting advanced features like copy-on-write for databases.
In our upcoming cloud solution, providing RBAC, letting the organization control which teams have access to which parts of the staging environment.

By using mirrord, developers can run the one service they’re working on locally, without juggling configurations or straining their development machine; they can run it against a full environment with mature database state and managed services; and they can do it quickly, easily and without risk.

We’ve recently released our first stable version, and would love to hear your thoughts (on GitHub or Discord).

It can also be supplemented with a cloud development environment, as provided by products like Gitpod or Codespaces. For a comprehensive survey of the non-localhost space see this excellent article. ↩

(Re)Introducing mirrord (or: the War on the Long Dev Loop)

Eyal Bukchin — Tue, 31 May 2022 09:12:57 +0000

Originally posted on the MetalBear blog.

The Dev Loop (or: know your enemy)

Imagine you’re a backend developer at post series B SaaS soonicorn. You’ve spent half a sprint adding a new feature to your microservice. You’ve thoroughly researched possible inputs and database states and built an elaborate test suite. Your code was mercilessly reviewed by two of your teammates. Finally, the tests pass, the pull request is approved, and as a final verification, you deploy your new code to the staging environment…

Where it crashes almost immediately. Turns out you forgot to test some obscure configuration, or some complex database state that would only ever occur on a mature, complex environment - like staging, and unlike your local machine.
So you roll the service back on staging, fix the bug, write some more tests, get it reviewed again, build the code, deploy it to staging… and it crashes again. While you were fixing the bug, your teammate deployed a new version of some other microservice - except they're not as thorough a tester as you are, and the new version is broken, so you can’t test your own service. You wait for them to roll their service back to a working version, the sprint is long over, and you’ve started on your next feature during your downtime so now you’re dividing your focus between two things.

The worst part being that you know it’s going to happen again next sprint.

mirrord (or: how you win)

I’ve encountered some variation of the dev loop at every startup I’ve ever worked in, and it’s this pain exactly that we’re trying to solve with mirrord. mirrord runs in two places - on your local development machine, and in your staging environment. By wrapping your local process and hooking system calls, it lets you plug your process non-invasively into that service’s instance on the staging environment.
Simply put, mirrord lets you run a local process in the context of your cloud service. This means you can test your code on staging, without actually deploying it there. Not only does it save you the hassle of deployment every time you make a change, but it also keeps the staging environment free from untested versions, and constantly usable for everyone else in the organization.

Getting Started (and: other practical info)

Today we’re releasing mirrord 2.0, which supports incoming traffic. That means whatever traffic reaches your service in staging is duplicated by mirrord and sent to your local process. We’ll soon be adding support for outgoing traffic, file access, and environment variables - everything needed for your local process to “think” it’s running on the staging environment.
All you need to run mirrord is kubectl access to your staging environment¹. You can use it as either a CLI tool, or an extension for VS Code (IntelliJ support is on the way). Try it out here, and let us know what you think at hi@metalbear.co, or in our repo!

note that mirrord doesn’t install anything persistent in your Kubernetes cluster. It runs a job that self-deletes when mirrord terminates. ↩

DEV Community: Eyal Bukchin

Testing AI-Generated Code in a Shared Kubernetes Environment

The shared environment problem

HTTP traffic filtering

Queue splitting

Database branching

Additional safeguards

"It'll never work"

How Our Engineering Team Uses AI, Part II: Meat Proxies

We heard you like tools

Where it helps

Where it still struggles

We mostly just review now

Building for the agent that will maintain it

One engineer's experiment with agents on a schedule

So what changed?

Build vs. Buy Your Kubernetes Development Environment

What is mirrord?

What the other 90% actually costs

Sharing the cluster is the hard part

Your R&D org pays the real bill

Finished isn't adopted

Long term support

Why vibe coding this isn't as easy as it seems

The TCO comparison

A decision checklist

Where mirrord fits

Frequently asked build vs. buy questions

FAQ

Should I build or buy a Kubernetes development environment?

Does AI make it easier to build internal tools in-house?

What does it cost to maintain an in-house Kubernetes dev tool?

How does mirrord handle multiple developers sharing one staging cluster?

Auto-verifying your AI-SRE's fixes (Part II): HolmesGPT end-to-end on a real cluster

The demo cluster

Scenario 1: error-rate alert (PASS)

Scenario 2: latency alert (REJECT)

What the two scenarios show

Run it yourself

Auto-verifying your AI-SRE's fixes against your real cluster, with mirrord

What's an AI-SRE?

Meet mirrord exec

Where mirrord fits

What your AI-SRE already owns

How to set it up

Prerequisites

Automated verification with mirrord exec

Wiring it into your AI-SRE

Bonus: hand humans a live preview (mirrord Enterprise)

Why not just stage a copy of the service?

Try it

Six Claude Code Skills That Close the AI Agent Feedback Loop

1. mirrord-quickstart

2. mirrord-config

3. mirrord-operator

4. mirrord-ci

5. mirrord-db-branching

6. mirrord-kafka

Install

mirrord as an alternative to Telepresence

Try it out!

Approaches in Cloud Development Ergonomics

Contemporary Cloud Development Workflows

Run Locally

Personal Cloud Environment

Shared Cloud Environment

The Sell

(Re)Introducing mirrord (or: the War on the Long Dev Loop)

The Dev Loop (or: know your enemy)

mirrord (or: how you win)

Getting Started (and: other practical info)

Meet `mirrord exec`

Automated verification with `mirrord exec`

1. `mirrord-quickstart`

2. `mirrord-config`

3. `mirrord-operator`

4. `mirrord-ci`

5. `mirrord-db-branching`

6. `mirrord-kafka`