<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: MetalBear</title>
    <description>The latest articles on DEV Community by MetalBear (metalbear).</description>
    <link>https://dev.to/metalbear</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Forganization%2Fprofile_image%2F10935%2F5e6a0e22-733f-4582-8243-d0dfdbc47bba.png</url>
      <title>DEV Community: MetalBear</title>
      <link>https://dev.to/metalbear</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/metalbear"/>
    <language>en</language>
    <item>
      <title>Stop Deploying to ECS Just to See If Your Code Works</title>
      <dc:creator>Eyal Bukchin</dc:creator>
      <pubDate>Fri, 20 Nov 2026 09:09:18 +0000</pubDate>
      <link>https://dev.to/metalbear/stop-deploying-to-ecs-just-to-see-if-your-code-works-3fc4</link>
      <guid>https://dev.to/metalbear/stop-deploying-to-ecs-just-to-see-if-your-code-works-3fc4</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://metalbear.com/blog/stop-deploying-to-ecs/" rel="noopener noreferrer"&gt;metalbear.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;On ECS, the usual way to find out whether a change works is to deploy it. You build the image, push it, register a new task definition revision, wait for the rollout and look at what happened, then do it all again if it's wrong. A loop like that might have been tenable for humans, but a coding agent writes a change in seconds, then waits for a full rollout to learn whether it worked, and it rarely gets there on the first try. Most of the speed you got from the agent ends up spent on waiting for rollouts.&lt;/p&gt;

&lt;p&gt;Today we're launching mirrord for ECS. Your code can now run on your laptop, in your CI runner or in your coding agent's sandbox as if it were a live task in your staging environment, with the task's network, environment variables and IAM task role. It reaches the same databases, queues, and internal and external APIs as the deployed service. Only the requests that carry your session header come to it, so you, your teammates and your agents can each run your own version of the service at the same time, on the staging environment you already share. You find out whether a change works by simply running it, without deploying anything or spinning up an environment of your own.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this didn't exist on ECS
&lt;/h2&gt;

&lt;p&gt;Every tool in this category, Telepresence, Signadot and mirrord included, was built on Kubernetes, and leans on it for two main things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Kubernetes lets a tool put its own component next to your service&lt;/li&gt;
&lt;li&gt;The Kubernetes API gives a developer's laptop a way to reach that component, controlled by RBAC&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;ECS has none of it. You can't add a container to a running task, and there's no hook that adds one when a task starts. AWS gives you ways to open a shell or forward a port into a task, but nothing that sends the task's traffic out to your machine, and no shared API that a laptop and a task can both reach and authenticate against. To make mirrord work on ECS, we had to build our own way into the task, and our own way to connect your machine to it and authenticate both sides.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we built
&lt;/h2&gt;

&lt;p&gt;You add a small setup container to each service's task definition and deploy it once. It stays in place from then on, so you don't redeploy to test changes, only to update mirrord itself. When the task starts, the setup container loads mirrord into your app container, where it runs with the app's network and environment. Your image stays the same.&lt;/p&gt;

&lt;p&gt;To connect the two sides, your machine and the task each make an outbound HTTPS connection to a component we call the sessions-manager, which pairs them and relays traffic between them. Neither your machine nor the task has to accept an inbound connection for this, so you don't open any new ports.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw5qs09g7uzdfvmnw0zt3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fw5qs09g7uzdfvmnw0zt3.png" alt="Architecture diagram: a client calls a public load balancer, which routes to an ECS task on Fargate inside an AWS VPC. mirrord runs inside the task's app container, next to the unchanged orders app (v1), which talks to an RDS Postgres database with no public endpoint. The task and the developer's machine or agent sandbox, running orders app v2, each connect out over HTTPS to the sessions-manager in MetalBear Cloud." width="800" height="359"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;mirrord for ECS with the sessions-manager in MetalBear Cloud. mirrord runs inside your app container, and both the task and your machine connect out to the sessions-manager, so your local code reaches the private database through the task.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You choose where the sessions-manager runs: in a Kubernetes cluster of your own, or hosted in MetalBear Cloud. Either way, session traffic passes through it over HTTPS, and both your tasks and your developers have to authenticate to it. The &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/beyond-kubernetes/metalbear-cloud/" rel="noopener noreferrer"&gt;security details are in the docs&lt;/a&gt;.&lt;/p&gt;
&lt;h2&gt;
  
  
  What it looks like after setup
&lt;/h2&gt;

&lt;p&gt;From there, you or your agent start the service locally with &lt;code&gt;mirrord up&lt;/code&gt;, then send requests with your session header to the service's usual staging address:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mirrord login
mirrord up --key &lt;span class="nv"&gt;$USER&lt;/span&gt;

&lt;span class="c"&gt;# from another terminal, or from your agent&lt;/span&gt;
curl -H &lt;span class="s2"&gt;"baggage: mirrord-session=$USER"&lt;/span&gt; https://orders.staging.example.com/orders
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You pick the session key with &lt;code&gt;--key&lt;/code&gt;. Requests to staging that carry it go to your machine, and everyone else's requests still go to the deployed task. Your code's outgoing calls leave from inside the task, so they reach the private databases and internal APIs that only the task can reach, and your local process runs with the task's environment and IAM task role. When something's wrong, you just change the code and run it again, with nothing to build or roll out in between.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fln6y81ilnxoor8v9iwdi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fln6y81ilnxoor8v9iwdi.png" alt="Request routing diagram: two requests go through the load balancer to the ECS task. mirrord in the app container checks each one. The request with the baggage header mirrord-session set to your key goes through MetalBear Cloud to the orders app v2 on your machine or agent sandbox. The request without the header is served by the deployed orders app v1." width="800" height="280"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;mirrord checks each request's session header. Requests with your key go to the code on your machine or in your agent's sandbox; every other request is served by the deployed task as usual.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Start at &lt;a href="https://metalbear.com/mirrord/ecs/" rel="noopener noreferrer"&gt;metalbear.com/mirrord/ecs&lt;/a&gt;. The &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/beyond-kubernetes/ecs-sample-app/" rel="noopener noreferrer"&gt;ECS sample app&lt;/a&gt; uses Terraform to deploy a small service next to a private RDS database, then walks you through running that service locally against the database.&lt;/p&gt;

&lt;p&gt;If you're on another platform, Lambda, Cloud Run, Vercel and Cloudflare are on &lt;a href="https://metalbear.com/mirrord/beyond/" rel="noopener noreferrer"&gt;the waitlist&lt;/a&gt;, and you can tell us there which one you need.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>devops</category>
      <category>ai</category>
      <category>testing</category>
    </item>
    <item>
      <title>Best Microservices Testing Tools for Kubernetes</title>
      <dc:creator>Arsh Sharma</dc:creator>
      <pubDate>Mon, 02 Nov 2026 12:31:24 +0000</pubDate>
      <link>https://dev.to/metalbear/best-microservices-testing-tools-for-kubernetes-2bmn</link>
      <guid>https://dev.to/metalbear/best-microservices-testing-tools-for-kubernetes-2bmn</guid>
      <description>&lt;p&gt;&lt;strong&gt;TLDR:&lt;/strong&gt; There is no single best tool for microservices testing. You need a stack that covers several layers: a test environment (mirrord, Signadot, Telepresence, Okteto, Tilt), test orchestration (Testkube), API testing (Postman), service mocking (WireMock), contract testing (Pact) and load testing (k6). The environment layer matters most, because it decides whether every other tool's results can be trusted or not.&lt;/p&gt;

&lt;p&gt;When teams ask us about microservices testing, the conversation usually starts with tools, when it should start with environments. Testing a monolith is mostly a question of writing good tests. Testing microservices on Kubernetes is a question of &lt;em&gt;where&lt;/em&gt; those tests run, because a microservice can't truly be verified without the services, databases and queues it talks to. Every common workaround has a cost:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Mocks drift away from how the real services behave.&lt;/li&gt;
&lt;li&gt;A shared staging environment turns into a queue where one broken deploy blocks everyone.&lt;/li&gt;
&lt;li&gt;Spinning up a full ephemeral environment for every pull request is slow and gets expensive quickly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And with &lt;a href="https://www.cncf.io/announcements/2026/01/20/kubernetes-established-as-the-de-facto-operating-system-for-ai-as-production-use-hits-82-in-2025-cncf-annual-cloud-native-survey/" rel="noopener noreferrer"&gt;59% of organizations reporting that much or nearly all of their development and deployment is now cloud native&lt;/a&gt; (CNCF Annual Cloud Native Survey 2025), this is a problem most engineering teams now have, not a niche one.&lt;/p&gt;

&lt;p&gt;It hits platform and DevOps engineers who own the clusters, backend developers who need fast feedback, and engineering leaders who pay for all of the above. AI coding agents make it worse. &lt;a href="https://blog.jetbrains.com/research/2026/08/how-much-code-do-developers-really-let-agents-write/" rel="noopener noreferrer"&gt;JetBrains’ 2026 research&lt;/a&gt; found that more than half of all developers now write less than 20% of their code manually, and one in five writes absolutely no code without AI help. Agents produce changes faster than teams can validate them.&lt;/p&gt;

&lt;p&gt;At MetalBear we build mirrord, which connects AI agents and developers to a live cluster so they get real-world context and a real feedback loop, so a lot of our time goes into working through exactly this problem with platform teams at companies like &lt;a href="https://monday.com" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt;, Podium and SurveyMonkey. Many of the tools below come up in those conversations, either as alternatives teams evaluated or as other parts of their testing stack, which is where most of the comparisons in this guide come from.&lt;/p&gt;

&lt;p&gt;In this guide, we'll compare 10 tools grouped by the layer of testing they cover, show them side by side in one table, list a few more worth knowing, and then look at how to combine them into a stack that fits your team.&lt;/p&gt;

&lt;h2&gt;
  
  
  Microservices testing tools compared at a glance
&lt;/h2&gt;

&lt;p&gt;The tools below appear in the same order as in the rest of this article, grouped by category, starting with test environments.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;Category&lt;/th&gt;
&lt;th&gt;Isolation model&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;th&gt;Best fit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;mirrord&lt;/td&gt;
&lt;td&gt;Test environments&lt;/td&gt;
&lt;td&gt;Per-session isolation in one shared cluster (&lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/incoming-traffic/filter-incoming-traffic" rel="noopener noreferrer"&gt;traffic filters&lt;/a&gt;, &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting" rel="noopener noreferrer"&gt;queue splitting&lt;/a&gt;, &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/db-branching" rel="noopener noreferrer"&gt;DB branching&lt;/a&gt;)&lt;/td&gt;
&lt;td&gt;Open source free; Team $50/seat/mo (20% less billed annually); Enterprise custom&lt;/td&gt;
&lt;td&gt;Many developers, CI runs and AI agents testing on one staging cluster at the same time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Signadot&lt;/td&gt;
&lt;td&gt;Test environments&lt;/td&gt;
&lt;td&gt;Sandboxes of changed services in a shared cluster, routed by request headers&lt;/td&gt;
&lt;td&gt;Starter free; Business $250/mo; Enterprise custom&lt;/td&gt;
&lt;td&gt;Isolated sandboxes per pull request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Telepresence&lt;/td&gt;
&lt;td&gt;Test environments&lt;/td&gt;
&lt;td&gt;Entire local machine connected to a remote cluster through an in-cluster Traffic Manager and Traffic Agents&lt;/td&gt;
&lt;td&gt;Free, open source&lt;/td&gt;
&lt;td&gt;Individuals or teams that want a fully open-source tool with no vendor behind it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Okteto&lt;/td&gt;
&lt;td&gt;Test environments&lt;/td&gt;
&lt;td&gt;Full environment per developer or per pull request in its own namespace&lt;/td&gt;
&lt;td&gt;Starter free (5 seats); Scale and Enterprise custom&lt;/td&gt;
&lt;td&gt;Personal ephemeral environments per developer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tilt&lt;/td&gt;
&lt;td&gt;Test environments&lt;/td&gt;
&lt;td&gt;Full stack you define, usually on a local cluster&lt;/td&gt;
&lt;td&gt;Free, open source&lt;/td&gt;
&lt;td&gt;Local Kubernetes inner loop&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Testkube&lt;/td&gt;
&lt;td&gt;Test orchestration&lt;/td&gt;
&lt;td&gt;In-cluster runner; tests run as Kubernetes workloads against whichever environment you target&lt;/td&gt;
&lt;td&gt;Open-source agent free; Pro $700/mo per runner agent; Enterprise custom&lt;/td&gt;
&lt;td&gt;Platform teams that want all test types run and tracked in one place&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Postman&lt;/td&gt;
&lt;td&gt;API testing&lt;/td&gt;
&lt;td&gt;Desktop or web client, plus a CLI in CI, calling whichever environment you point it at&lt;/td&gt;
&lt;td&gt;Free; Solo $9/mo; Team $19/user/mo (billed annually); Enterprise custom&lt;/td&gt;
&lt;td&gt;Teams that want shared API tests that both developers and QA work with&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WireMock&lt;/td&gt;
&lt;td&gt;Service mocking&lt;/td&gt;
&lt;td&gt;Mock services (in-process, standalone, Docker or cloud) standing in for a dependency&lt;/td&gt;
&lt;td&gt;Open source free; Cloud free tier (1,000 calls/mo); Enterprise quote&lt;/td&gt;
&lt;td&gt;Third-party APIs that are rate-limited or expensive to call&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pact&lt;/td&gt;
&lt;td&gt;Contract testing&lt;/td&gt;
&lt;td&gt;Consumer tests run against a Pact mock; providers verify recorded contracts in their own build; no shared environment needed&lt;/td&gt;
&lt;td&gt;Open source free; PactFlow Starter free, Team from $115.42/mo (billed yearly), Enterprise custom&lt;/td&gt;
&lt;td&gt;Organizations where many teams release services independently&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;k6&lt;/td&gt;
&lt;td&gt;Performance testing&lt;/td&gt;
&lt;td&gt;Load generator run locally, in CI, in-cluster via the k6 Operator, or in Grafana Cloud&lt;/td&gt;
&lt;td&gt;Open source free; Grafana Cloud k6 includes 500 VUh/mo free, then from $0.15/VUh&lt;/td&gt;
&lt;td&gt;Teams that want load tests as code, with pass/fail gates in CI&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Pricing checked on each vendor's official pricing page on October 14, 2026.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What does a microservices testing stack need to cover?
&lt;/h2&gt;

&lt;p&gt;The best tools for testing microservices architecture work as a stack, because no single tool covers every layer of microservices testing. A reliable stack combines several tools, and the environment they run in determines whether their results can be trusted.&lt;/p&gt;

&lt;p&gt;These are the six layers we'll use throughout this article:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Test environment:&lt;/strong&gt; where the service under test runs, whether its dependencies are real, mocked or copied, and how production-like the whole setup is.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test orchestration:&lt;/strong&gt; running and scheduling tests inside the cluster and collecting the results in one place.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;API and integration testing:&lt;/strong&gt; verifying a service's endpoints and its interactions with other services.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Service mocking:&lt;/strong&gt; replacing dependencies that are unavailable, unstable or expensive to call, also known as service virtualization.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Contract testing:&lt;/strong&gt; catching breaking changes between consumer and provider services before they're deployed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Performance testing:&lt;/strong&gt; checking how individual services and chains of services behave under load.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;End-to-end and browser frameworks like Playwright, Cypress and Selenium run on top of these layers. They're important, but they depend on an environment to point at, so we aren't covering them as separate entries here. Unit testing also isn't covered, since it doesn't change much between a monolith and microservices.&lt;/p&gt;

&lt;p&gt;Why treat the environment as the foundation? Because every other layer inherits its fidelity. An API test that passes against a mock tells you the code works against the mock, not that it works with the real services. A shared staging environment gives you real dependencies, but two people testing at once can overwrite each other's data or consume each other's queue messages, which leads to queuing up for your turn and long wait times as every change is tested independently. Duplicating the environment per pull request solves the conflicts but multiplies cloud cost and adds minutes of spin-up time to every change. And flows that go through message queues are the hardest to test before production: in a shared environment, a test consumer would pull messages meant for other consumers, and giving every test run its own broker is a lot of setup.&lt;/p&gt;

&lt;p&gt;Here's how the 10 tools map onto those layers:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fciawbeu1gj0lyyhygstf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fciawbeu1gj0lyyhygstf.png" alt="Best tools for microservices testing on Kubernetes, mapped to the stack: environment, orchestration, API, contract and load testing tools" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How we selected and ranked these tools
&lt;/h2&gt;

&lt;p&gt;We selected and ordered the tools using the same criteria for each, so the list can be compared fairly.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Selection criteria:&lt;/strong&gt; fit for Kubernetes-based microservices, how faithfully the tool reflects real dependencies, CI/CD readiness, and cost and maintenance effort.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Order of the list:&lt;/strong&gt; test environments first, then test orchestration, then tools for individual test types. Environments come first because they determine whether the results of everything else can be trusted. The category label under each heading tells you which layer a tool belongs to.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sources:&lt;/strong&gt; each tool's official documentation and pricing page, last checked October 14, 2026.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Transparency:&lt;/strong&gt; MetalBear builds mirrord. Every tool on this list, mirrord included, is described using the same criteria and only verifiable facts.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  1. mirrord: best for testing against a real cluster without deploying
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Test environments&lt;/p&gt;

&lt;p&gt;&lt;a href="https://metalbear.com/mirrord/" rel="noopener noreferrer"&gt;mirrord&lt;/a&gt; lets you run a service locally, in an agent's sandbox, or in a CI runner, as if it were running inside your Kubernetes cluster. It's best for teams that want to test against the real services, databases and queues in a shared cluster, without deploying code or maintaining copies of the environment. Developers use it for local development, but the same approach also runs in &lt;a href="https://metalbear.com/mirrord/docs/use-cases/mirrord-for-ci" rel="noopener noreferrer"&gt;CI&lt;/a&gt;, powers &lt;a href="https://metalbear.com/mirrord/docs/use-cases/preview-environments" rel="noopener noreferrer"&gt;preview environments&lt;/a&gt;, and gives AI agents a place to verify their own changes.&lt;/p&gt;

&lt;h3&gt;
  
  
  How mirrord works
&lt;/h3&gt;

&lt;p&gt;When you start a mirrord session, mirrord connects your local process to a target in the cluster, like a deployment or a pod. Your process receives a copy of that target's &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/incoming-traffic" rel="noopener noreferrer"&gt;incoming traffic&lt;/a&gt; (or takes it over, in steal mode), makes its &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/outgoing-traffic" rel="noopener noreferrer"&gt;outgoing connections&lt;/a&gt; and DNS lookups through the target pod, so cluster-internal databases and services are reachable, and reads the same &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/environment-variables" rel="noopener noreferrer"&gt;environment variables&lt;/a&gt; and &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/file-operations" rel="noopener noreferrer"&gt;files&lt;/a&gt; as the pod. As far as your code is concerned, it's running in the cluster. Nothing gets built, pushed or deployed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Run your service in the context of the deployed checkout service&lt;/span&gt;
mirrord &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nt"&gt;--target&lt;/span&gt; deployment/checkout &lt;span class="nt"&gt;--&lt;/span&gt; npm run dev
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The diagram below, from the mirrord docs, shows a mirrord session: the local process gets the target's incoming traffic, and its outgoing traffic, database access and file reads go through the cluster.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faxq917oljsqp8u4sy6as.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faxq917oljsqp8u4sy6as.png" alt="mirrord architecture: a local process connected to a target pod in the cluster through mirrord" width="800" height="373"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  How it supports microservices testing
&lt;/h3&gt;

&lt;p&gt;Because your service talks to its real dependencies, integration and end-to-end tests run against actual behavior instead of mocks. Developers do this from their IDE (VS Code, Cursor, JetBrains, etc.) or a terminal. In CI, &lt;a href="https://metalbear.com/mirrord/ci/" rel="noopener noreferrer"&gt;mirrord for CI&lt;/a&gt; starts the service under test inside the runner and connects it to your staging cluster, so you can run your existing test suite against real dependencies without deploying a fresh environment for every run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mirrord ci start &lt;span class="nt"&gt;--target&lt;/span&gt; deployment/checkout &lt;span class="nt"&gt;--&lt;/span&gt; npm start
npm run &lt;span class="nb"&gt;test&lt;/span&gt;:e2e
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;mirrord ci&lt;/code&gt; works with the open-source version. If the &lt;a href="https://metalbear.com/mirrord/docs/getting-started/installing-mirrord/operator" rel="noopener noreferrer"&gt;mirrord Operator&lt;/a&gt; is installed in your cluster (which the team features below need), CI runs authenticate with a CI API key, which is part of the &lt;a href="https://metalbear.com/mirrord/pricing/" rel="noopener noreferrer"&gt;Enterprise plan&lt;/a&gt; and keeps CI sessions from taking up developer seats.&lt;/p&gt;

&lt;p&gt;This is also what makes mirrord useful for AI coding agents. An agent can run the code it just wrote against real services, see what breaks, and iterate, in seconds rather than waiting on a full CI deploy cycle. MetalBear maintains an open-source &lt;a href="https://metalbear.com/mirrord/docs/use-cases/using-mirrord-with-ai/ai-skills-plugin" rel="noopener noreferrer"&gt;skills plugin&lt;/a&gt; that teaches agents like Claude Code and Cursor how to configure and run mirrord.&lt;/p&gt;

&lt;h3&gt;
  
  
  Isolation for teams sharing one cluster
&lt;/h3&gt;

&lt;p&gt;The obvious question with a shared cluster is: what stops people from stepping on each other? mirrord handles traffic isolation, message isolation and data isolation per session, rather than per environment:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;HTTP traffic filtering&lt;/strong&gt; routes only matching requests (for example, ones carrying your session header, or a specific path) to your local process, while all other traffic keeps going to the service in the cluster. With traffic mirroring, your process gets a copy of the requests instead, and the service in the cluster keeps handling the originals.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Queue splitting&lt;/strong&gt; sends the messages that match your filter to your local consumer and leaves the rest for the shared consumer, so the two don't compete for the same messages. You can also mirror matching messages instead, so both your local consumer and the deployed one process a copy. It supports Kafka, SQS, RabbitMQ, Google Cloud Pub/Sub, Azure Service Bus, NATS, Redis Pub/Sub, BullMQ and Temporal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Database branching&lt;/strong&gt; gives each session an isolated branch of the database, so writes and migrations don't affect anyone else's data. It supports MySQL, MariaDB, PostgreSQL, MSSQL, MongoDB, Redis, DynamoDB, ClickHouse, CockroachDB, Google Spanner, S3 and turbopuffer, plus a generic option for any other service you run from your own image.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Beyond isolation, the Operator adds features for running mirrord across a team. &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/policies" rel="noopener noreferrer"&gt;Policies&lt;/a&gt; let platform teams limit which mirrord features can be used on which targets, for example blocking traffic stealing without an HTTP filter. &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/profiles" rel="noopener noreferrer"&gt;Profiles&lt;/a&gt; give everyone shared, reusable mirrord configurations. &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/sessions" rel="noopener noreferrer"&gt;Session management&lt;/a&gt; shows which sessions are active on which targets and lets you stop them. The &lt;a href="https://metalbear.com/mirrord/docs/managing-mirrord/admin-dashboard" rel="noopener noreferrer"&gt;admin dashboard&lt;/a&gt; shows who uses mirrord, on which targets and in which CI pipelines, and how much time it saves.&lt;/p&gt;

&lt;h3&gt;
  
  
  Preview environments
&lt;/h3&gt;

&lt;p&gt;Local and CI sessions end when the process stops. A preview environment keeps a change running in the cluster so someone else can try it: a product manager, a QA engineer, a reviewer, or an AI agent checking its own work. A preview runs only the changed service, as a pod built from the existing workload's spec with your new image. Requests carrying the preview's key go to it, everything else keeps going to the existing workload, and the preview talks to the real databases, queues and upstream services in the cluster. You can share a preview as a link, and it stays up for a time-to-live you set, independent of anyone's machine.&lt;/p&gt;

&lt;p&gt;That's the main difference from other preview environment tools that build a full copy of the environment for every pull request: nothing else gets duplicated. With &lt;a href="https://metalbear.com/mirrord/docs/use-cases/preview-environments#idle-behavior" rel="noopener noreferrer"&gt;idle mode&lt;/a&gt;, a preview also scales down to zero pods when it gets no traffic and starts again when a matching request or queue message arrives. A preview for every open pull request costs almost nothing while nobody is using it.&lt;/p&gt;

&lt;p&gt;Previews also change how you review work you hand to an &lt;a href="https://metalbear.com/mirrord/docs/use-cases/using-mirrord-with-ai" rel="noopener noreferrer"&gt;AI coding agent&lt;/a&gt;. With mirrord set up in its environment, the agent picks up a ticket, makes the change, builds and pushes an image, and runs &lt;code&gt;mirrord preview start&lt;/code&gt; with the ticket ID as the key. Instead of judging the change from the diff alone, you open the preview, try it yourself against the cluster's real dependencies, and then decide whether it's ready to merge.&lt;/p&gt;

&lt;h3&gt;
  
  
  Chaos testing
&lt;/h3&gt;

&lt;p&gt;Testing against real dependencies also means testing what happens when they misbehave. With &lt;a href="https://metalbear.com/mirrord/docs/use-cases/chaos-testing" rel="noopener noreferrer"&gt;chaos testing&lt;/a&gt;, you add rules to your session that can slow down or break your service's outgoing connections to specific hosts: added latency (with jitter), or connection resets, timeouts and refusals, applied to every matching connection or a percentage of them. The rules only affect your session, so you can see how your code handles a slow payment API or a dropped database connection without disrupting anyone else on the cluster.&lt;/p&gt;

&lt;p&gt;We didn't find built-in latency or connection-failure injection in the docs of the other environment tools in this list. The closest is &lt;a href="https://www.signadot.com/docs/concepts/middleware" rel="noopener noreferrer"&gt;Signadot's Override middleware&lt;/a&gt;, which can return canned or error responses for requests to the sandboxed workload.&lt;/p&gt;

&lt;p&gt;Here's what each plan includes, per the &lt;a href="https://metalbear.com/mirrord/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; and docs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Open source (free):&lt;/strong&gt; the CLI and IDE extensions, mirroring and stealing traffic, HTTP filtering, chaos testing, and &lt;code&gt;mirrord ci&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Team:&lt;/strong&gt; adds the mirrord Operator in the cluster, which enables concurrent sessions against the same target, queue splitting, database branching, policies, RBAC and usage monitoring.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Enterprise:&lt;/strong&gt; everything in Team, plus preview environments, CI API keys, high availability and support for air-gapped clusters.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Because isolation happens per session rather than per environment, adding another developer, CI runner or agent doesn't add another copy of your infrastructure, so cloud cost stays flat as the number of concurrent testers grows.&lt;/p&gt;

&lt;h3&gt;
  
  
  Real-world results
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://metalbear.com/mirrord/case-study/monday/" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; moved from per-developer cloud environments to one shared environment. Environment setup dropped from 30 minutes to 10 seconds, onboarding from 3–5 days to about an hour, and retiring the per-developer environments saves around $450 per developer per month. 350+ engineers now work on one cluster.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://metalbear.com/mirrord/case-study/colab/" rel="noopener noreferrer"&gt;CoLab&lt;/a&gt; cut the time to get a change running in the cloud from over 15 minutes to seconds, a reduction in dev cycle time of up to 98%.&lt;/li&gt;
&lt;li&gt;At &lt;a href="https://metalbear.com/mirrord/case-study/daylight-agents/" rel="noopener noreferrer"&gt;Daylight Security&lt;/a&gt;, Claude Code verifies its own changes in a mirrord-powered preview environment against real pre-prod services. Each verification takes about 5 seconds, compared to the 5+ minutes a CI loop would take.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Trade-offs
&lt;/h3&gt;

&lt;p&gt;mirrord isn't a load testing tool, and it doesn't replace contract tests or unit tests. It assumes you already have a cluster (usually staging) that's in reasonably good shape, because that's what your code runs against. Filtering traffic for several services in a call chain also requires that your services propagate the routing header between them. The team features described above need the operator installed in the cluster, which is a conversation to have with your platform team.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pricing and licensing
&lt;/h3&gt;

&lt;p&gt;Per the &lt;a href="https://metalbear.com/mirrord/pricing/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt;, the open-source version is free. The Team plan costs $50 per seat per month billed monthly, or 20% less billed annually, with no credit card required to start. Seats are counted by monthly activity. Enterprise pricing is custom and covers preview environments, CI API keys, high availability, support for air-gapped clusters and dedicated support.&lt;/p&gt;

&lt;p&gt;See also: the &lt;a href="https://metalbear.com/mirrord/compare/signadot/" rel="noopener noreferrer"&gt;mirrord vs. Signadot&lt;/a&gt; and &lt;a href="https://metalbear.com/mirrord/compare/telepresence/" rel="noopener noreferrer"&gt;mirrord vs. Telepresence&lt;/a&gt; comparisons, and the &lt;a href="https://metalbear.com/mirrord/docs/getting-started/quick-start" rel="noopener noreferrer"&gt;quickstart&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Signadot: best for isolated sandboxes per pull request
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Test environments&lt;/p&gt;

&lt;p&gt;Signadot creates lightweight sandboxes inside an existing Kubernetes cluster. It's best for teams that want an isolated environment for every pull request without duplicating the whole stack.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does
&lt;/h3&gt;

&lt;p&gt;A sandbox deploys only the services you changed. Everything else is served by the shared baseline environment, meaning the unchanged services already running in the cluster. According to &lt;a href="https://www.signadot.com/docs/overview" rel="noopener noreferrer"&gt;Signadot's documentation&lt;/a&gt;, requests are routed to the sandbox or the baseline through header propagation (it names OpenTelemetry and B3), using its built-in routing or a service mesh like Istio or Linkerd. Signadot installs an operator in your cluster that keeps a secure tunnel open to the &lt;a href="https://www.signadot.com/docs/concepts/architecture/control-plane" rel="noopener noreferrer"&gt;Signadot Control Plane&lt;/a&gt;, a service Signadot runs outside your cluster. The control plane serves the API for creating and managing sandboxes, and requests to the preview URLs it hosts are proxied through it into your cluster.&lt;/p&gt;

&lt;h3&gt;
  
  
  How it supports testing
&lt;/h3&gt;

&lt;p&gt;You can run your existing test suites against a sandbox, create sandboxes per pull request from CI, and use Signadot's Smart Tests to compare a sandbox's behavior against the baseline. Signadot also supports local development against a sandbox.&lt;/p&gt;

&lt;h3&gt;
  
  
  When it fits, and when it doesn't
&lt;/h3&gt;

&lt;p&gt;Signadot fits teams with many services who want PR-level isolation and are comfortable with their cluster keeping a tunnel open to a control plane that Signadot runs, rather than running every component inside their own infrastructure. Like any tool that routes by request headers (mirrord included), it depends on every service in a call chain forwarding the routing header. Its plans also count sandboxes per month, so teams that open many short-lived PRs should check the limits against their volume.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pricing
&lt;/h3&gt;

&lt;p&gt;Per &lt;a href="https://www.signadot.com/pricing/" rel="noopener noreferrer"&gt;Signadot's pricing page&lt;/a&gt;: Starter is free (50 sandboxes and 100 test invocations per month, one cluster). Business costs $250 per month (100 sandboxes per month, three clusters, extras billed in blocks). Enterprise is custom.&lt;/p&gt;

&lt;p&gt;See also: &lt;a href="https://metalbear.com/mirrord/compare/signadot/" rel="noopener noreferrer"&gt;mirrord vs. Signadot&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Telepresence: best for connecting a local service to a remote cluster
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Test environments&lt;/p&gt;

&lt;p&gt;Telepresence is an open-source tool that lets you run a service on your machine while the cluster's network and services stay reachable. It's best for individual developers who want to debug a service locally against a remote cluster, or for teams that want a completely open-source, vendor-free tool and don't need commercial support.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does
&lt;/h3&gt;

&lt;p&gt;Telepresence connects your workstation to the cluster network and can intercept traffic meant for a workload and route it to your machine, or send a copy of it ("wiretap") while the original keeps flowing. Intercepts can be filtered by header or path.&lt;/p&gt;

&lt;h3&gt;
  
  
  What gets installed
&lt;/h3&gt;

&lt;p&gt;Per the &lt;a href="https://telepresence.io/docs/concepts/architecture" rel="noopener noreferrer"&gt;Telepresence architecture docs&lt;/a&gt;, a cluster administrator installs the &lt;strong&gt;Traffic Manager&lt;/strong&gt;, which coordinates connections and tracks active sessions. By default, a &lt;strong&gt;Traffic Agent&lt;/strong&gt; is added to each workload you intercept, usually as a sidecar container, or optionally as a node-hosted agent. The pod restarts when the sidecar is first injected, and the sidecar stays in the pods after you disconnect until it's uninstalled. On the workstation, Telepresence runs a user daemon and a root daemon, which sets up a virtual network device and DNS for the whole machine. A Docker mode runs this inside a container instead.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where it fits
&lt;/h3&gt;

&lt;p&gt;Telepresence &lt;a href="https://telepresence.io/about" rel="noopener noreferrer"&gt;describes itself&lt;/a&gt; as "an open source tool for Kubernetes application developers," and its documentation focuses on local development and debugging against a remote cluster. The documentation doesn't describe a way to isolate databases between developers. For queues, it provides an &lt;a href="https://telepresence.io/docs/reference/restapi" rel="noopener noreferrer"&gt;API&lt;/a&gt; that consumers of an intercepted workload can call with a message's headers: the consumer in the cluster leaves messages that match an active intercept on the queue, and your local consumer takes them. That check has to be added to the consumer's code. It also documents a &lt;code&gt;replace&lt;/code&gt; mode that takes the application container out of the cluster pod while you're attached, so only your local copy consumes messages and none get processed twice.&lt;/p&gt;

&lt;h3&gt;
  
  
  Licensing
&lt;/h3&gt;

&lt;p&gt;Telepresence is open source under the Apache 2.0 license and a &lt;a href="https://telepresence.io/about" rel="noopener noreferrer"&gt;CNCF Sandbox project&lt;/a&gt;, originally built by Ambassador Labs. It's community-supported.&lt;/p&gt;

&lt;p&gt;See also: &lt;a href="https://metalbear.com/mirrord/compare/telepresence/" rel="noopener noreferrer"&gt;mirrord vs. Telepresence&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Okteto: best for ephemeral environments per developer or pull request
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Test environments&lt;/p&gt;

&lt;p&gt;Okteto is a platform for development, preview and testing environments that run inside Kubernetes. It's best for teams that want each developer or pull request to get its own complete copy of the application.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does
&lt;/h3&gt;

&lt;p&gt;Okteto gives developers remote ephemeral development environments with local-to-cloud code sync and remote debugging. For each pull request, it creates a preview environment defined in an Okteto manifest, triggered from CI tools like GitHub Actions or GitLab CI. It also offers testing environments for running test suites in the cluster.&lt;/p&gt;

&lt;h3&gt;
  
  
  How it supports microservices testing
&lt;/h3&gt;

&lt;p&gt;By default, each environment is a full copy of the stack in its own namespace, with every service plus its own databases and queues. That makes isolation simple, since nothing is shared and nothing collides, but cloud cost grows with every open PR and active developer, and creating an environment takes as long as deploying the whole stack. Data in those environments typically comes from seed data rather than the real shared databases.&lt;/p&gt;

&lt;p&gt;Okteto's &lt;a href="https://www.okteto.com/docs/core/divert/" rel="noopener noreferrer"&gt;Divert&lt;/a&gt; feature narrows this: you deploy only the services you change, and requests for the rest are routed to a shared environment using the &lt;code&gt;baggage&lt;/code&gt; header, as long as your services propagate it. That shared environment has to be deployed by Okteto, though, so unless your staging already runs on Okteto, you first have to redeploy or duplicate your services there.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pricing
&lt;/h3&gt;

&lt;p&gt;Per &lt;a href="https://www.okteto.com/pricing/" rel="noopener noreferrer"&gt;Okteto's pricing page&lt;/a&gt;, the Starter plan is free for 5 seats. Scale and Enterprise pricing isn't published. Okteto is self-hosted on your own cluster, so you also pay the cloud cost of the environments it creates. The Okteto CLI is open source.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Tilt: best for local Kubernetes inner-loop development
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Test environments&lt;/p&gt;

&lt;p&gt;Tilt is an open-source tool, maintained by Docker, that automates the inner loop of Kubernetes development. It's best for developers who want fast rebuilds against a local cluster.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does
&lt;/h3&gt;

&lt;p&gt;You describe your services in a Tiltfile, run &lt;code&gt;tilt up&lt;/code&gt;, and Tilt watches your files and, by default, rebuilds the container image and redeploys it on every change. If you configure Live Update, Tilt syncs changed files straight into the running containers instead, which skips the image rebuild. The Tilt UI shows logs and status for every resource in one dashboard.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where it fits
&lt;/h3&gt;

&lt;p&gt;Tilt is great when the whole system you need fits on your machine, and that's the setup its docs steer you toward. Its guide to &lt;a href="https://docs.tilt.dev/choosing_clusters.html" rel="noopener noreferrer"&gt;choosing a dev cluster&lt;/a&gt; covers local clusters (kind, Docker Desktop, MicroK8s, Minikube, k3d and Rancher Desktop) first, and treats remote clusters as the advanced option. Tests you run against a Tilt environment hit real containers of your services, but only the ones you've defined, often with local databases and seed data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where it stops being enough
&lt;/h3&gt;

&lt;p&gt;The ceiling is your machine. The same guide points out that some local cluster options, like Docker Desktop and Minikube, are heavy on local resources. Tilt can also work against a remote dev cluster, where developers can share common services like a dev database, but the same guide says "we only recommend remote clusters for large engineering teams where a dedicated dev infrastructure team can maintain your dev cluster," or for debugging "something that only reproduces in a complete cluster." So moving past the local machine means taking on a dev cluster that someone has to run and maintain.&lt;/p&gt;

&lt;h3&gt;
  
  
  Licensing and pricing
&lt;/h3&gt;

&lt;p&gt;Tilt is free and open source under the Apache 2.0 license.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Testkube: best for running any test framework inside the cluster
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Test orchestration&lt;/p&gt;

&lt;p&gt;Testkube runs your tests as Kubernetes-native workloads inside the cluster. It's best for platform teams that want one place to run, schedule and see the results of every kind of test.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does
&lt;/h3&gt;

&lt;p&gt;In most pipelines, the CI system (GitHub Actions or GitLab CI, for example) runs integration, API, end-to-end and load tests on its own runner machines, which reach the cluster over the network. Testkube runs those same tests as workflows inside the cluster instead, with scheduling, Kubernetes event triggers, CI/CD integration, and logs and artifacts collected centrally.&lt;/p&gt;

&lt;h3&gt;
  
  
  Supported frameworks
&lt;/h3&gt;

&lt;p&gt;Per &lt;a href="https://docs.testkube.io/articles/examples/overview" rel="noopener noreferrer"&gt;Testkube's documentation&lt;/a&gt;, examples cover k6, JMeter, Gatling, Locust and Artillery for load testing; Cypress, Playwright, Selenium and Robot Framework for end-to-end tests; Postman, SoapUI, REST Assured and cURL for API testing; and Pytest, Maven, Gradle, NUnit and Testcontainers for unit and integration tests.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it doesn't do
&lt;/h3&gt;

&lt;p&gt;Testkube orchestrates tests, but it doesn't provide the environment they run against. It complements environment tools: you still need to decide whether your tests hit staging, a per-PR copy or a service connected through something like mirrord.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pricing
&lt;/h3&gt;

&lt;p&gt;Per &lt;a href="https://testkube.io/pricing" rel="noopener noreferrer"&gt;Testkube's pricing page&lt;/a&gt;, the open-source agent is free with community support. Pro costs $700 per month per runner agent and adds a hosted control plane, parallel execution, analytics and flakiness detection. Enterprise is custom. AI add-ons are priced separately.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Postman: best for API testing of microservices
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; API testing&lt;/p&gt;

&lt;p&gt;Postman is an API platform for building, testing and documenting APIs. It's best for teams that want shared, versioned API tests that both developers and QA can work with.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does for microservices
&lt;/h3&gt;

&lt;p&gt;You build requests and group them into collections, define environments with variables (for example, local, staging and production base URLs), and write test scripts that assert on responses. The Collection Runner runs every request in a collection in the order you choose and logs the test results for each one, and the Postman CLI runs collections the same way in CI. Postman can also host mock servers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where it fits
&lt;/h3&gt;

&lt;p&gt;Postman sits at the API and integration layer, and it's strongest as a collaboration tool: a shared collection becomes the living reference for how a service's API is supposed to behave. When teams shortlist the best API testing tools for microservices, Postman usually sits next to code-first options like REST Assured (for Java) and Karate, which keep tests in the repository next to the service.&lt;/p&gt;

&lt;h3&gt;
  
  
  Limitations
&lt;/h3&gt;

&lt;p&gt;Postman tests whatever environment you point it at, so how much a passing collection tells you depends on that environment. Large collections also become code you have to maintain, with the scripts living inside Postman rather than next to the service in your repository by default. Postman can send HTTP, GraphQL, gRPC, WebSocket, Socket.IO and MQTT requests, but &lt;a href="https://learning.postman.com/docs/tests-and-scripts/running-collections/intro-to-collection-runs" rel="noopener noreferrer"&gt;its docs&lt;/a&gt; describe collection runs for HTTP, GraphQL and gRPC tests only, and GraphQL and gRPC collection runs need a paid plan.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pricing
&lt;/h3&gt;

&lt;p&gt;Per &lt;a href="https://www.postman.com/pricing/" rel="noopener noreferrer"&gt;Postman's pricing page&lt;/a&gt;, checked October 14, 2026: Free (1 user), Solo at $9 per month and Team at $19 per user per month, both billed annually, plus a custom Enterprise plan. The Collection Runner and mock servers are included on every plan, with GraphQL and gRPC collection runs limited to paid plans.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. WireMock: best for mocking unavailable dependencies
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Service mocking&lt;/p&gt;

&lt;p&gt;WireMock is an open-source tool for simulating HTTP APIs, a practice also called service virtualization. It's best for teams that need to test against dependencies that are unavailable, unstable, rate-limited or expensive to call.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does
&lt;/h3&gt;

&lt;p&gt;WireMock stubs HTTP endpoints with the responses you define. It supports stateful behavior, fault injection (malformed responses, dropped connections), artificial delays, and record-and-replay, where it proxies a real API and saves the responses as stubs. It runs embedded in a test, as a standalone process or as a Docker container.&lt;/p&gt;

&lt;h3&gt;
  
  
  When to mock, and when not to
&lt;/h3&gt;

&lt;p&gt;Mocking is the right choice for third-party APIs you can't call freely (payment providers, for example), for dependencies that don't exist yet, and for failure scenarios that are hard to trigger on demand. For your own internal services, testing against the real thing usually catches more, because a stub only knows about the behavior you told it about.&lt;/p&gt;

&lt;h3&gt;
  
  
  The maintenance cost
&lt;/h3&gt;

&lt;p&gt;Every stub is a copy of another service's behavior at one point in time. When that service changes, the stub doesn't, and tests keep passing against behavior that no longer exists. In our experience, this drift is the most common reason teams with large mock suites still get surprised when they ship to production. Pairing mocks with contract tests (see Pact below) helps keep them honest.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pricing and licensing
&lt;/h3&gt;

&lt;p&gt;WireMock is open source under the Apache 2.0 license. Per the &lt;a href="https://www.wiremock.io/get-pricing" rel="noopener noreferrer"&gt;WireMock Cloud pricing page&lt;/a&gt;, the hosted Free plan includes 1,000 mock API calls per month for a single user. Enterprise pricing is quote-only and adds stateful mocking, team management and Kubernetes self-hosting.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Pact: best for contract testing between services
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Contract testing&lt;/p&gt;

&lt;p&gt;Pact is an open-source framework for consumer-driven contract testing. It's best for organizations with many services that are deployed independently by different teams. If you're comparing the best tools for contract testing microservices, the usual alternatives are Spring Cloud Contract for JVM teams, and spec-driven tools like Specmatic and Microcks, which turn OpenAPI and AsyncAPI specs into contract tests and mocks.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does
&lt;/h3&gt;

&lt;p&gt;The consumer of an API writes tests describing the requests it sends and the responses it expects. Those tests run against a Pact mock and produce a contract. The provider then verifies that contract against its real implementation in its own build. If the provider changes in a way that breaks a consumer, the provider's build fails, before anything is deployed.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pact Broker, PactFlow and can-i-deploy
&lt;/h3&gt;

&lt;p&gt;Contracts are shared through the Pact Broker (self-hosted) or PactFlow (the managed version). The &lt;code&gt;can-i-deploy&lt;/code&gt; command checks the broker to tell you whether a specific version of a service is compatible with everything already in the target environment. Per the &lt;a href="https://docs.pact.io/" rel="noopener noreferrer"&gt;Pact docs&lt;/a&gt;, official implementations exist for JavaScript, the JVM, .NET, Go, Python, Swift, Scala, PHP, Ruby, Rust and C++.&lt;/p&gt;

&lt;h3&gt;
  
  
  When it pays off, and the limitations
&lt;/h3&gt;

&lt;p&gt;Pact pays off when many teams release independently, so a provider team can't check every change by hand with each of the teams that consume its API. Those consumer teams need to be known and willing to write Pact tests too, which is why Pact's docs list public APIs as a poor fit. The cost is coordination: consumer and provider teams both need to adopt it, and there's a learning curve around provider states and the broker workflow. Contract tests also only check the shape of interactions, not whether the full flow works end to end.&lt;/p&gt;

&lt;h3&gt;
  
  
  Licensing and pricing
&lt;/h3&gt;

&lt;p&gt;Pact is open source, under the MIT or Apache 2.0 license depending on the language implementation (Pact JVM, for example, uses Apache 2.0), and the Pact Broker can be self-hosted for free. Per &lt;a href="https://pactflow.io/pricing/" rel="noopener noreferrer"&gt;PactFlow's pricing page&lt;/a&gt;, the Starter plan is free (2 integrations), Team starts at $115.42 per month billed yearly, and Enterprise is custom.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. k6: best load testing tool for microservices
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Category:&lt;/strong&gt; Performance testing&lt;/p&gt;

&lt;p&gt;k6 is an open-source load testing tool from Grafana Labs. It's best for teams that want performance tests written as code and enforced in CI. The best load testing tools for microservices usually come down to a shortlist of k6, Gatling and Apache JMeter. k6 stands out for its JavaScript scripting and its Kubernetes operator.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it does
&lt;/h3&gt;

&lt;p&gt;You write load tests in JavaScript, define thresholds (for example, p95 latency under 300 ms or an error rate under 1%) that act as pass/fail criteria, and stream results to tools like Prometheus or Grafana. Because thresholds fail the run, k6 fits naturally as a gate in a CI pipeline.&lt;/p&gt;

&lt;h3&gt;
  
  
  Running k6 in Kubernetes
&lt;/h3&gt;

&lt;p&gt;The &lt;a href="https://grafana.com/docs/k6/latest/testing-guides/running-distributed-tests/" rel="noopener noreferrer"&gt;k6 Operator&lt;/a&gt; adds a &lt;code&gt;TestRun&lt;/code&gt; custom resource. You set a &lt;code&gt;parallelism&lt;/code&gt; value, and the operator starts that many k6 pods and splits the workload between them. Testkube can also run k6 tests in-cluster.&lt;/p&gt;

&lt;h3&gt;
  
  
  Limitations
&lt;/h3&gt;

&lt;p&gt;k6's documentation notes that a single machine may not generate enough load for very large tests, which is what distributed runs are for. Results also only reflect the environment you test: load testing a scaled-down staging cluster tells you about staging, not production.&lt;/p&gt;

&lt;h3&gt;
  
  
  Licensing and pricing
&lt;/h3&gt;

&lt;p&gt;k6 is open source under the AGPL-3.0 license. Per &lt;a href="https://grafana.com/pricing/" rel="noopener noreferrer"&gt;Grafana's pricing page&lt;/a&gt;, Grafana Cloud k6 includes 500 virtual user hours (VUh) per month for free, with additional usage starting at $0.15 per VUh. The Pro plan has a $19 monthly platform fee.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other microservices testing tools worth knowing
&lt;/h2&gt;

&lt;p&gt;The 10 tools above cover each layer of the stack, but they aren't the only options. These are worth a look if one of the tools above doesn't fit your setup.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;Category&lt;/th&gt;
&lt;th&gt;What it is&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Release&lt;/td&gt;
&lt;td&gt;Test environments&lt;/td&gt;
&lt;td&gt;Environments-as-a-service platform that creates full-stack ephemeral environments per pull request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bunnyshell&lt;/td&gt;
&lt;td&gt;Test environments&lt;/td&gt;
&lt;td&gt;Environments-as-a-service platform for ephemeral preview and development environments&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;vCluster&lt;/td&gt;
&lt;td&gt;Test environments&lt;/td&gt;
&lt;td&gt;Open-source virtual Kubernetes clusters, each with its own API server, running inside a namespace of a host cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Microcks&lt;/td&gt;
&lt;td&gt;Service mocking, contract testing&lt;/td&gt;
&lt;td&gt;CNCF Incubating open-source tool for API mocking and contract testing across OpenAPI, AsyncAPI, gRPC and GraphQL&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Specmatic&lt;/td&gt;
&lt;td&gt;Contract testing&lt;/td&gt;
&lt;td&gt;Turns OpenAPI and AsyncAPI specs into executable contract tests and service virtualization&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gatling&lt;/td&gt;
&lt;td&gt;Performance testing&lt;/td&gt;
&lt;td&gt;Open-source load testing tool with tests written as code&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Apache JMeter&lt;/td&gt;
&lt;td&gt;Performance testing&lt;/td&gt;
&lt;td&gt;Long-standing open-source load testing tool with a GUI test plan editor and broad protocol support&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Playwright&lt;/td&gt;
&lt;td&gt;End-to-end testing&lt;/td&gt;
&lt;td&gt;Open-source browser automation framework from Microsoft for end-to-end tests across Chromium, Firefox and WebKit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Testcontainers&lt;/td&gt;
&lt;td&gt;Integration testing&lt;/td&gt;
&lt;td&gt;Open-source library that starts throwaway Docker containers (databases, Kafka and more) from your test code&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Which combination of tools fits your team?
&lt;/h2&gt;

&lt;p&gt;The right combination depends on your team size, the number of services, how many teams share your staging environment, and how independently your teams release. Here are four common profiles.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A small team with a few services and no shared staging pain yet.&lt;/strong&gt; Tilt for the inner loop, Postman for API tests and k6 for a basic load test in CI is plenty. If you're running everything locally without trouble, you don't need a shared-cluster tool yet. Revisit when laptops start struggling or the service count grows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A platform team supporting many developers on a shared staging cluster.&lt;/strong&gt; This is where environment isolation matters most. mirrord lets everyone test against the same cluster with traffic filtering, queue splitting and database branching, without per-developer environments. Signadot is the closest alternative, with sandboxes for both local development and pull requests. The difference is in how databases and queues get isolated. Signadot does it through resource plugins (it publishes examples for MariaDB, PostgreSQL and SQS) or routing-key checks in consumer code, while mirrord has database branching and queue splitting built in, configured in mirrord rather than in each consumer's code. Okteto fits if you'd rather give each developer and PR a full copy and can absorb the cloud cost. Add Testkube if you want one place to run and track all your test suites in-cluster.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A team using AI coding agents that generate changes faster than the pipeline can validate them.&lt;/strong&gt; The bottleneck here is verification, and every agent needs a real place to test its work. Per-PR environment copies get expensive quickly when each agent spins one up for every change. mirrord's per-session isolation lets many agents test against one cluster concurrently without adding infrastructure for each one, and the mirrord skills plugin helps agents set it up. Keep Pact in place so agents can't silently break an API that another team depends on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A larger organization with independently released services owned by different teams.&lt;/strong&gt; Pact with &lt;code&gt;can-i-deploy&lt;/code&gt; keeps teams from breaking each other's APIs. Pair it with an environment tool for integration testing (mirrord or Signadot on a shared cluster), WireMock for third-party dependencies, and k6 for performance budgets on critical paths.&lt;/p&gt;

&lt;p&gt;Whichever profile you fit, start with one layer, usually the environment, since it affects everything else. Run a pilot on one service, measure feedback time and flaky test rate before and after, and only then roll it out further.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The best tools for microservices testing work as a layered stack, and the fidelity of the environment decides how far you can trust the results of every other tool in it.&lt;/p&gt;

&lt;p&gt;To recap: mirrord, Signadot, Telepresence, Okteto and Tilt cover the environment layer, each with a different isolation model. Testkube orchestrates tests inside the cluster. Postman covers API testing, WireMock handles dependencies you can't call, Pact catches breaking changes between teams, and k6 tells you how services behave under load.&lt;/p&gt;

&lt;p&gt;For teams that want to test against real dependencies without deploying or duplicating environments, mirrord is a strong option. &lt;a href="https://monday.com" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; uses it to run 350+ engineers on a single cluster instead of hundreds of per-developer environments, and teams like Daylight use it to let AI agents verify their own changes in seconds.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://app.metalbear.com/account/sign-up" rel="noopener noreferrer"&gt;Try mirrord for free&lt;/a&gt; or &lt;a href="https://metalbear.com/contact/" rel="noopener noreferrer"&gt;book a demo&lt;/a&gt;, and check out the &lt;a href="https://metalbear.com/mirrord/docs/getting-started/quick-start" rel="noopener noreferrer"&gt;quickstart guide&lt;/a&gt; to run your first session in a few minutes.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>microservices</category>
      <category>kubernetes</category>
      <category>backend</category>
    </item>
    <item>
      <title>Best Preview Environment Tools for Kubernetes</title>
      <dc:creator>Jake Page</dc:creator>
      <pubDate>Wed, 21 Oct 2026 13:39:30 +0000</pubDate>
      <link>https://dev.to/metalbear/best-preview-environment-tools-for-kubernetes-4ilg</link>
      <guid>https://dev.to/metalbear/best-preview-environment-tools-for-kubernetes-4ilg</guid>
      <description>&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; The best tool depends on how it isolates each pull request, and what that costs in licenses and cloud spend. mirrord and Signadot run only the changed services on your existing cluster. Qovery, Okteto, and Northflank copy the full stack per PR, and vCluster gives each PR its own virtual cluster.&lt;/p&gt;

&lt;p&gt;Most Kubernetes teams share one staging cluster among every developer who needs to test a change, and that arrangement was already strained before AI agents started opening pull requests of their own. With more people and agents sharing one testing environment, a queue forms, and developers, QA, product owners, and the platform teams who own the cluster all wait for their turn behind the changes ahead of theirs.&lt;/p&gt;

&lt;p&gt;Preview environments remove that queue by giving every pull request its own place to run, but the way a tool isolates each pull request comes with its own trade-offs in cost, speed, and fidelity, meaning how closely a preview behaves like your real production environment. This article compares 6 tools across three isolation models (request-level routing, full-stack copies, and virtual clusters). It also includes a comparison table, the pitfalls teams hit after rollout, and how to choose the right one for your system.&lt;/p&gt;

&lt;h2&gt;
  
  
  Kubernetes preview environment tools compared at a glance
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;Category&lt;/th&gt;
&lt;th&gt;Isolation model&lt;/th&gt;
&lt;th&gt;Cloud cost per PR&lt;/th&gt;
&lt;th&gt;Best fit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;mirrord&lt;/td&gt;
&lt;td&gt;Request-level routing&lt;/td&gt;
&lt;td&gt;Shared cluster, request routing, queue splitting, and database branching&lt;/td&gt;
&lt;td&gt;Changed services × open PRs&lt;/td&gt;
&lt;td&gt;Many users and agents on real staging traffic and data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Signadot&lt;/td&gt;
&lt;td&gt;Request-level routing&lt;/td&gt;
&lt;td&gt;Shared cluster, request routing&lt;/td&gt;
&lt;td&gt;Changed services × open PRs&lt;/td&gt;
&lt;td&gt;Self-serve start on a free tier&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qovery&lt;/td&gt;
&lt;td&gt;Full-stack copy per PR&lt;/td&gt;
&lt;td&gt;Namespace in your own cloud account&lt;/td&gt;
&lt;td&gt;All services × open PRs&lt;/td&gt;
&lt;td&gt;Teams wanting a full PaaS on their cloud&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Okteto&lt;/td&gt;
&lt;td&gt;Full-stack copy per PR&lt;/td&gt;
&lt;td&gt;Namespace per PR, deployed by your CI&lt;/td&gt;
&lt;td&gt;All services × open PRs&lt;/td&gt;
&lt;td&gt;Teams that want a full copy of the stack per developer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Northflank&lt;/td&gt;
&lt;td&gt;Full-stack copy per PR&lt;/td&gt;
&lt;td&gt;Copy per PR, deployed on Git events&lt;/td&gt;
&lt;td&gt;All services × open PRs&lt;/td&gt;
&lt;td&gt;Teams wanting full-stack previews billed by usage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;vCluster&lt;/td&gt;
&lt;td&gt;Virtual cluster per PR&lt;/td&gt;
&lt;td&gt;Virtual cluster in a host namespace&lt;/td&gt;
&lt;td&gt;All services × open PRs plus a control plane&lt;/td&gt;
&lt;td&gt;Teams already using vCluster for multitenancy&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What makes a good preview environment?
&lt;/h2&gt;

&lt;p&gt;A preview environment is an on-demand environment created for a pull request or branch to test a new code change, reachable by a shareable URL, and removed automatically when the PR closes. It's a kind of ephemeral environment focused on letting people review a change before it merges, and it runs alongside staging, which still handles integration after merge.&lt;/p&gt;

&lt;p&gt;To be worth relying on, a preview environment needs six things.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A shareable URL for every pull request, usable by non-engineers and AI coding agents&lt;/li&gt;
&lt;li&gt;An automatic lifecycle, created when the PR opens, updated on every push, and removed when it closes, with a time-to-live as a backup&lt;/li&gt;
&lt;li&gt;Production-like dependencies and realistic data&lt;/li&gt;
&lt;li&gt;Isolation between preview environments, so one change doesn't break another&lt;/li&gt;
&lt;li&gt;Cost control, including limits on concurrent previews, idle shutdown, and cleanup of forgotten environments&lt;/li&gt;
&lt;li&gt;Integration with the existing CI pipeline and Git provider&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At a high level, every tool in this article wires into the same pipeline, where a pull request triggers CI, CI builds an image and stores it in a registry, the image deploys into some form of isolated environment, a URL gets shared on the PR, and the environment tears down on merge or close. The tools below follow the same pipeline and differ only in what "isolated environment" means.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F29j8yyatkrc3cndq37m5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F29j8yyatkrc3cndq37m5.png" alt="Preview Environment diagram" width="800" height="400"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How do preview environments isolate changes?
&lt;/h2&gt;

&lt;p&gt;There are three isolation models, and the choice decides cost, speed, and fidelity. Request-level routing rebuilds only the services that changed and shares everything else with the baseline environment, while full-stack copies and virtual clusters rebuild the whole environment for every pull request.&lt;/p&gt;

&lt;h3&gt;
  
  
  Request-level routing on a shared cluster
&lt;/h3&gt;

&lt;p&gt;Only the changed services get deployed per PR, as separate pods that connect to the dependencies already running in an existing environment like staging, with requests routed to them according to e.g. a header. The cost of each extra PR stays low as you scale, because nothing else gets duplicated.&lt;/p&gt;

&lt;p&gt;Databases and queues stay shared as well, so previews work with real staging data, while a full-stack copy runs its own copies with seed data. This model fits teams with many services where duplicating the stack per PR would be too slow or expensive.&lt;/p&gt;

&lt;p&gt;Tools: &lt;a href="https://metalbear.com/mirrord/" rel="noopener noreferrer"&gt;mirrord&lt;/a&gt;, &lt;a href="https://www.signadot.com/" rel="noopener noreferrer"&gt;Signadot&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Full-stack copy per pull request
&lt;/h3&gt;

&lt;p&gt;Each pull request gets a complete copy of the application, deployed into its own namespace. It's a simple yet strong isolation method, since no services or data are shared between pull requests. A key consideration here is that cost scales with the number of services multiplied by the number of open PRs. This model fits teams with a small-to-medium service count who want every PR fully separated and can absorb the cost of a full copy per PR.&lt;/p&gt;

&lt;p&gt;Tools: &lt;a href="https://www.qovery.com/" rel="noopener noreferrer"&gt;Qovery&lt;/a&gt;, &lt;a href="https://www.okteto.com/" rel="noopener noreferrer"&gt;Okteto&lt;/a&gt;, &lt;a href="https://northflank.com/" rel="noopener noreferrer"&gt;Northflank&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Virtual cluster per pull request
&lt;/h3&gt;

&lt;p&gt;Each PR gets its own virtual cluster, with its own API server and RBAC, running in a namespace of a shared host cluster. Isolation is stronger than a plain namespace, because each PR gets its own cluster-level resources and permissions. A key consideration here is that the application inside each virtual cluster is still a full copy, so cost scales with the number of services multiplied by the number of open PRs, plus a control plane for each PR. This model fits teams that need that stronger isolation, or the occasional PR that changes cluster-level configuration.&lt;/p&gt;

&lt;p&gt;Tool: &lt;a href="https://www.vcluster.com/" rel="noopener noreferrer"&gt;vCluster&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The three isolation models side by side show what is copied and what is shared.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcd5z79q8ya2l9ecg5fzi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcd5z79q8ya2l9ecg5fzi.png" alt="Three isolation models" width="800" height="476"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How we selected and ordered these tools
&lt;/h2&gt;

&lt;p&gt;These 6 tools were selected and ordered using the same criteria, so the list can be compared fairly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Selection criteria:&lt;/strong&gt; all 6 tools run on Kubernetes, and each was compared on isolation model and cost as PRs grow, fidelity of dependencies and data, setup effort and operational ownership, lifecycle and cleanup control, and pricing and licensing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Order:&lt;/strong&gt; grouped by isolation model, with request-level routing first, then full-stack copies, and finally the virtual cluster.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sources:&lt;/strong&gt; each tool's own official documentation and pricing page, checked October 15, 2026.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Transparency:&lt;/strong&gt; MetalBear builds mirrord. mirrord is described below using the same criteria as every other tool, and only from verifiable facts.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. mirrord
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Isolation:&lt;/strong&gt; Shared cluster, header/path-based routing to the changed service | &lt;strong&gt;Setup:&lt;/strong&gt; mirrord Operator via Helm, plus the &lt;a href="https://github.com/metalbear-co/mirrord-preview" rel="noopener noreferrer"&gt;mirrord GitHub Action&lt;/a&gt; | &lt;strong&gt;Pricing from:&lt;/strong&gt; &lt;a href="https://metalbear.com/mirrord/pricing/" rel="noopener noreferrer"&gt;Team $50/seat/mo; Enterprise custom&lt;/a&gt; | &lt;strong&gt;Best for:&lt;/strong&gt; Many users and agents on real staging traffic and data&lt;/p&gt;

&lt;p&gt;mirrord fits teams that want a preview for every pull request, from developers and AI agents alike, on the staging cluster they already run.&lt;/p&gt;

&lt;p&gt;Only the changed service runs, as an isolated pod alongside its existing counterpart, and requests reach it by a per-PR header or another filter, such as the path. The &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/browser-extension" rel="noopener noreferrer"&gt;mirrord browser extension&lt;/a&gt; can set that header for you. Pods for the same PR share one environment key, so a PR that changes several services runs as one environment. Separate PRs can also be combined into one preview by sharing a key, so a reviewer can test changes to different services together. The &lt;a href="https://github.com/metalbear-co/mirrord-preview" rel="noopener noreferrer"&gt;mirrord GitHub Action&lt;/a&gt; starts a preview when a PR opens and stops it when the PR closes. Idle preview environments automatically scale to zero, keeping costs low even in clusters with very high scale.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting" rel="noopener noreferrer"&gt;Queue splitting&lt;/a&gt; lets each preview consume only the messages meant for it from the same queue, and &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/db-branching" rel="noopener noreferrer"&gt;database branching&lt;/a&gt; gives each preview its own branch of the staging database. Both are built into the mirrord Operator, so neither needs code changes.&lt;/p&gt;

&lt;p&gt;Requests that reach the preview through other services need those services to forward the routing header, which many clusters already do, for example through OpenTelemetry. Where they don't, MetalBear's open-source &lt;a href="https://github.com/metalbear-co/skills/tree/main/skills/mirrord-header-propagation" rel="noopener noreferrer"&gt;header propagation skill&lt;/a&gt; helps AI coding agents add it.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Signadot
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Isolation:&lt;/strong&gt; Shared cluster, routed per PR by a routing key in a header | &lt;strong&gt;Setup:&lt;/strong&gt; Signadot operator in the cluster, and each service forwards the routing header | &lt;strong&gt;Pricing from:&lt;/strong&gt; &lt;a href="https://www.signadot.com/pricing" rel="noopener noreferrer"&gt;Free (50 sandboxes/mo), Business $250/mo&lt;/a&gt; | &lt;strong&gt;Best for:&lt;/strong&gt; Self-serve start on a free tier&lt;/p&gt;

&lt;p&gt;Signadot fits teams that want to start with request-level previews on a self-serve plan, since its free and Business tiers include pull request previews with monthly sandbox limits.&lt;/p&gt;

&lt;p&gt;Signadot calls each preview a sandbox, which runs only the changed services on top of a shared baseline cluster. Requests carry a routing key in a &lt;code&gt;baggage&lt;/code&gt; header, which Signadot's browser extension can set for you. A preview gets created per PR, and several PRs can be combined into one preview using route groups, so a reviewer can test two in-flight changes together. Signadot's Smart Tests, which users write in Starlark, can run API and end-to-end tests against a sandbox and compare it with the baseline.&lt;/p&gt;

&lt;p&gt;Setup requires the Signadot operator in the cluster. For asynchronous flows, Signadot carries the routing key in message headers, and each consumer needs &lt;a href="https://www.signadot.com/blog/testing-microservices-message-isolation-for-kafka-sqs-more/" rel="noopener noreferrer"&gt;selective consumption logic&lt;/a&gt; added to its code, often through a wrapper around the client library, so it processes only its sandbox's messages. Database isolation uses &lt;a href="https://www.signadot.com/docs/guides/set-up-data-isolation" rel="noopener noreferrer"&gt;resource plugins&lt;/a&gt;, which run bash or Python scripts that the user writes or adapts from Signadot's examples to create a temporary schema or database for each sandbox.&lt;/p&gt;

&lt;p&gt;See also the &lt;a href="https://metalbear.com/mirrord/compare/signadot/" rel="noopener noreferrer"&gt;mirrord vs. Signadot comparison&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Qovery
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Isolation:&lt;/strong&gt; Full namespace copy in your own cloud account | &lt;strong&gt;Setup:&lt;/strong&gt; Connect a cloud account or cluster, then define a blueprint environment | &lt;strong&gt;Pricing from:&lt;/strong&gt; &lt;a href="https://www.qovery.com/pricing" rel="noopener noreferrer"&gt;Business $2,999/mo (up to 25 users, previews included)&lt;/a&gt; | &lt;strong&gt;Best for:&lt;/strong&gt; Teams wanting a full PaaS on their cloud&lt;/p&gt;

&lt;p&gt;Qovery fits teams that want a managed control plane handling previews inside their own cloud account, rather than building that themselves.&lt;/p&gt;

&lt;p&gt;Qovery deploys into your AWS, GCP, Azure, or Scaleway account, or into a Kubernetes cluster you already run. A preview environment is created from a blueprint environment for each pull request, with a shareable URL that updates on every push. Teardown happens automatically when the PR closes. Each preview duplicates the full stack, namespace, services, and dependencies, from the blueprint.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Okteto
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Isolation:&lt;/strong&gt; Full namespace copy defined by an Okteto manifest | &lt;strong&gt;Setup:&lt;/strong&gt; Okteto manifest plus a CI trigger | &lt;strong&gt;Pricing from:&lt;/strong&gt; &lt;a href="https://www.okteto.com/pricing" rel="noopener noreferrer"&gt;Free (5 seats); Scale and Enterprise custom&lt;/a&gt; | &lt;strong&gt;Best for:&lt;/strong&gt; Teams that want a full copy of the stack per developer&lt;/p&gt;

&lt;p&gt;Okteto fits teams that want a full copy of the stack for every developer and every pull request, with previews and day-to-day development on the same platform.&lt;/p&gt;

&lt;p&gt;Each pull request gets an isolated environment with a full copy of the application, defined in an Okteto manifest, and a shareable preview URL. Previews start only from your own CI, where a step in GitHub Actions or GitLab CI triggers their creation, updates, and removal. Beyond previews, Okteto also offers development and testing environments on the same platform, so adopting it for previews alone makes little sense. Like Qovery, Okteto duplicates the whole application for each preview, so cost grows with each pull request.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Northflank
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Isolation:&lt;/strong&gt; Full copy per PR, defined by a preview blueprint | &lt;strong&gt;Setup:&lt;/strong&gt; Link a Git account, then create a preview blueprint | &lt;strong&gt;Pricing from:&lt;/strong&gt; &lt;a href="https://northflank.com/pricing" rel="noopener noreferrer"&gt;Free Sandbox tier; previews on pay-as-you-go, billed per second; Enterprise custom&lt;/a&gt; | &lt;strong&gt;Best for:&lt;/strong&gt; Teams wanting full-stack previews billed by usage&lt;/p&gt;

&lt;p&gt;Northflank fits teams that want a choice in how previews start, since a preview can be triggered straight from a pull request or push, from an existing CI pipeline through a webhook, or by an API call.&lt;/p&gt;

&lt;p&gt;Each preview comes from a &lt;a href="https://northflank.com/docs/v1/application/release/set-up-preview-blueprints" rel="noopener noreferrer"&gt;preview blueprint&lt;/a&gt;, a template that builds the application, provisions a fresh database, and deploys the services, jobs, and secrets the pull request needs. A preview gets created or updated when a pull request opens or a branch gets a new push, and Northflank deletes it when the PR closes or merges. Each preview gets its own URL and can have a time-to-live, after which Northflank deletes it. Each preview duplicates the whole application, so cost grows with each pull request.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. vCluster
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Isolation:&lt;/strong&gt; Virtual cluster in a host namespace | &lt;strong&gt;Setup:&lt;/strong&gt; CI creates one virtual cluster per PR on a host cluster | &lt;strong&gt;Pricing from:&lt;/strong&gt; &lt;a href="https://www.vcluster.com/pricing" rel="noopener noreferrer"&gt;Free (open-source core); platform and Enterprise tiers&lt;/a&gt; | &lt;strong&gt;Best for:&lt;/strong&gt; Teams already using vCluster for multitenancy&lt;/p&gt;

&lt;p&gt;vCluster fits teams that need stronger isolation between previews than a plain namespace gives, or the occasional PR that changes cluster-level configuration.&lt;/p&gt;

&lt;p&gt;Each PR gets its own virtual cluster, with its own API server, running inside a namespace of a host cluster, with resources synced down to that host. CI creates a virtual cluster per PR, deploys into it, exposes the application, and removes it when done.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common pitfalls of rolling out preview environments
&lt;/h2&gt;

&lt;p&gt;Most problems with preview environments show up after rollout. They usually come down to three things, which are data, cost, and shared state.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data and secrets
&lt;/h3&gt;

&lt;p&gt;Every preview needs realistic data without copying production data unmasked. The usual options are seed data, masked copies of production data, or synthetic data, each trading fidelity against setup effort. A useful starting point is to find which tables carry personal data and mask those first.&lt;/p&gt;

&lt;p&gt;Secrets and credentials for many short-lived environments need the same discipline as production, pulled from a real secrets manager at runtime, scoped to that preview, and never baked into a manifest.&lt;/p&gt;

&lt;h3&gt;
  
  
  Forgotten environments and runaway cost
&lt;/h3&gt;

&lt;p&gt;Environments that keep running when nobody uses them, or that stay up after their pull request closes, are a common way for preview environments to backfire and become costly. Ways to address this include a time-to-live on every environment, idle shutdown for ones with no active traffic, a hard limit on concurrent previews, and cost visibility per environment.&lt;/p&gt;

&lt;p&gt;Measure what a single preview actually costs before setting limits, so the thresholds are set on real numbers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Stateful and asynchronous dependencies
&lt;/h3&gt;

&lt;p&gt;Databases, queues, and event-driven flows are the hardest part of isolation because they carry state that two previews can collide over.&lt;/p&gt;

&lt;p&gt;For databases, the common patterns are a dedicated database per preview (expensive but simple) or database branching (fast and realistic). A mostly read-only database can share a snapshot, while one written to during tests needs its own branch.&lt;/p&gt;

&lt;p&gt;Queues need their own pattern, because two previews that consume from the same topic can process each other's messages. Most teams either duplicate the whole environment, including the queue, or stop the staging consumer so the preview reads its messages instead. A third option is to split messages per preview, so each preview receives only the messages meant for it, which mirrord does with queue splitting.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The choice of a preview environment tool comes down to the isolation model. Full-stack copies and virtual clusters copy every service for every open PR, so they suit smaller systems or teams that need the strongest isolation. Request-level routing runs only what each PR changes, so cost stays low as PRs grow. If choosing the latter, the next question is how well the tool supports the databases, queues, and managed services your system already shares in staging, with the least instrumentation.&lt;/p&gt;

&lt;p&gt;For teams with a shared staging cluster and a growing number of PRs and agents, mirrord is a strong starting point, with queue splitting and database branching built in. &lt;a href="https://metalbear.com/mirrord/case-study/monday/" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; already runs 350+ engineers on one shared cluster with mirrord.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://app.metalbear.com/account/sign-up" rel="noopener noreferrer"&gt;Try mirrord free&lt;/a&gt; or &lt;a href="https://metalbear.com/contact/" rel="noopener noreferrer"&gt;book a demo&lt;/a&gt;, and read the &lt;a href="https://metalbear.com/mirrord/docs/use-cases/preview-environments" rel="noopener noreferrer"&gt;preview environments documentation&lt;/a&gt; for setup details.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>ai</category>
      <category>agents</category>
      <category>software</category>
    </item>
    <item>
      <title>Testing AI-Generated Code in a Shared Kubernetes Environment</title>
      <dc:creator>Eyal Bukchin</dc:creator>
      <pubDate>Tue, 13 Oct 2026 06:40:48 +0000</pubDate>
      <link>https://dev.to/metalbear/testing-ai-generated-code-in-a-shared-kubernetes-environment-12o</link>
      <guid>https://dev.to/metalbear/testing-ai-generated-code-in-a-shared-kubernetes-environment-12o</guid>
      <description>&lt;p&gt;&lt;em&gt;Written by Arsh Sharma. Originally published on &lt;a href="https://metalbear.com/blog/testing-ai-code-shared-env/" rel="noopener noreferrer"&gt;metalbear.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Research on &lt;a href="https://www.faros.ai/blog/ai-software-engineering" rel="noopener noreferrer"&gt;AI productivity&lt;/a&gt; have been finding out that while AI coding tools measurably increase code output, they don't produce a proportional increase in code merged to production. The gap is the feedback loop, specifically in how long it takes to find out whether generated code actually works in a real environment. In our previous post, we looked at how &lt;a href="https://metalbear.com/blog/testing-bottleneck-ai/#a-better-approach-remocal-testing" rel="noopener noreferrer"&gt;remocal testing&lt;/a&gt; — running your service locally while connected to all of its dependencies in a real Kubernetes cluster — closes that loop for individual developers. But closing it for a &lt;em&gt;team&lt;/em&gt; is a different problem entirely: if multiple developers and/or AI agents are all running local services against the same staging cluster, what prevents them from interfering with each other?&lt;/p&gt;

&lt;h2&gt;
  
  
  The shared environment problem
&lt;/h2&gt;

&lt;p&gt;When an AI agent tests generated code against staging, it interacts with other services in the cluster. It sends requests, consumes messages from queues, and reads and writes to databases. In a shared cluster, those interactions can collide.&lt;/p&gt;

&lt;p&gt;A few specific scenarios:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Traffic collisions&lt;/strong&gt;: Two developers both have a locally running instance of the same service to test out code generated by their AI agents. A request comes into the cluster, which they both want redirected to their locally running process to test their code. But which local instance gets it?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Queue interference&lt;/strong&gt;: An AI agent generates a queue consumer and starts processing messages from the shared queue. Those messages were meant for other consumers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Database corruption&lt;/strong&gt;: AI-generated code runs a migration or executes writes against the shared staging database. Everyone's test data is now affected.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The knee-jerk response is to give each developer or AI agent their own isolated environment. But as discussed in the previous post, that &lt;a href="https://metalbear.com/blog/testing-bottleneck-ai/#how-the-industry-is-responding-and-where-it-falls-short" rel="noopener noreferrer"&gt;trades one problem for another&lt;/a&gt;: high infrastructure costs, and environment drift.&lt;/p&gt;

&lt;p&gt;mirrord takes a different approach. Rather than isolating at the environment level, it isolates at the interaction level, filtering which traffic reaches your local process, splitting queues so each developer gets their own copy of messages, and branching databases so changes don't affect shared data. This lets the shared staging cluster stay fully operational for everyone, while each developer's local testing stays isolated to their own interactions.&lt;/p&gt;

&lt;p&gt;Let’s now look at some mirrord features that make testing AI-generated code in a shared Kubernetes cluster safe.&lt;/p&gt;

&lt;h2&gt;
  
  
  HTTP traffic filtering
&lt;/h2&gt;

&lt;p&gt;By default, when mirrord steals incoming traffic from the cluster to your local process, it steals all of it. Every request that is meant to hit the target service in the cluster gets forwarded to your local instance. Without a way to filter traffic, stealing all incoming traffic to a local process means the service in the cluster stops receiving requests, breaking the environment for everyone else who depends on it.&lt;/p&gt;

&lt;p&gt;mirrord's &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/incoming-traffic/filter-incoming-traffic" rel="noopener noreferrer"&gt;HTTP traffic filtering&lt;/a&gt; lets you route only requests that match specific criteria (a header, a path, or other filters) to your local process. For example, you can configure mirrord to only forward requests containing a header like &lt;code&gt;baggage: mirrord-session=arsh&lt;/code&gt; to your locally running instance. Every other request continues to go to the live service in the cluster.&lt;/p&gt;

&lt;p&gt;The result is that multiple developers and their AI agents can simultaneously test different changes against the same cluster without worrying about any traffic getting routed to the wrong place.&lt;/p&gt;

&lt;p&gt;For the AI agent, this is particularly useful since it can now send requests repeatedly, observe responses, adjust the generated code, and iterate. All of this while interacting with real dependencies but without touching traffic that belongs to anyone else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Queue splitting
&lt;/h2&gt;

&lt;p&gt;Most modern applications include a message queue like Kafka, SQS, RabbitMQ, or something similar. When an AI agent generates code for a consumer service, testing it against the real cluster creates an immediate conflict because the agent's locally running consumer competes for messages with the existing consumer in the cluster.&lt;/p&gt;

&lt;p&gt;If the generated code is wrong, which it often is on the first iteration, it processes or drops messages other services were expecting to receive. Depending on the system, this can cause cascading failures across your staging environment.&lt;/p&gt;

&lt;p&gt;mirrord's &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting" rel="noopener noreferrer"&gt;queue splitting&lt;/a&gt; handles this by creating an isolated temporary queue for the local process. The original consumer in the cluster continues processing from the main queue as normal. The locally running service receives its own copy of those messages in a separate queue that only it sees.&lt;/p&gt;

&lt;p&gt;This means the AI agent can run, fail, adjust, and re-run without affecting the shared queue or any other consumers. It gets real messages from the cluster to test against, but those messages are exclusively its own.&lt;/p&gt;

&lt;h2&gt;
  
  
  Database branching
&lt;/h2&gt;

&lt;p&gt;Database interactions are where AI-generated code most commonly breaks in ways that are hard to catch from reading the code alone. An AI agent might generate a query that doesn't match the actual schema, a migration that conflicts with existing indexes, or write logic that violates data constraints. These only surface when the code runs against a real database.&lt;/p&gt;

&lt;p&gt;But testing against the shared staging database directly is risky. A bad migration, an accidental mass update, or a schema change can break the environment for everyone.&lt;/p&gt;

&lt;p&gt;mirrord's &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/db-branching" rel="noopener noreferrer"&gt;database branching&lt;/a&gt; creates an isolated branch of the database for the local process. The branch mirrors the schema and data of the real staging database, so tests run against realistic conditions. But writes, migrations, and schema changes only affect that branch, while the shared database is untouched. This lets AI agents test the full range of database interactions without any risk to the rest of the team.&lt;/p&gt;

&lt;h2&gt;
  
  
  Additional safeguards
&lt;/h2&gt;

&lt;p&gt;HTTP filtering, queue splitting, and database branching cover the most common collision points. But mirrord includes a few more capabilities that help make testing AI-generated code safer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/policies" rel="noopener noreferrer"&gt;&lt;strong&gt;mirrord Policies&lt;/strong&gt;&lt;/a&gt; let platform teams define guardrails at the cluster level. For example, you can enforce that locally running AI-generated code can never write to shared databases by default, or restrict the namespaces where services can be targeted for stealing. Policies apply across the team so you're not relying on each developer to configure isolation correctly every time.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://metalbear.com/mirrord/docs/use-cases/preview-environments" rel="noopener noreferrer"&gt;&lt;strong&gt;Preview environments&lt;/strong&gt;&lt;/a&gt; allow AI agents to deploy code generated for a service as an isolated pod inside the shared staging cluster. This pod runs alongside the stable version of the service and only traffic explicitly routed to it reaches the new pod and everything else continues hitting the live service as normal. This lets you see and share a working version of the changes with stakeholders like product managers, QA, or sales before anything is merged, without needing a separate environment or disrupting anyone else.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://metalbear.com/mirrord/docs/use-cases/mirrord-for-ci" rel="noopener noreferrer"&gt;&lt;strong&gt;mirrord for CI&lt;/strong&gt;&lt;/a&gt; extends the remocal testing model to CI pipelines. Instead of provisioning new environments or relying on mocks for CI tests, your pipeline can run tests directly against the same shared staging cluster. This further reduces your cloud bill and the number of environments you have to manage for testing code.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  "It'll never work"
&lt;/h2&gt;

&lt;p&gt;Sharing a staging cluster across a full team (and AI agents on top of that) makes people itch with anxiety. The most common objection we hear to the shared environment approach is: there’s no way an entire team can safely work on the same cluster at the same time. What if one AI agent’s generated code does something that breaks the cluster for everyone else? As a response to that personal dev environments for each AI agent seem like the safest bet.&lt;/p&gt;

&lt;p&gt;But in practice, these personal environments don’t hold up because they drift from production, data goes stale, and infrastructure costs compound. And when AI agents are generating and iterating on code faster than any individual developer, the operational overhead of maintaining an environment per agent becomes completely impractical.&lt;/p&gt;

&lt;p&gt;The only model that scales in this case is a shared staging cluster where isolation happens at the interaction level, not the infrastructure level. One cluster, equipped with traffic filtering, queue splitting, and database branching, can support an entire team of developers and AI agents simultaneously, without interference. Each person's testing stays contained to their own interactions. And the environment remains operational for everyone else to keep working.&lt;/p&gt;

&lt;p&gt;This is what mirrord is built for. Large organizations like &lt;a href="https://metalbear.com/mirrord/case-study/monday/" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt;, &lt;a href="https://metalbear.com/mirrord/case-study/zooplus/" rel="noopener noreferrer"&gt;zooplus&lt;/a&gt;, and &lt;a href="https://metalbear.com/mirrord/case-study/surveymonkey/" rel="noopener noreferrer"&gt;SurveyMonkey&lt;/a&gt; already have hundreds of developers using mirrord concurrently, sharing a Kubernetes environment in their organization every day. And as AI agents start running at higher velocity, generating complete features in minutes and iterating on tests in real time, having this infrastructure in place is what will allow you to truly see the benefits from using AI coding tools.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>ai</category>
      <category>testing</category>
      <category>microservices</category>
    </item>
    <item>
      <title>How Our Engineering Team Uses AI, Part II: Meat Proxies</title>
      <dc:creator>Eyal Bukchin</dc:creator>
      <pubDate>Thu, 08 Oct 2026 10:33:43 +0000</pubDate>
      <link>https://dev.to/metalbear/how-our-engineering-team-uses-ai-part-ii-meat-proxies-148g</link>
      <guid>https://dev.to/metalbear/how-our-engineering-team-uses-ai-part-ii-meat-proxies-148g</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://metalbear.com/blog/engineering-ai-use-part-2/" rel="noopener noreferrer"&gt;metalbear.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In January we published &lt;a href="https://metalbear.com/blog/engineering-ai-use/" rel="noopener noreferrer"&gt;How Our Engineering Team Uses AI&lt;/a&gt;. The tl;dr was that AI was useful around the edges of our work (getting oriented in unfamiliar code, exploring approaches, writing scripts) but not much use at the center of it, because mirrord is a low-level Rust tool with an unusual architecture, and general models couldn't really deal with it. We wrapped up the post by calling AI "a powerful tool around the edges of software development."&lt;/p&gt;

&lt;p&gt;Eight months later, we asked the engineering team the same questions again. AI has since moved from the edges to the center: most of the engineers who replied barely write code by hand anymore. Someone still has to read all that code, and so the part of the job that grew the most is review, not only of pull requests, but also the inner loop of reading and correcting (or re-prompting) every chunk of code the agent writes.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I'm not really writing any code, I just review the slop and try to tame the beast.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  We heard you like tools
&lt;/h2&gt;

&lt;p&gt;In Part I, tools converged on ChatGPT, Gemini, Claude Code and Cursor. We still don’t mandate any specific tooling, and now there’s even less standardization among what the team uses. Claude Code and Codex are the most common agents, used through the CLI, the desktop apps, or an editor integration. Beyond that, it's a long tail. Some examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Claude inside Emacs, through &lt;a href="https://github.com/xenodium/agent-shell" rel="noopener noreferrer"&gt;agent-shell&lt;/a&gt;, an Emacs package for running coding agents inside the editor.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://ampcode.com/" rel="noopener noreferrer"&gt;Amp&lt;/a&gt;, a coding agent and development environment that can work in your local checkout from the terminal or on remote machines it spins up per task.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/stablyai/orca" rel="noopener noreferrer"&gt;Orca&lt;/a&gt;, a desktop app for running several agents in parallel, each in its own git worktree.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/badlogic/pi-mono" rel="noopener noreferrer"&gt;pi&lt;/a&gt;, a minimal open-source agent harness that runs Codex, Claude and Copilot models from one place and is extended with plugins and your own markdown files.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/safishamsi/graphify" rel="noopener noreferrer"&gt;Graphify&lt;/a&gt;, which turns a codebase into a graph agents can query, used to make cheaper Codex models more accurate on our code.&lt;/li&gt;
&lt;li&gt;Gemma running locally through &lt;a href="https://github.com/ggml-org/llama.cpp" rel="noopener noreferrer"&gt;llama.cpp&lt;/a&gt;, for small chores like commits and rebases.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://zed.dev/" rel="noopener noreferrer"&gt;Zed&lt;/a&gt; keeps showing up as well, both for its agent integrations and as a fast, light editor for when a human needs to edit or clean up what the agent wrote.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cost has entered the equation too. By one engineer's estimate, Fable uses about ten times as much of a usage allowance as Opus, and for their work the quality gain was moderate at best, so they stay on Opus. Another engineer tried to save tokens by having a strong model write a detailed implementation guide and handing it to a cheaper model to write the code, since generating output costs more than reading input (alas, the cheaper models wrote code bad enough to make the savings pointless, so this experiment failed).&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it helps
&lt;/h2&gt;

&lt;p&gt;People now use agents for pretty much everything: planning, breaking tickets into subtasks, researching bugs, exploring unfamiliar parts of the codebase, writing code (both on the edges and in the core of the product), writing and running tests, pre-reviewing PRs before asking a human to look at it, diagnosing and fixing CI failures, and investigating production incidents. Some use cases in the general consensus:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Finding things by description rather than by name.&lt;/strong&gt; When you're working with e.g. an unfamiliar Kubernetes data model, you often don't know what the thing you're looking for is called, so you can't grep for it, but you can describe it. Agents are very good at this kind of search.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code outside the product.&lt;/strong&gt; Build tooling, bash scripts, and throwaway tests for checking locally how our infrastructure behaves in an edge case (like how a queue handles some unusual sequence of events). The stakes are low, since a mistake breaks a build step or a local experiment rather than something a customer runs. Our &lt;code&gt;xtask&lt;/code&gt; build scripts were described as something "no human hand has ever touched."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Status updates.&lt;/strong&gt; The age-old chore of keeping Linear issues and status posts current can finally be outsourced.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Questions about past decisions.&lt;/strong&gt; Why a component was designed the way it was, or what was decided about a feature, is often faster to ask an agent than to track down the person who made the call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Collecting and analyzing messy data.&lt;/strong&gt; That's how we found our flakiest tests and measured how our CI changes affected run durations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Work a script should be doing.&lt;/strong&gt; For repetitive tasks nobody has automated yet, having an agent do the work each time is slower and more expensive than a script would be, but we’re lazy.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Where it still struggles
&lt;/h2&gt;

&lt;p&gt;Rust quality remains a recurring complaint. Agent-written Rust compiles, but it tends to be much longer than it needs to be, with unnecessary &lt;code&gt;.to_owned()&lt;/code&gt;, &lt;code&gt;.clone()&lt;/code&gt; and &lt;code&gt;.collect()&lt;/code&gt; calls that someone then has to go through and remove. Agents are also eager to write tests, and love splitting code into tiny functions so they can add tiny tests for them. Even the engineers who have moved furthest toward AI still evaluate every line and the overall structure of a change themselves, and they often find it wanting.&lt;/p&gt;

&lt;p&gt;Agents also tend to reinvent things. Ask one how to solve a problem and it will often come up with its own solution instead of finding the library or well-known approach that already solves it, so its research is a starting point rather than an answer. So most engineers prefer to do the initial research for a new area by hand. The point is for the engineer to understand the problem space deeply enough to make good calls on everything built on top of it.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Writing the PR description/commits makes me look at the problem from a higher level, which ensures I at the very least understand the solution and can judge it appropriately.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The strongest opinions were about writing style. Several engineers write every commit message, PR description and code comment themselves, because agents produce too much of it and too little of it makes sense. Those who let agents draft docs or RFCs usually end up rewriting most of it, because, as one put it, "AI generated text reads like word pulp." But the best argument for doing it by hand is that writing the PR description forces you to look at the problem from a higher level, which can help ensure you understood the solution well enough.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;AI generated text reads like word pulp.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  We mostly just review now
&lt;/h2&gt;

&lt;p&gt;Asked how their usage had changed since January, one engineer described much more parallelism and much more fatigue:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;My week used to be 20-40% reviews, 60-80% code gen, now it's 95% reviews, it's so tiring.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Others described something close to burnout. A task now takes more rounds of reviewing and correcting before it's actually finished, so finishing feels further away than it used to, and with several agents running, you're in that state on several tasks at once. Some refuse the parallelism altogether, running a single agent thread with a narrow purpose and thinking carefully about what comes back.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;i only use 1 thread at a time with a very specific purpose and think about things carefully. all those agent orchestrators and stuff are too much for me, it feels like the fastest way to get burnout&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The most difficult part is keeping control over quality and maintaining an understanding of how the system works. Even the engineers who no longer write code by hand still go through several review passes after the code works, until they understand and have tested every line. Everyone works hard to not become meat proxies.&lt;/p&gt;

&lt;p&gt;We’re trying to find solutions, though, and right now we’re hoping we can successfully automate the first pass. Greptile reviews our pull requests automatically (we’re currently evaluating it against Qodo, using AI to run the evaluation), and many of us have an agent review our own PR before a person sees it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building for the agent that will maintain it
&lt;/h2&gt;

&lt;p&gt;There were also changes in how people design things. Some of us now build with the assumption that an agent will maintain the code, which in practice means thorough logging, so that an agent debugging it later has something to read (a courtesy I’m sure many humans would have also appreciated). Some write personal skills that encode their repeated rules and fixes.&lt;/p&gt;

&lt;p&gt;That has a company-level version. The January post showed the CLAUDE.md in our mirrord repo, which back then included a long architecture walkthrough. It has since become an &lt;a href="https://github.com/metalbear-co/mirrord/blob/main/AGENTS.md" rel="noopener noreferrer"&gt;AGENTS.md&lt;/a&gt;, with CLAUDE.md as a link to it, and it got shorter: the walkthrough was deemed unnecessary and cut, leaving a one-line description of each core component plus the commands for building, testing and linting. It now sits alongside a company-wide one. Everyone at MetalBear imports one shared context repository from their personal &lt;code&gt;~/.claude/CLAUDE.md&lt;/code&gt;, which describes our teams, the tools we use and the known quirks of each, so every agent starts from the same background. The repository is private, but the pattern is simple enough to copy.&lt;/p&gt;

&lt;h2&gt;
  
  
  One engineer's experiment with agents on a schedule
&lt;/h2&gt;

&lt;p&gt;One of our engineers has been experimenting with agents that run on a schedule rather than on request. One agent triages CI every night. Another picks one small, open bug from Linear every morning and opens a PR to fix it. Tiny but non-critical is a category of work that people never get around to and agents don't mind.&lt;/p&gt;

&lt;p&gt;The same engineer also built a support bot that our solutions engineers use to draft answers, which a human then checks and sends. After it confidently described an unverified configuration as "confirmed working in a past similar case," it got two standing rules: search past resolutions before reasoning from scratch, and end every answer with what it is based on.&lt;/p&gt;

&lt;h2&gt;
  
  
  So what changed?
&lt;/h2&gt;

&lt;p&gt;In January, AI was a tool around the edges of our work. Now, thanks to models improving, our processes maturing, and simply the time it takes workflow changes to propagate, AI writes most of the code for many of the engineers who answered. The work that grew is making sure a person understood that code before it ships. It is certainly more tiring, and hopefully our next post on this will be about how we solved that.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>engineering</category>
      <category>productivity</category>
      <category>programming</category>
    </item>
    <item>
      <title>Build vs. Buy Your Kubernetes Development Environment</title>
      <dc:creator>Eyal Bukchin</dc:creator>
      <pubDate>Wed, 07 Oct 2026 10:43:26 +0000</pubDate>
      <link>https://dev.to/metalbear/build-vs-buy-your-kubernetes-development-environment-13hm</link>
      <guid>https://dev.to/metalbear/build-vs-buy-your-kubernetes-development-environment-13hm</guid>
      <description>&lt;p&gt;&lt;em&gt;Written by Arsh Sharma. Originally published on &lt;a href="https://metalbear.com/blog/build-vs-buy/" rel="noopener noreferrer"&gt;metalbear.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Every conversation about buying a dev tool now includes a brief silence in which someone mentally scopes building it instead. The instinct isn't new, engineers have always been one whiteboard session away from replacing any product with a quarterly project. But Claude Code has compressed that quarterly project into a weekend.&lt;/p&gt;

&lt;p&gt;And the truth is, with AI, a capable platform engineer probably &lt;em&gt;could&lt;/em&gt; build a working prototype in a sprint: something that connects a developer's code (running on a laptop, in an agent's sandbox, or in CI) and the remote staging cluster, letting them test against real services without deploying every single time. The mistake is thinking that sprint gets you the finished product. Noting the obvious bias we have as a vendor, here's what we've learned from building and maintaining a Kubernetes development platform: what the other 90% of the work looks like, what it actually costs, and how to decide whether buying or building is the right call.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is mirrord?
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://metalbear.com/mirrord/" rel="noopener noreferrer"&gt;mirrord&lt;/a&gt; is a Kubernetes development platform that lets developers and AI coding agents test code in a production-like environment before deploying it. Traditionally, giving developers a realistic environment to test in means one of two things: spinning up an ephemeral dev/test environment for each of them or having everyone wait their turn to deploy to staging. &lt;a href="https://metalbear.com/blog/enabling-developers-to-ship-cloud-native-applications-faster/" rel="noopener noreferrer"&gt;Both are slow&lt;/a&gt; because every iteration means building images and deploying. Ephemeral environments also get expensive fast since you're duplicating infrastructure for every developer.&lt;/p&gt;

&lt;p&gt;mirrord takes a different approach: all developers in the organization share a single staging cluster, and each runs their service wherever they're working, be it locally on their machine, in a CI runner, or in an agent's sandbox. mirrord then proxies incoming and outgoing traffic, environment variables, and files between the local process and its counterpart deployed in the cluster. The running service behaves as if it were deployed since it receives real requests and talks to the cluster's real services, databases, and queues. Because of this, the code behaves as it would in the context of the entire application without actually deploying it. And because everyone shares a cluster that already exists, there's nothing ephemeral to spin up and no duplicated infrastructure to pay for.&lt;/p&gt;

&lt;p&gt;At first glance, building what mirrord does might look like a sprint's worth of work. A platform engineer can stitch together a proof of concept with &lt;code&gt;kubectl port-forward&lt;/code&gt; and some scripting in a week or two: run the new version of a service beside the deployed one, tag requests with a header, and route only the tagged requests to it. The demo works, the new code is testable against real dependencies without deploying it, and paying for a tool starts to feel absurd when your own team just built the core of one in a sprint.&lt;/p&gt;

&lt;p&gt;LaunchDarkly &lt;a href="https://launchdarkly.com/blog/feature-management-platform-build-or-buy/" rel="noopener noreferrer"&gt;wrote about this exact pattern&lt;/a&gt; with feature flags: a feature flag is just an if statement, so building a feature management platform looks trivial until you realize that every capability that makes it actually useful turns out to be its own project, requiring you to invest more and more engineering time into it. Our version of "it's just an if statement" is "it's just a proxy." And in both cases, what people miss is that the demo is not the finished product.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the other 90% actually costs
&lt;/h2&gt;

&lt;p&gt;Here's what that hidden 90% of work after the initial prototype consists of.&lt;/p&gt;

&lt;h3&gt;
  
  
  Sharing the cluster is the hard part
&lt;/h3&gt;

&lt;p&gt;We've &lt;a href="https://metalbear.com/blog/celonis-mirrord/" rel="noopener noreferrer"&gt;written extensively&lt;/a&gt; about the benefits of shared environments over dedicated ephemeral ones in the past, so we'll assume here that the tool you're building targets a shared cluster, usually staging, like mirrord does.&lt;/p&gt;

&lt;p&gt;The tool has two jobs. The first is letting a developer quickly test their locally running code against that production-like cluster, and this is probably what your proof of concept will cover. The second job is letting every developer do this at the same time, against the same cluster, without stepping on each other. This is where the scope explodes, especially because developers aren't the only users anymore. With every developer now launching multiple agents and pointing them to the same cluster to verify the code they just generated, a hundred engineers can easily mean several hundred sessions sharing the cluster.&lt;/p&gt;

&lt;p&gt;The conflict everyone scopes for is traffic: two developers can't both receive all the requests hitting a service, so you obviously need &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/incoming-traffic/filter-incoming-traffic" rel="noopener noreferrer"&gt;HTTP filtering&lt;/a&gt; that routes each developer's requests to the right session, and you could say that part is fairly easy to build as well. What usually goes unscoped is everything else that collides. Two local processes compete for the same queue messages, so you need queue-splitting logic that routes each message to the right session. They write to the same database, so you need a way to give each developer an isolated copy. This is why mirrord has features like &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting" rel="noopener noreferrer"&gt;queue splitting&lt;/a&gt;, &lt;a href="https://metalbear.com/blog/db-branching/" rel="noopener noreferrer"&gt;database branching&lt;/a&gt;, and &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/policies" rel="noopener noreferrer"&gt;mirrord Policies&lt;/a&gt; that make sharing a cluster &lt;a href="https://metalbear.com/mirrord/case-study/zooplus/" rel="noopener noreferrer"&gt;among thousands of developers&lt;/a&gt; possible. Each is a significant engineering effort on its own, and none of them will be in your v1.&lt;/p&gt;

&lt;h3&gt;
  
  
  Your R&amp;amp;D org pays the real bill
&lt;/h3&gt;

&lt;p&gt;Say your five-person platform team spends six months building this. The visible cost is five salaries for six months plus some ongoing maintenance cost, and depending on your budget, that might look fine. But there's a bigger number. Every day the tool isn't finished is another day your hundred-person R&amp;amp;D org keeps working with the slow dev loop you set out to fix (&lt;a href="https://metalbear.com/blog/kubernetes-dev-loop-cost/" rel="noopener noreferrer"&gt;here's how to put a number on what that costs&lt;/a&gt;). The meter isn't just running on the team building the tool, it's running on everyone waiting for it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Finished isn't adopted
&lt;/h3&gt;

&lt;p&gt;Internal devtools only pay off if engineers actually use them, and the adoption bar is high: the tool should be easy to set up, easy to use, and stable enough that developers trust it with their daily workflow. That trust is hard to earn in the first place because developers don't rush to try untested internal tools. And it's even harder to earn back: a developer who tried v1, hit a rough edge, and went back to their old workflow isn't returning for v2 just because you tell them that things got better. If the experience is clunky, engineers are less likely to file feature requests and more likely to just quietly work around the tool and avoid using it.&lt;/p&gt;

&lt;p&gt;This creates an awkward loop for an in-house built tool. You don't reach that quality bar with just a good design document and a prompt. You reach it by iterating against real developer feedback repeatedly, which means that even after you're "done" building, you're not really done. You still need to see what features your team needs that are missing, what parts aren't stable, and constantly build towards them. But that iteration needs developers actually using the tool, and the rough early versions are exactly what teaches them to stay away. It's also the stage AI doesn't really speed up, because the bottleneck is developers hitting friction and telling you about it, not writing the code faster.&lt;/p&gt;

&lt;h3&gt;
  
  
  Long term support
&lt;/h3&gt;

&lt;p&gt;The work doesn't end once the tool is built, because the ground underneath it never stops moving: Kubernetes ships three releases a year, service mesh upgrades can change how traffic flows, and new protocols show up in your stack. Each of these lands on your team as a support request, and each one forces the same choice, either sink more engineering time into the in-house tool or leave the gap unsupported.&lt;/p&gt;

&lt;p&gt;What makes this a losing game internally is who you're doing it for. Every one of those investments serves the same few hundred engineers at your company, so each round of maintenance gets harder to justify against everything else on the platform team's backlog. That exact same work is easy for us to justify, because a fix or a newly supported protocol ships to tens of thousands of engineers across all the companies using mirrord.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why vibe coding this isn't as easy as it seems
&lt;/h2&gt;

&lt;p&gt;AI has changed the math around the build vs. buy argument: an agent can scaffold a proof of concept in an afternoon instead of a sprint, so building is cheaper than ever. The first half of that is true, and it actually makes the trap worse, because the demo now arrives even faster while the 90% below the waterline still remains a hard problem to solve.&lt;/p&gt;

&lt;p&gt;We have some direct experience here. mirrord is a low-level systems tool built in Rust, our engineering team uses AI every day while building it, and we've written candidly about &lt;a href="https://metalbear.com/blog/engineering-ai-use/" rel="noopener noreferrer"&gt;how that actually goes&lt;/a&gt;. AI is genuinely useful for things like getting oriented in unfamiliar code, exploring approaches, generating scripts to debug things. But all these require the code to already exist. Vibe coding a tool like this from scratch is a totally different experience. With no existing codebase to ground the model in, no domain experts to review the output, and months of sustained work instead of a single scoped session, you're trusting the model exactly where it's the weakest: short on context, unsupervised, and running for months.&lt;/p&gt;

&lt;p&gt;Even with engineers who know the architecture reviewing every line, and CLAUDE.md files we maintain specifically to feed models that context, we've had very few cases of AI shipping a real feature in mirrord without heavy manual intervention.&lt;/p&gt;

&lt;h2&gt;
  
  
  The TCO comparison
&lt;/h2&gt;

&lt;p&gt;Here's what the two options actually cost. The build numbers come from what we've seen teams scope once they get past the proof of concept.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Build internally&lt;/th&gt;
&lt;th&gt;Buy&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Time to a usable v1&lt;/td&gt;
&lt;td&gt;6+ months&lt;/td&gt;
&lt;td&gt;~1 week&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Project maturity at launch&lt;/td&gt;
&lt;td&gt;Your engineers discover every failure mode first&lt;/td&gt;
&lt;td&gt;Years of hardening across various customers (4+ years and 20,000+ users, in mirrord's case)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Shared-environment support&lt;/td&gt;
&lt;td&gt;Another 3-6 months of engineering work.&lt;/td&gt;
&lt;td&gt;Already supported through features like DB branching, queue splitting, and traffic filtering&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ongoing maintenance&lt;/td&gt;
&lt;td&gt;Your roadmap, indefinitely&lt;/td&gt;
&lt;td&gt;The vendor's roadmap, no maintenance on your side&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;When it breaks&lt;/td&gt;
&lt;td&gt;Your team gets paged, your developers wait&lt;/td&gt;
&lt;td&gt;A dedicated team that has probably seen the failure mode before&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  A decision checklist
&lt;/h2&gt;

&lt;p&gt;Questions worth answering with your team before committing either way:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Can you staff it like a product?&lt;/strong&gt; Not just 3–5 systems engineers for the six month, but permanent ownership afterward including being on-call and supporting upgrades.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does your v1 scope include more than the happy path?&lt;/strong&gt; If the plan doesn't mention queue splitting, database isolation, or mesh compatibility, the plan covers only the 10% above the waterline.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What does an outage cost you?&lt;/strong&gt; If the answer is "every developer is blocked," your internal tool needs production-grade reliability from day one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Will engineers actually adopt it?&lt;/strong&gt; If the experience isn't seamless, engineers work around it, and you end up maintaining a tool nobody uses.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does the plan account for AI agents?&lt;/strong&gt; Coding agents that test their own changes multiply concurrent sessions and need the same isolation guarantees as developers. Scoping for humans only means rebuilding for agents later.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Do you have a genuinely unusual constraint?&lt;/strong&gt; Regulatory requirements that no vendor can meet are real reasons to build.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you answered yes to most of these, building may honestly be the right call for you. For everyone else, the math points the other way.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where mirrord fits
&lt;/h2&gt;

&lt;p&gt;mirrord lets developers and AI agents run their code on a laptop, in a CI runner, or in an agent's sandbox while connected to a real cluster. Incoming traffic, outgoing calls, environment variables, and files all behave as if the process were running inside the cluster, with no image builds or deployments in the loop. And because a mirrord session is just a CLI command, an agent can run the code it just wrote against the real cluster and verify it actually works, instead of &lt;a href="https://metalbear.com/blog/after-ai-writes-code/" rel="noopener noreferrer"&gt;guessing or making assumptions&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The shared-environment features that make up the bulk of the iceberg (queue splitting, database branching, traffic filtering) are &lt;a href="https://metalbear.com/mirrord/docs" rel="noopener noreferrer"&gt;already built&lt;/a&gt;, which is how teams like &lt;a href="https://metalbear.com/mirrord/case-study/monday/" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; and &lt;a href="https://www.linkedin.com/posts/the-metalbear-team-had-the-pleasure-of-delivering-ugcPost-7476264187795533824-Ge1f/?utm_source=share&amp;amp;utm_medium=member_desktop&amp;amp;rcm=ACoAACZv8bIBvRMtv8vhCxDwho1OafMsqDmeHtM" rel="noopener noreferrer"&gt;National Australia Bank (NAB)&lt;/a&gt; run it for hundreds of engineers against a single shared environment.&lt;/p&gt;

&lt;p&gt;But that's not only what mirrord does. Since this is our main product we've built tons of features on top of it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://metalbear.com/blog/preview-environments/" rel="noopener noreferrer"&gt;&lt;strong&gt;Preview Environments&lt;/strong&gt;&lt;/a&gt;: Every pull request spins up a thin preview environment where only the work-in-progress service is duplicated in the cluster as an isolated pod. Reviewers like product managers, QA, or designers get a URL to see real behavior against real services before anything is merged.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://metalbear.com/blog/mirrord-ci/" rel="noopener noreferrer"&gt;&lt;strong&gt;mirrord for CI&lt;/strong&gt;&lt;/a&gt;: The same model applied to pipelines. Your PR's code runs in the CI runner but is tested against the real cluster, so there's no ephemeral environment to spin up in CI and no images to build and deploy, which saves teams 20–25 minutes per CI run, with concurrent runs kept isolated from each other and from developers using the same cluster.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://metalbear.com/mirrord/docs/managing-mirrord/admin-dashboard" rel="noopener noreferrer"&gt;&lt;strong&gt;Admin Dashboard&lt;/strong&gt;&lt;/a&gt;: A web interface for the platform team running mirrord, with real-time visibility into mirrord sessions, users, targets, CI pipelines, and adoption trends across the organization. Allows platform teams who buy the tools to show adoption, usage, and ROI to stakeholders.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Whichever way your team decides, price the whole iceberg and not just the part above the water.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently asked build vs. buy questions
&lt;/h2&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Should I build or buy a Kubernetes development environment?
&lt;/h3&gt;

&lt;p&gt;It depends on whether you can staff it like a product, not just a project. A capable platform team can prototype the core behavior you need in a sprint, but a production-grade version that supports your whole org with features like sharing a cluster across hundreds of developers without collisions, and getting maintained as Kubernetes and your stack evolve, usually takes 6+ months and permanent ongoing ownership. If you have that staffing and a genuinely unusual constraint a vendor can't meet, building can be the right call. For most teams, the math favors buying.&lt;/p&gt;

&lt;h3&gt;
  
  
  Does AI make it easier to build internal tools in-house?
&lt;/h3&gt;

&lt;p&gt;AI makes the demo faster to reach, not the finished product. AI coding agents are genuinely useful once a codebase already exists: getting oriented, exploring approaches, writing debug scripts. But building a Kubernetes development platform from scratch means months of sustained work with no existing codebase to ground the model and no domain experts reviewing every line, which is exactly the setup AI handles worst. It also doesn't speed up the part of the work that matters most for adoption: iterating on real developer feedback.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does it cost to maintain an in-house Kubernetes dev tool?
&lt;/h3&gt;

&lt;p&gt;Beyond the initial build, ongoing maintenance means keeping up with three Kubernetes releases a year, service mesh upgrades that can change how traffic flows, and new protocols showing up in your stack, each landing on your platform team as a support request. That work only serves the few hundred engineers at your company, so it competes against everything else on the team's backlog indefinitely. A vendor absorbs that same maintenance across tens of thousands of engineers, which is why it's easier for them to justify.&lt;/p&gt;

&lt;h3&gt;
  
  
  How does mirrord handle multiple developers sharing one staging cluster?
&lt;/h3&gt;

&lt;p&gt;mirrord uses features purpose-built for this: &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/incoming-traffic/filter-incoming-traffic" rel="noopener noreferrer"&gt;HTTP traffic filtering&lt;/a&gt; routes each developer's requests to their session, &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting" rel="noopener noreferrer"&gt;queue splitting&lt;/a&gt; routes each queue message to the right session, and &lt;a href="https://metalbear.com/blog/db-branching/" rel="noopener noreferrer"&gt;database branching&lt;/a&gt; gives each developer an isolated copy of the data they touch. These are already in use for &lt;a href="https://metalbear.com/mirrord/case-study/zooplus/" rel="noopener noreferrer"&gt;thousands of developers&lt;/a&gt; across companies like monday.com and NAB, rather than something a team has to build and harden on its own.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>platformengineering</category>
      <category>productivity</category>
    </item>
    <item>
      <title>How monday.com Runs Agent Evals Against Real Dependencies: Webinar Recap</title>
      <dc:creator>Arsh Sharma</dc:creator>
      <pubDate>Mon, 21 Sep 2026 11:08:48 +0000</pubDate>
      <link>https://dev.to/metalbear/how-mondaycom-runs-agent-evals-against-real-dependencies-webinar-recap-41ge</link>
      <guid>https://dev.to/metalbear/how-mondaycom-runs-agent-evals-against-real-dependencies-webinar-recap-41ge</guid>
      <description>&lt;p&gt;An agent eval suite's outcome can only be trustworthy if it's operating in an environment similar to production. You can have the best grading logic in the world, but if the agent is calling mocked databases and fake APIs, you're not testing how it behaves in the real world, you're testing how it behaves in the mocks you built. That gap is what we dug into on September 15, when we sat down with Dor Cohen, AI Engineering Director at &lt;a href="http://monday.com" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt;, for an interview and demo on how his team architected running agent evals against a live staging cluster instead of mocks.&lt;/p&gt;

&lt;p&gt;If you missed the session, here's the recap: what agent evals actually need to check, why mocks fall short compared to the agent calling real systems, and how &lt;a href="http://monday.com" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; built their agent evals infrastructure.&lt;/p&gt;

&lt;h2&gt;
  
  
  What are agent evals and why you need them
&lt;/h2&gt;

&lt;p&gt;Testing an agent isn't the same as testing a model. An agent is a package: the model, the tools it calls, and the dependencies behind those tools. Unlike a unit test, where you know exactly what output to expect, an agent is non-deterministic, as it can pick a different tool or path on every run. So the eval has to check the full trajectory it takes to get there, not just whether the final answer sounded right.&lt;/p&gt;

&lt;p&gt;Dor shared an example of why testing the full chain matters. They had an agent that was asked to retrieve 600 items. It called the right tools, formatted a confident, well-structured answer, and returned a response that read as correct. Except it had only processed about 500 of them. Every individual tool call worked. The final answer looked fine, but the task wasn't actually done. An eval that only grades the final answer would have passed it but an eval that checks trajectory and completeness catches it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where mocks fall short
&lt;/h2&gt;

&lt;p&gt;One very important part in the agent eval chain we discussed above is the dependencies your agent calls. The default way to run evals is against mocks because they're lightweight, fast to set up, and isolated, and for a lot of testing that's exactly what you want. But for an agent that's calling real databases, services, and APIs in production, mocks introduce three problems.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;They drift&lt;/strong&gt;: a mock has to be kept in sync with your production environment by hand, and it never is.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The data isn't real&lt;/strong&gt; either, synthetic rows present in mocks are cleaner and thinner than what actually lives in your database, so they can't expose the edge cases production will.&lt;/li&gt;
&lt;li&gt;And mocks &lt;strong&gt;can't represent state&lt;/strong&gt;. When an agent reads data and takes a downstream action, a mock either forgets the write happened or fakes it, so you can't actually tell what the agent left behind.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Staging doesn't have any of those problems by construction. It stays current with production because it's the last stop before it. It runs on real data, at close to production scale, with all the variety and mess that comes with that. And because the agent's actions actually land, you can check the real end state, not a simulated one. Fidelity to production is what makes or breaks the quality of your agent evals.&lt;/p&gt;

&lt;p&gt;The objection to running evals against staging (or other pre-prod environments) is reasonable, though: it's not lightweight, not isolated, and not easy to wire up. That's the problem &lt;a href="http://monday.com" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; solved with mirrord.&lt;/p&gt;

&lt;h2&gt;
  
  
  How mirrord fits into the pipeline
&lt;/h2&gt;

&lt;p&gt;mirrord connects a process running locally, in a CI job, or in an eval runner, to a real Kubernetes cluster, so it behaves as if it's deployed there: real traffic mirrored or stolen from a live pod, real env vars and secrets imported straight from the target, real reads and writes on the remote filesystem. &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/db-branching" rel="noopener noreferrer"&gt;DB branching&lt;/a&gt; and &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting" rel="noopener noreferrer"&gt;queue splitting&lt;/a&gt; isolate each eval session's writes, so multiple runs, from different engineers or different CI jobs, can hit the same cluster at the same time without stepping on each other's data.&lt;/p&gt;

&lt;p&gt;There's no separate deploy step as well, you just wrap your existing eval command and run it with mirrord:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mirrord &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nt"&gt;--target&lt;/span&gt; deployment/orders-api &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;--&lt;/span&gt; pytest evals/agent_suite.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And in CI you can just use &lt;a href="https://metalbear.com/mirrord/docs/use-cases/mirrord-for-ci" rel="noopener noreferrer"&gt;&lt;code&gt;mirrord ci start&lt;/code&gt;&lt;/a&gt; instead of &lt;code&gt;mirrord exec&lt;/code&gt;, with the same arguments. When a model, a prompt, or a tool changes, your CI job runs the suite against real dependencies before the change reaches a merge gate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Inside monday.com's agent evals pipeline
&lt;/h2&gt;

&lt;p&gt;monday.com runs 700+ microservices behind its agents, with the usual sprawl that comes with that scale: authentication, feature flags, permission layers, and a long list of third-party integrations. Mocking these dependencies or setting them up each time in an ephemeral environment would've been a huge investment for them in terms of both time and money. Their estimate for a dedicated evaluation environment came out to about a month of setup, plus ongoing maintenance across 700+ services just to keep it in sync. Obviously, they ended up going the more practical route of pointing their evals at staging with mirrord, which their developers &lt;a href="https://metalbear.com/mirrord/case-study/monday/" rel="noopener noreferrer"&gt;were already using&lt;/a&gt; for local development.&lt;/p&gt;

&lt;p&gt;The difference shows up in the day-to-day now: agent eval suites run in minutes instead of hours, the added infrastructure cost is close to zero since &lt;a href="http://monday.com" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; already pays for staging, and they get an output that they can actually trust.&lt;/p&gt;

&lt;h3&gt;
  
  
  Catching a regression before it ships
&lt;/h3&gt;

&lt;p&gt;monday.com treats a model upgrade the same way it treats any other version change to the agent: it runs the full eval suite against the new model before switching. When the team tested Sonnet 5 against their existing default model, goal completion dropped by about 10%, and agent correctness dropped by about the same. The suite failed to cross the pass threshold.&lt;/p&gt;

&lt;p&gt;This was great for the team as it gave them a clear signal that the upgrade wasn't safe to ship as-is, pointed to specific failure points in tool selection and prompt adherence, and gave them a concrete direction for what to adjust before trying again. Without a suite running against real dependencies, this is the kind of regression that tends to surface only after a customer hits it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Three ways monday.com actually runs AI agent evals
&lt;/h3&gt;

&lt;p&gt;Dor shared that their eval pipeline isn't just limited to CI pipelines, it's the same underlying infrastructure triggered from three different places, all producing consistent results through mirrord and LangSmith.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;From Claude Code, a developer working locally can invoke an offline evals skill right after making a change. It asks for the eval scope, all cases or a specific category, connects to dependencies living in staging through mirrord, and runs the suite. Results land in LangSmith with input, reference output, actual output, and a score per metric.&lt;/li&gt;
&lt;li&gt;From Slack, &lt;a href="http://monday.com" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; has an internal Claude-based evaluation agent, nicknamed "Eevee," that can be asked to run evals with specific parameters, like swapping in a different model for a test run. It uses the same infrastructure, produces the same LangSmith output, and posts a summary back in Slack. This is actually how the team caught the ~10% drop in goal completion during the Sonnet 5 test.&lt;/li&gt;
&lt;li&gt;From CI, eval results show up alongside regular checks on the pull request, each with a score and a mode: observed, which is informational, or blocked, which prevents the merge, depending on the severity of the change and the product it's touching.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;monday.com runs these "offline evals" alongside "online" ones. Offline evals run against their staging environment with controlled datasets, gating changes before they ship. Online evals grade live agent behavior continuously after it is shipped to production. When something surfaces in production that the offline dataset didn't cover, it gets pulled into that dataset for future regression testing, closing the loop between what ships and what gets tested next.&lt;/p&gt;

&lt;h3&gt;
  
  
  Metrics to track for agent evals
&lt;/h3&gt;

&lt;p&gt;monday.com's base eval pipelines track six metrics:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Goal completion: did the agent finish the whole task, not just appear to&lt;/li&gt;
&lt;li&gt;Agent correctness: was the answer factually and logically right&lt;/li&gt;
&lt;li&gt;Tool precision: did it pick the correct tools for the task and call them in the correct sequence&lt;/li&gt;
&lt;li&gt;Trajectory: was the path taken the expected one&lt;/li&gt;
&lt;li&gt;Groundedness: are the agent's claims backed by evidence it actually had&lt;/li&gt;
&lt;li&gt;Intercompleteness: did it gather everything it needed before answering&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Some of these are fully deterministic, like whether the required tool got called. Others need an LLM-as-judge, for things like whether the agent understood the goal or stayed in scope. And individual product teams at &lt;a href="http://monday.com" rel="noopener noreferrer"&gt;monday.com&lt;/a&gt; add their own domain-specific evaluators on top of the base set, because what counts as a passing score depends entirely on the product and the use case.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;p&gt;A few things worth carrying away from the session:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agent evals should test more than just the final output: tools picked, dependencies used, trajectory taken, and completeness all matter.&lt;/li&gt;
&lt;li&gt;Mocks drift, and synthetic test data can't replicate the variety and edge cases that live in production.&lt;/li&gt;
&lt;li&gt;Most teams already have the right environment for running agent evals properly. Staging is current with production by definition. Building a dedicated eval environment from scratch is usually more expensive and slower than pointing your suite at the one you already run.&lt;/li&gt;
&lt;li&gt;Online and offline evals reinforce each other. Edge cases from production belong in your offline dataset, so the two tracks compound over time instead of drifting apart.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you'd like to see the full webinar recording including the demo Dor showed, you can get it &lt;a href="https://attendee.gotowebinar.com/recording/5989946292184245593" rel="noopener noreferrer"&gt;here&lt;/a&gt;. And if you want to set up a 1:1 session with one of our engineers who'll walk you through how to connect your agent evals to real dependencies with mirrord, you can &lt;a href="https://metalbear.com/contact/" rel="noopener noreferrer"&gt;book a demo with us&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>llm</category>
      <category>agents</category>
    </item>
    <item>
      <title>How To Reduce Token Costs of AI Coding Agents</title>
      <dc:creator>Arsh Sharma</dc:creator>
      <pubDate>Thu, 13 Aug 2026 11:01:59 +0000</pubDate>
      <link>https://dev.to/metalbear/how-to-reduce-token-costs-of-ai-coding-agents-5772</link>
      <guid>https://dev.to/metalbear/how-to-reduce-token-costs-of-ai-coding-agents-5772</guid>
      <description>&lt;p&gt;Costs related to AI coding tools &lt;a href="https://www.gartner.com/en/newsroom/press-releases/2026-06-24-gartner-predicts-ai-coding-costs-will-surpass-average-developer-salary-by-2028-as-token-consumption-surges" rel="noopener noreferrer"&gt;are projected&lt;/a&gt; to only go up from what they are today. We've seen this before with Anthropic adding weekly rate limits on top of Claude Pro and Max's existing five-hour windows, &lt;a href="https://venturebeat.com/ai/anthropic-throttles-claude-rate-limits-devs-call-foul" rel="noopener noreferrer"&gt;with little warning&lt;/a&gt;. So the question on everyone's minds these days is how do you get more work out of the same AI budget.&lt;/p&gt;

&lt;p&gt;One answer is you have your AI agents burn less tokens. But token burn isn't one problem. An agent stuck looping on a bug it can't see and an agent reloading the same instructions on every single prompt look identical on your bill: a long session, a high token count. But the causes are completely different, and so are the fixes.&lt;/p&gt;

&lt;p&gt;In this post, we'll go through five specific ways this happens:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Agents reading half your repo to change three files&lt;/li&gt;
&lt;li&gt;Agents doing mechanical work on your most expensive model&lt;/li&gt;
&lt;li&gt;Agents guessing at what a live dependency looks like, or trusting a mock that drifted away from it&lt;/li&gt;
&lt;li&gt;Agents looping without any way to tell if their fix actually worked&lt;/li&gt;
&lt;li&gt;Agents loading instructions on every prompt instead of only when needed&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For #3 and #4 we'll show how &lt;a href="https://metalbear.com/mirrord/" rel="noopener noreferrer"&gt;mirrord&lt;/a&gt;, our Kubernetes development platform, closes the gap: it connects code to a real cluster without deploying it, whether that code runs on a laptop, in CI, or in an agent's sandbox.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading half the repo to change three files
&lt;/h2&gt;

&lt;p&gt;Point an agent at a large codebase with a vague instruction and it will go exploring, and that exploration is usually the single largest line item in a session: file after file pulled into context, most of which never ends up informing the change. What makes it expensive isn't the reading itself so much as the compounding, because a file read on turn three is still being re-sent on turn forty.&lt;/p&gt;

&lt;p&gt;The easiest fix to this is just &lt;strong&gt;being specific in your prompts&lt;/strong&gt;. If you already know the change lives in three files, just say so. If the search genuinely is the task, hand it to a subagent, which does the fan-out reading and returns a conclusion, so the forty files it opened never enter your main context at all. And when you move on to something unrelated, &lt;strong&gt;start a fresh session rather than continuing the old one&lt;/strong&gt;, because a long-running session carries its full history into every new request whether or not any of it is still relevant.&lt;/p&gt;

&lt;h2&gt;
  
  
  Paying premium model rates for mechanical work
&lt;/h2&gt;

&lt;p&gt;Not every task needs your best model. Renaming a symbol across a repo, writing a migration from a schema you've already specified, reformatting test fixtures, drafting a commit message: these are all mechanical, and a smaller model does them at a fraction of the cost.&lt;/p&gt;

&lt;p&gt;The mistake is treating model choice as a per-project setting rather than a per-task one. Reach for the expensive model when the work is genuinely architectural, when you're debugging something subtle or designing an interface or deciding how a change should be structured, and drop down for everything downstream of that decision. &lt;strong&gt;Most agent tools let you switch mid-session, and most people never utilize that&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Guessing at environment shape vs. reading it live
&lt;/h2&gt;

&lt;p&gt;An AI agent writing code for a microservice doesn't get to see the cluster where the entire application is deployed. It only sees your repo, whatever context you gave it, and patterns from its training data, and it fills every other gap with an assumption: what response a downstream service returns, what a queue message actually looks like, what a config value resolves to once it's deployed. Some of those assumptions are wrong, and the agent has no way to know which ones until the code runs against the real thing.&lt;/p&gt;

&lt;p&gt;Say you ask an agent to add a consumer for order events, and it writes the handler expecting a flat &lt;code&gt;total_cents&lt;/code&gt; field, because that's what the schema doc in the repo still says. In staging, that event has looked like &lt;code&gt;{"pricing": {"total": ...}}&lt;/code&gt; for the last two months, ever since a migration that never made it back into the docs. The agent doesn't find out until it runs the code and the field comes back &lt;code&gt;None&lt;/code&gt;, and at that point fixing it means re-deriving the real shape from the failure and rewriting the parsing logic from scratch. Every downstream call, config value, queue, or database the code touches carries the same risk, and each wrong guess costs a whole new bag of tokens.&lt;/p&gt;

&lt;p&gt;The usual answer to this is a mock, and mocks make it worse rather than better, because now the agent isn't missing information, it's being handed information that's confidently wrong. A mock is a snapshot of what a dependency looked like the day someone created it, and nothing keeps that snapshot in sync automatically, so the gap only ever grows. Say the real order queue gains a required &lt;code&gt;currency&lt;/code&gt; field, and the agent knows about it from a recent PR, so it writes a consumer that correctly expects the field. Locally it fails, because the mock is still emitting the old shape. The agent has no way to know the mock is the outdated one, so it assumes its own code is wrong and starts "fixing" it, loosening the field to optional, adding fallback logic, trying variation after variation, none of which touch the actual bug because there isn't one. The bug is the mock. The agent just burned a dozen turns proving that correct code doesn't satisfy an incorrect test.&lt;/p&gt;

&lt;p&gt;The fix for both is to &lt;strong&gt;stop guessing&lt;/strong&gt;. With &lt;a href="https://metalbear.com/mirrord/" rel="noopener noreferrer"&gt;mirrord&lt;/a&gt; connected to your cluster, an agent can read the real shape of a dependency before it writes a single line: what's actually on the queue right now, what a service returns today, or what a ConfigMap resolves to in the running pod. &lt;strong&gt;That collapses the guess-then-fail-then-regenerate cycle most AI-generated code goes through into just one or two passes, because the first draft is already written against what's true instead of what's assumed&lt;/strong&gt;. And there's no need to maintain mocks that drift, because you can get a response from the dependencies running in the cluster directly.&lt;/p&gt;

&lt;h2&gt;
  
  
  No integration feedback
&lt;/h2&gt;

&lt;p&gt;Without a way to verify its own work, an agent can make a change, declare it done, and open a PR with a bug baked in that only shows up once the code touches the real dependencies in the cluster. Nobody catches it at write time. It gets caught in review instead after a CI run or a staging deploy, which means going back, explaining what's actually wrong, and waiting on another round of fixes, and possibly another after that if the first fix is incomplete. Every one of those review-fix-review rounds costs tokens the agent wouldn't have spent if it had caught the bug itself the moment it introduced it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Give the agent the same tests your team already runs in CI, but let it run them itself, against a real environment, right after it makes a change&lt;/strong&gt;. Then it catches that kind of bug in the same turn it introduced it, not two review cycles later. We've walked through exactly this kind of case in detail in our previous blog &lt;a href="https://metalbear.com/blog/how-to-prevent-token-burn-using-mirrord-with-e2e-tests/" rel="noopener noreferrer"&gt;How to prevent token burn using mirrord with e2e tests&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Context loaded on every prompt vs. on-demand skills
&lt;/h2&gt;

&lt;p&gt;Most AI coding agents support an instructions file like &lt;a href="https://agents.md/" rel="noopener noreferrer"&gt;AGENTS.md&lt;/a&gt; or &lt;a href="https://code.claude.com/docs/en/memory#claude-md-vs-auto-memory" rel="noopener noreferrer"&gt;CLAUDE.md&lt;/a&gt; that the agent reads in full on every single prompt in a session. It's the simplest way to tell an agent "always do X, never do Y," but that simplicity has a cost baked in. The file loads regardless of whether the current prompt has anything to do with it. If you've added a paragraph asking the agent to use mirrord to verify its code, one about your auth conventions, one about deployment rules, the agent is now paying to re-read all three on every request, including one that touches none of them.&lt;/p&gt;

&lt;p&gt;Agent Skills fix this by making the loading conditional. Each skill ships as its own file with its own description, and the model only pulls it into context when a prompt actually matches what that skill covers. We packaged mirrord's own instructions this way: &lt;a href="https://metalbear.com/blog/claude-code-skills-for-kubernetes/" rel="noopener noreferrer"&gt;six Agent Skills&lt;/a&gt; covering commonly used mirrord use cases, installed in one command. Ask an agent to set up &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting" rel="noopener noreferrer"&gt;queue splitting&lt;/a&gt; and only the &lt;code&gt;mirrord-kafka&lt;/code&gt; skill loads. Ask it to fix a typo in your frontend and none of the six load because nothing in that prompt matches any of them. &lt;strong&gt;The rule of thumb is that your always-loaded file should only hold what's true for every prompt, and everything else belongs in a skill.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Token costs are going up, but your bill doesn't have to
&lt;/h2&gt;

&lt;p&gt;The first two scenarios trace back to the same root cause: an agent that can't see what's actually true about your cluster, so it guesses at a dependency's shape, gets fooled by a mock that no longer matches reality, or ships a change with no way to check it actually works. mirrord closes that gap by connecting the agent's locally running code to your real cluster. The agent now knows what's actually on the queue, what a service returns, what the values of the environment variables the process inherits in the cluster are, so it is testing against reality instead of an assumption.&lt;/p&gt;

&lt;p&gt;The other three, on-demand skills and tighter context scoping and matching the model to the task, are solving different problems, but the same principle applies to all of them: only pay for what the current prompt actually needs, and not a wasted guess, an irrelevant paragraph of instructions, or forty files the agent read once and re-sends forever.&lt;/p&gt;

&lt;p&gt;If your AI agents are writing code for Kubernetes services, &lt;a href="https://metalbear.com/mirrord/ai/" rel="noopener noreferrer"&gt;give mirrord a try&lt;/a&gt;. Our &lt;a href="https://metalbear.com/mirrord/docs/use-cases/using-mirrord-with-ai" rel="noopener noreferrer"&gt;guide on using mirrord with AI agents&lt;/a&gt; walks through the setup for each of these workflows.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>kubernetes</category>
      <category>microservices</category>
      <category>programming</category>
    </item>
    <item>
      <title>How Prathyusha Ramanjaneyulu at Celonis Eliminated Dev Environment Overhead for 600 Engineers</title>
      <dc:creator>Arsh Sharma</dc:creator>
      <pubDate>Wed, 01 Jul 2026 13:05:29 +0000</pubDate>
      <link>https://dev.to/metalbear/how-prathyusha-ramanjaneyulu-at-celonis-eliminated-dev-environment-overhead-for-600-engineers-5h19</link>
      <guid>https://dev.to/metalbear/how-prathyusha-ramanjaneyulu-at-celonis-eliminated-dev-environment-overhead-for-600-engineers-5h19</guid>
      <description>&lt;p&gt;I recently spoke with Prathyusha Ramanjaneyulu, a Senior Software Engineer at Celonis. She owns the local development component of the organization's internal developer tooling and is responsible for maintaining high velocity for over 600 developers. The conversation was about a problem that comes up at nearly every large engineering org: how do you give hundreds of developers a way to test their code against a real environment without having them being slowed down by CI and staging deployments? She described environments that took weeks to set up, 3+ hours a week lost to VM maintenance, and teams blocked whenever someone pushed a bad change. This is a recap of that conversation. I'll share the approaches she was using, the issues she faced with those, and how mirrord helped her finally solve this problem.&lt;/p&gt;

&lt;p&gt;The problems she told me are ones I think many engineers at large companies will relate to and can learn from.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Nature of the Bottleneck at Scale
&lt;/h2&gt;

&lt;p&gt;The engineering org she works with has around 600 developers building applications that have a microservices architecture and are deployed on Kubernetes. The nature of that setup means that when an engineer is working on a single service, their code doesn't run in isolation but it depends on a web of other services, databases, and queues. Testing even a small change means the service needs to interact with those dependencies in a way that reflects what actually happens in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  What she had tried
&lt;/h2&gt;

&lt;p&gt;Before landing on a solution that worked, she went through two approaches.&lt;/p&gt;

&lt;h3&gt;
  
  
  Approach 1: Run everything locally
&lt;/h3&gt;

&lt;p&gt;Every engineer would clone, compile, and run all the services their code depended on locally, which in an org of that size meant a lot of repos and a lot of compute. And even after spending the time to do all that, the machines were so resource-constrained while running all of it that engineers couldn't do much else at the same time. And because each engineer was running services they hadn't written, they'd inevitably run into issues they had no context to debug. This led to them spending time on problems that had nothing to do with the code they were actually trying to test.&lt;/p&gt;

&lt;h3&gt;
  
  
  Approach 2: Shared VM + some local services
&lt;/h3&gt;

&lt;p&gt;The second approach moved some services to a shared virtual machine (VM) while keeping others running locally. That got the initial environment setup time down from the weeks required by the all-local approach to roughly a day and a half, which was meaningful progress. But it introduced a different set of problems. Engineers were spending 3+ hours a week on VM maintenance and things like debugging connectivity issues. This was time that had nothing to do with writing code. And because the VM was shared, when one engineer's change broke something, everyone working against that environment was blocked until it was resolved. The bottleneck had moved, but it hadn't gone away.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Selection Criteria for Scaling Local Development
&lt;/h2&gt;

&lt;p&gt;As the owner of the Local Development component, Prathyusha is tasked with maintaining high engineering velocity for 600 developers. She mentioned her team maintains a proactive watch over the cloud-native ecosystem; it was through this strategic evaluation that she and her team identified both &lt;a href="https://telepresence.io/" rel="noopener noreferrer"&gt;Telepresence&lt;/a&gt; and &lt;a href="https://metalbear.com/mirrord/" rel="noopener noreferrer"&gt;mirrord&lt;/a&gt; as potential solutions to address their bottlenecks.&lt;/p&gt;

&lt;p&gt;Both tools solve the same core problem: connecting a local development environment to a remote Kubernetes cluster. Here's the reasoning she gave me for landing on mirrord.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Philosophy.&lt;/strong&gt; Telepresence's model is "your workstation becomes part of the cluster network." mirrord's model is "your local process impersonates a process inside a pod." With Telepresence, a broader portion of your machine inherits cluster connectivity and behavior. With mirrord, only the specific process being run is affected. That narrower scope reduces blast radius and configuration complexity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Operational overhead.&lt;/strong&gt; Telepresence requires managing daemon processes and per-user configuration across the team, which adds ongoing maintenance work for the platform team. mirrord is plug-and-play. Engineers pick it up and use it without her having to maintain it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Security.&lt;/strong&gt; Telepresence relies on Preview URLs, which are public endpoints, to enable collaboration. mirrord operates entirely within the private network, so there's no risk of accidentally exposing data through a public endpoint.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resource footprint.&lt;/strong&gt; mirrord only consumes cluster resources during an active debug session. Nothing persists between sessions, which keeps the staging cluster clean and means the platform team isn't maintaining persistent dev-tooling components on top of everything else.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What changed
&lt;/h2&gt;

&lt;p&gt;When Prathyusha walked me through the impact, I asked for the numbers first. She mentioned that with the all-local approach, environment setup took weeks per engineer. With the shared VM, that came down to about a day and a half. With mirrord, it takes around 15 minutes. That's a 144x speedup over the shared VM approach! She also mentioned how the time between an engineer writing code and knowing if it works went from an hour down to seconds.&lt;/p&gt;

&lt;p&gt;But beyond the speed, she pointed to the elimination of an entire category of problem. With mirrord, testing locally became equivalent to testing in staging as engineers connect their local process directly to the real cluster, with real dependencies. That closed the environment gap that caused the "it works on my machine but not in staging" problem. She described it as a shift in the confidence of the engineers: if it works with mirrord, engineers don't have to wonder whether it'll work in staging or not.&lt;/p&gt;

&lt;p&gt;The shared environment also solved the blocking problem that plagued the shared VM approach. With the VM, one engineer's bad change could bring the environment down for everyone. With mirrord, any number of engineers can test against the same staging cluster at the same time without affecting each other. mirrord handles the isolation through features like &lt;a href="https://metalbear.com/mirrord/docs/using-mirrord/incoming-traffic/filter-incoming-traffic" rel="noopener noreferrer"&gt;HTTP traffic filtering&lt;/a&gt;, &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/queue-splitting" rel="noopener noreferrer"&gt;queue splitting&lt;/a&gt; and &lt;a href="https://metalbear.com/mirrord/docs/sharing-the-cluster/db-branching" rel="noopener noreferrer"&gt;DB branching&lt;/a&gt; which ensure each engineer's test traffic and changes don't affect other people using the environment.&lt;/p&gt;

&lt;p&gt;What all of this added up to was engineers getting their time back.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;No more 3+ hours a week wasted on VM maintenance and configuration.&lt;/li&gt;
&lt;li&gt;No debugging of services they didn't write.&lt;/li&gt;
&lt;li&gt;No waiting on the dev tools team to fix a broken shared environment.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To put the capacity impact in perspective: if mirrord saves each engineer just one hour per week, at 600 engineers that's 600 hours reclaimed every week. Over a year, that's 30,000 hours of engineering capacity redirected from infrastructure maintenance to product work.&lt;/p&gt;

&lt;h2&gt;
  
  
  Closing remarks
&lt;/h2&gt;

&lt;p&gt;The thing that struck me most from talking with Prathyusha was how much of the improvement came from changing the model, not the tooling. At this scale, even small amounts of per-engineer overhead compound into a really big cost for the organization as a whole. But the solution wasn't faster local tooling or more compute. It was changing the model: instead of every engineer maintaining their own isolated environment, share one staging cluster and just have the DevOps team maintain that.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>productivity</category>
      <category>docker</category>
    </item>
    <item>
      <title>Auto-verifying your AI-SRE's fixes (Part II): HolmesGPT end-to-end on a real cluster</title>
      <dc:creator>Eyal Bukchin</dc:creator>
      <pubDate>Wed, 24 Jun 2026 12:18:12 +0000</pubDate>
      <link>https://dev.to/metalbear/auto-verifying-your-ai-sres-fixes-part-ii-holmesgpt-end-to-end-on-a-real-cluster-594p</link>
      <guid>https://dev.to/metalbear/auto-verifying-your-ai-sres-fixes-part-ii-holmesgpt-end-to-end-on-a-real-cluster-594p</guid>
      <description>&lt;p&gt;&lt;em&gt;Part II of two. See &lt;a href="https://metalbear.com/blog/ai-sre/" rel="noopener noreferrer"&gt;Part I&lt;/a&gt; for the recipe.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In &lt;a href="https://metalbear.com/blog/ai-sre/" rel="noopener noreferrer"&gt;Part I&lt;/a&gt; we've discussed how you can plug mirrord into your AI-SRE so it can autonomously test its fix in the real cluster. In this post, we'll show this running end-to-end on a real GKE cluster against &lt;a href="https://github.com/robusta-dev/holmesgpt" rel="noopener noreferrer"&gt;HolmesGPT&lt;/a&gt;, the only OSS AI-SRE we found that is self-hostable. We planted two bugs to test both possible verdicts: PASS when the patched run clears the alert's SLO without any regressions, REJECT when it doesn't (the patched run still violates the alert's SLO, or cleared the SLO but added a regression).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Setup:&lt;/strong&gt; &lt;strong&gt;HolmesGPT&lt;/strong&gt; investigates an alert and returns a markdown report. A &lt;strong&gt;small Claude wrapper&lt;/strong&gt; turns the report into a code patch. The &lt;strong&gt;verifier&lt;/strong&gt; runs that patch under &lt;code&gt;mirrord exec&lt;/code&gt;, compares against an unpatched baseline, and returns a verdict tied to the alert's actual SLO.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fss4lp3lpxeumlmozf1qj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fss4lp3lpxeumlmozf1qj.png" alt="AI-SRE verification architecture" width="800" height="1185"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;What's HolmesGPT?&lt;/strong&gt; A prebuilt LLM-backed agent tailored for on-call work: it can run &lt;code&gt;kubectl&lt;/code&gt; commands, fetch logs from the cluster, and read the runbooks you've already written (in Notion, Confluence, or markdown files) to ground its investigation. Without it, you'd be building this scaffolding around Claude yourself.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The demo cluster
&lt;/h2&gt;

&lt;p&gt;A Python service called &lt;code&gt;checkout&lt;/code&gt; handles HTTP requests; on each request it calls a &lt;code&gt;pricing&lt;/code&gt; service for the item price. A &lt;code&gt;loadgen&lt;/code&gt; pod hits &lt;code&gt;checkout&lt;/code&gt; continuously to keep the system warm. Prometheus scrapes &lt;code&gt;checkout&lt;/code&gt;'s metrics; Alertmanager fires when SLOs are violated. Everything sits in one namespace.&lt;/p&gt;

&lt;p&gt;HolmesGPT runs as a pod in the cluster, subscribed to Alertmanager. When an alert fires, HolmesGPT investigates and hands its markdown report off to a separate verifier pod. The verifier runs the bridge (the Claude call that turns the report into a code patch), then uses &lt;code&gt;mirrord exec&lt;/code&gt; to run the patched code inside the verifier pod with &lt;code&gt;deploy/checkout&lt;/code&gt;'s network identity, env, and mounts. When the patched copy calls &lt;code&gt;pricing&lt;/code&gt;, it hits the real &lt;code&gt;pricing&lt;/code&gt; pod.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1k20qfp7va1thxax4wzh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1k20qfp7va1thxax4wzh.png" alt="Demo cluster layout" width="800" height="905"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Scenario 1: error-rate alert (PASS)
&lt;/h2&gt;

&lt;p&gt;A recent change introduced a request format &lt;code&gt;checkout()&lt;/code&gt; doesn't handle: every request for an &lt;code&gt;item_id&lt;/code&gt; ending in &lt;code&gt;-3&lt;/code&gt; raises &lt;code&gt;ValueError&lt;/code&gt; and returns 500. Loadgen sends a mix; ~10% of requests fail. A Prometheus rule fires &lt;code&gt;CheckoutErrorRateHigh&lt;/code&gt; once the error rate climbs over 5%.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ask HolmesGPT to investigate.&lt;/strong&gt; This is the call HolmesGPT runs in-cluster when Alertmanager fires:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;holmes investigate alertmanager &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--alertmanager-url&lt;/span&gt; http://localhost:9093 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--alertmanager-alertname&lt;/span&gt; CheckoutErrorRateHigh &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--model&lt;/span&gt; &lt;span class="s1"&gt;'anthropic/claude-sonnet-4-20250514'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;HolmesGPT investigated for ~30 seconds (pulled pod descriptions, fetched logs, walked the service config) and concluded:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Root Cause:&lt;/strong&gt; Application logic error causing 500 responses for specific item (item-3)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Error Details:&lt;/strong&gt; Error rate: 20.09% (above 5% SLO). Specific error: &lt;code&gt;ValueError: unsupported catalog shape for item_id=item-3&lt;/code&gt;. Pattern: repeated failures for item-3 checkout requests returning HTTP 500.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Remediation:&lt;/strong&gt; Fix application code to handle item-3 catalog shape or add proper validation.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Clean diagnosis: HolmesGPT pulled the exception out of the logs and attributed it correctly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bridge to a code patch.&lt;/strong&gt; HolmesGPT outputs a markdown report, not a &lt;code&gt;git diff&lt;/code&gt;. The bridging step is a small Claude call running in the verifier pod that takes the report plus the service's source and returns a code patch. We asked it to faithfully implement HolmesGPT's recommendation: handle the item-3 case (Claude chose to return zero) rather than raising. Claude produced the minimal edit to &lt;code&gt;checkout.py&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Verify.&lt;/strong&gt; Two runs (baseline and patched), 100 requests each. The verifier compares the alert's SLO condition against the patched run, plus a regression watchlist on the other signals. The "baseline" here is the verifier's own load test on the unpatched code, same load and same downstream as the patched run, not the live error rate HolmesGPT observed. That's why the number below differs from the 20.09% in HolmesGPT's report.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SLO under verification:&lt;/strong&gt; &lt;code&gt;error_rate &amp;gt; 5%&lt;/code&gt; (alert fires while this holds)&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Baseline&lt;/th&gt;
&lt;th&gt;Patched&lt;/th&gt;
&lt;th&gt;SLO threshold&lt;/th&gt;
&lt;th&gt;Status&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Error rate&lt;/td&gt;
&lt;td&gt;10.00%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.00%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;5.00%&lt;/td&gt;
&lt;td&gt;✅ satisfies&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Regression watchlist&lt;/strong&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;Baseline&lt;/th&gt;
&lt;th&gt;Patched&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p50 latency (ms)&lt;/td&gt;
&lt;td&gt;469.2&lt;/td&gt;
&lt;td&gt;470.5&lt;/td&gt;
&lt;td&gt;+0.3%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 latency (ms)&lt;/td&gt;
&lt;td&gt;505.7&lt;/td&gt;
&lt;td&gt;531.8&lt;/td&gt;
&lt;td&gt;+5.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Verdict: PASS.&lt;/strong&gt; Alert condition &lt;code&gt;error_rate &amp;gt; 5%&lt;/code&gt; no longer satisfied (10% → 0%). Regression watchlist clean.&lt;/p&gt;

&lt;h2&gt;
  
  
  Scenario 2: latency alert (REJECT)
&lt;/h2&gt;

&lt;p&gt;The error-rate bug was easy: a literal exception in the logs with an obvious fix. What does the loop look like for a less log-visible bug? We swapped in the second planted defect: &lt;code&gt;fetch_price()&lt;/code&gt; has no client-side timeout, and &lt;code&gt;pricing&lt;/code&gt; has a long tail: 1 in 10 calls takes ~1.5s. p99 drifts to ~2 seconds, well above the 300ms SLO. &lt;code&gt;CheckoutP99High&lt;/code&gt; fires.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ask HolmesGPT:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;holmes investigate alertmanager &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--alertmanager-url&lt;/span&gt; http://localhost:9093 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--alertmanager-alertname&lt;/span&gt; CheckoutP99High &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--model&lt;/span&gt; &lt;span class="s1"&gt;'anthropic/claude-sonnet-4-20250514'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Root Cause Analysis:&lt;/strong&gt; Latency caused by synchronous dependency calls (each checkout request triggers pricing API calls), no apparent caching, potential bottleneck in pricing service.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Recommendations:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Implement caching for pricing data to reduce API calls&lt;/li&gt;
&lt;li&gt;Add async processing for non-critical pricing lookups&lt;/li&gt;
&lt;li&gt;Monitor pricing service performance and scaling&lt;/li&gt;
&lt;li&gt;Consider request batching to pricing service&lt;/li&gt;
&lt;/ol&gt;
&lt;/blockquote&gt;

&lt;p&gt;Caching is a reasonable optimization. It's also generally a good idea. The question is whether it resolves &lt;em&gt;this specific alert&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bridge:&lt;/strong&gt; Of the four recommendations, only caching is a single code change; the others are runtime tuning or architectural refactors. Claude implemented it as an in-memory TTL cache around &lt;code&gt;fetch_price()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Verify.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SLO under verification:&lt;/strong&gt; &lt;code&gt;p99 latency (ms) &amp;gt; 300&lt;/code&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Baseline&lt;/th&gt;
&lt;th&gt;Patched&lt;/th&gt;
&lt;th&gt;SLO threshold&lt;/th&gt;
&lt;th&gt;Status&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p99 latency (ms)&lt;/td&gt;
&lt;td&gt;2162.5&lt;/td&gt;
&lt;td&gt;2010.3&lt;/td&gt;
&lt;td&gt;300.0&lt;/td&gt;
&lt;td&gt;❌ violates&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Regression watchlist:&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;Baseline&lt;/th&gt;
&lt;th&gt;Patched&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p50 latency (ms)&lt;/td&gt;
&lt;td&gt;529.8&lt;/td&gt;
&lt;td&gt;0.9&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;−99.8%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Error rate&lt;/td&gt;
&lt;td&gt;0.00%&lt;/td&gt;
&lt;td&gt;0.00%&lt;/td&gt;
&lt;td&gt;no change&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Verdict: REJECT.&lt;/strong&gt; Alert would still be firing post-merge. SLO &lt;code&gt;p99_ms &amp;gt; 300.0&lt;/code&gt; (baseline 2162, patched 2010).&lt;/p&gt;

&lt;p&gt;p50 dropped 99.8% (cache hits return instantly), but the 99th-percentile request is still a cache miss hitting the 1.5s tail. The alert-aware classifier says the thing the on-call needs to hear: &lt;em&gt;the alert that fired would still fire on this code; SLO threshold 300ms; patched value 2010ms.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The fix HolmesGPT didn't propose: a short client-side timeout on &lt;code&gt;fetch_price()&lt;/code&gt; to fail-fast under the SLO instead of waiting out the 1.5s tail. We re-ran the bridge with a hand-crafted patch doing that; verifier returned PASS at patched p99 ~330ms. The point isn't that HolmesGPT missed the right fix, it's that the verifier surfaced the gap before the wrong patch shipped.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the two scenarios show
&lt;/h2&gt;

&lt;p&gt;The loop closes against a real OSS AI-SRE you can install yourself. HolmesGPT investigates, the bridge turns its report into a patch, mirrord runs it against the live cluster, the classifier returns a verdict tied to the actual alert. No integration code from us.&lt;/p&gt;

&lt;p&gt;PASS and REJECT both have to be useful or the verifier becomes decorative. Scenario 1 signs off on a real fix. Scenario 2 stops one that &lt;em&gt;looks&lt;/em&gt; good (p50 −99.8%) but doesn't clear the alert. Without both, on-call learns to ignore the verifier and starts merging on faith.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run it yourself
&lt;/h2&gt;

&lt;p&gt;The sample &lt;code&gt;checkout&lt;/code&gt;/&lt;code&gt;pricing&lt;/code&gt; services, Prometheus rules, verifier code, and bridge script are small (~500 lines of Python plus the ~80-line bridge). The code lives at &lt;a href="https://github.com/metalbear-co/holmes-mirrord-verifier" rel="noopener noreferrer"&gt;github.com/metalbear-co/holmes-mirrord-verifier&lt;/a&gt;. You'll need the &lt;a href="https://metalbear.com/mirrord/docs/operator/setup/installation" rel="noopener noreferrer"&gt;mirrord operator&lt;/a&gt; installed in your cluster.&lt;/p&gt;




&lt;p&gt;If you haven't read &lt;a href="https://metalbear.com/blog/ai-sre/" rel="noopener noreferrer"&gt;Part I&lt;/a&gt; yet, it has the recipe: the pattern, the verification code, and how to wire it into the AI-SRE you're already running.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;mirrord: &lt;a href="https://metalbear.com/mirrord" rel="noopener noreferrer"&gt;https://metalbear.com/mirrord&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Questions: &lt;a href="mailto:hi@metalbear.com"&gt;hi@metalbear.com&lt;/a&gt; · &lt;a href="https://metalbear.com/slack" rel="noopener noreferrer"&gt;https://metalbear.com/slack&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>sre</category>
      <category>kubernetes</category>
      <category>devops</category>
    </item>
    <item>
      <title>Auto-verifying your AI-SRE's fixes against your real cluster, with mirrord</title>
      <dc:creator>Eyal Bukchin</dc:creator>
      <pubDate>Thu, 11 Jun 2026 11:02:50 +0000</pubDate>
      <link>https://dev.to/metalbear/auto-verifying-your-ai-sres-fixes-against-your-real-cluster-with-mirrord-2p16</link>
      <guid>https://dev.to/metalbear/auto-verifying-your-ai-sres-fixes-against-your-real-cluster-with-mirrord-2p16</guid>
      <description>&lt;p&gt;&lt;em&gt;Part I of two. Part II will be published next week, and will show the same loop end-to-end against HolmesGPT on a real cluster.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You're rolling out an AI-SRE (your own homegrown in-house AI loop, or one of the new products: Resolve AI, incident.io's Investigator, Datadog Bits). When an alert fires, it proposes a fix: a timeout here, a retry there, a config bump. The suggestion looks plausible. Do you merge it?&lt;/p&gt;

&lt;p&gt;Today, you read the diff, eyeball it against your mental model of the service, and decide, or you deploy to staging and wait. The bugs an AI-SRE gets paged on (an unhandled exception under a rare request shape, a downstream that times out for some requests, a webhook that races with itself, a slow query that only surfaces at scale) live in the gap between local and production. Unit tests and mocks don't reach there. Staging does, but staging is slow (a deploy on every iteration) and shared (you're competing with everyone else trying to reproduce their own bugs).&lt;/p&gt;

&lt;p&gt;This is where mirrord can help. mirrord runs a separate copy of your service as if it were a pod in your cluster, wired into the same real downstreams and upstreams without a deploy. This guide shows how to wire that into your AI-SRE so every suggested fix is verified against the real cluster before it reaches a human, automatically, in no time, with no SDK or code to import.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's an AI-SRE?
&lt;/h2&gt;

&lt;p&gt;A new category of tools that take the first investigative pass when an alert fires, before the on-call human is even awake. The AI-SRE pulls together the context a human would have spent the first ten minutes assembling (recent deploys, related logs, metric anomalies, runbooks, similar past incidents) and proposes a remediation. Most just produce suggestions; a human is still the one acting on them. A few are starting to auto-execute low-risk actions (pod restarts, config flag flips, even small auto-merged code patches).&lt;/p&gt;

&lt;p&gt;AI-SREs come in three flavors: standalone (Resolve AI, NeuBird, Cleric), AI bolted onto existing incident or observability tools (incident.io's Investigator, Datadog Bits, Rootly's Copilot, etc.), and homegrown LLM loops, which are the most common shape.&lt;/p&gt;

&lt;p&gt;Below are some examples of what an AI-SRE actually outputs. Every one of these is a fix that can't be verified without real cluster state, which is why this guide exists:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Alert&lt;/th&gt;
&lt;th&gt;AI-SRE's suggestion&lt;/th&gt;
&lt;th&gt;What verification has to prove&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Spike of &lt;code&gt;KeyError: 'address'&lt;/code&gt; in checkout handler&lt;/td&gt;
&lt;td&gt;Default the field to &lt;code&gt;None&lt;/code&gt; and guard the deref&lt;/td&gt;
&lt;td&gt;The previously-failing requests now return 2xx, no new errors introduced&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Checkout p99 latency above SLO&lt;/td&gt;
&lt;td&gt;Add &lt;code&gt;timeout=300ms&lt;/code&gt; to the pricing-service call&lt;/td&gt;
&lt;td&gt;The slow outliers get cut off; p99 actually drops&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5xx burst after deploy&lt;/td&gt;
&lt;td&gt;Roll back commit &lt;code&gt;abc123&lt;/code&gt;, or guard the new code path with a feature flag&lt;/td&gt;
&lt;td&gt;The error rate stops climbing on real production traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Webhook handler creating duplicate orders&lt;/td&gt;
&lt;td&gt;Check idempotency key in Redis before insert&lt;/td&gt;
&lt;td&gt;A retried webhook produces one row, not two&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kafka consumer lag growing&lt;/td&gt;
&lt;td&gt;Bump &lt;code&gt;max.poll.records&lt;/code&gt;, add a backoff on a flaky downstream&lt;/td&gt;
&lt;td&gt;Lag actually drains, downstream stays healthy&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;None of these can be verified with unit tests or mocks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Meet &lt;code&gt;mirrord exec&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;mirrord exec&lt;/code&gt; runs a local process as if it were a pod in your cluster. Its outbound calls resolve through the target pod's network context, so when your patched code calls &lt;code&gt;payments&lt;/code&gt; or &lt;code&gt;inventory&lt;/code&gt; or your database, it hits the real in-cluster service. mirrord also routes incoming traffic to the local process when you ask it to. Environment variables, mounted ConfigMaps and Secrets, and the target's filesystem are inherited by default. This is what allows your AI-SRE to test the code fix it came up with instantly and without having to deploy to staging or production. The deployed pod is never touched.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where mirrord fits
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F4iv5h65h2ng7mrhg2s7z.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F4iv5h65h2ng7mrhg2s7z.png" alt="AI-SRE verification loop" width="800" height="1014"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What your AI-SRE already owns
&lt;/h2&gt;

&lt;p&gt;This guide assumes three things about your AI-SRE: it (a) ingests an alert, (b) produces a candidate fix (a code diff, or a recommendation that needs a small bridge step to actually turn it into code, covered in Part II), and (c) has &lt;em&gt;some&lt;/em&gt; check for whether the bug is fixed (a repro script, a load test, the alerting query, or a known-good test case).&lt;/p&gt;

&lt;p&gt;mirrord doesn't replace any of these. It runs your patched service under &lt;code&gt;mirrord exec&lt;/code&gt;, so (c) can hit the real cluster, without a deploy.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to set it up
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Prerequisites
&lt;/h3&gt;

&lt;p&gt;Two things to install:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The mirrord &lt;strong&gt;operator&lt;/strong&gt; in your cluster (&lt;a href="https://metalbear.com/mirrord/docs/operator/setup/installation" rel="noopener noreferrer"&gt;install docs&lt;/a&gt;), one helm command.&lt;/li&gt;
&lt;li&gt;The mirrord &lt;strong&gt;CLI&lt;/strong&gt; wherever the AI-SRE will run the verification step: typically the AI-SRE's own worker, or a sidecar container alongside it (&lt;a href="https://metalbear.com/mirrord/docs/getting-started/quick-start" rel="noopener noreferrer"&gt;install docs&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Automated verification with &lt;code&gt;mirrord exec&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;When the AI-SRE produces a candidate fix, run a short verification block from inside its own hook:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Check out a working copy of the service code (&lt;code&gt;git clone&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Apply the AI-SRE's patch to it (&lt;code&gt;git apply patch.diff&lt;/code&gt;, or whatever your AI-SRE delivers)&lt;/li&gt;
&lt;li&gt;Run that patched copy under &lt;code&gt;mirrord exec&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;mirrord makes the local process behave as if it were the target pod (same network identity, same downstreams, same env), but the deployed service is untouched.&lt;/p&gt;

&lt;p&gt;Run it twice, once with the patch, once without. Both runs see the same live downstreams and upstreams, so the only variable is the patch.&lt;/p&gt;

&lt;p&gt;You'll want mirrord in &lt;strong&gt;steal mode with a header filter&lt;/strong&gt;, so the verifier diverts only its own tagged requests to the candidate and leaves production traffic alone. The mirrord config (write once, reuse per run):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="err"&gt;//&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;.mirrord/verifier.json&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"feature"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"network"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"incoming"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"steal"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"http_filter"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
          &lt;/span&gt;&lt;span class="nl"&gt;"header_filter"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"^X-Verifier-Run: {{ key }}$"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
          &lt;/span&gt;&lt;span class="nl"&gt;"ports"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;8080&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;{{ key }}&lt;/code&gt; is a mirrord template variable that resolves to whatever you pass via &lt;code&gt;mirrord exec --key&lt;/code&gt;, so each verification run can tag itself uniquely without templating the config file.&lt;/p&gt;

&lt;p&gt;If your check is a &lt;strong&gt;repro script&lt;/strong&gt; that exits non-zero when the bug is present:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;RUN_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;uuidgen&lt;span class="si"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# Start the candidate; mirrord will steal only requests tagged with $RUN_ID.&lt;/span&gt;
mirrord &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nt"&gt;--config-file&lt;/span&gt; .mirrord/verifier.json &lt;span class="nt"&gt;--key&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$RUN_ID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--target&lt;/span&gt; deploy/&amp;lt;your-service&amp;gt; &lt;span class="nt"&gt;--&lt;/span&gt; ./service/run-server.sh &amp;amp;

&lt;span class="c"&gt;# Drive the repro at the real service URL, tagged with the header.&lt;/span&gt;
&lt;span class="c"&gt;# Other traffic to &amp;lt;your-service&amp;gt; keeps hitting the deployed pods.&lt;/span&gt;
./repro.sh https://&amp;lt;your-service&amp;gt;.svc.cluster.local &lt;span class="nt"&gt;--header&lt;/span&gt; &lt;span class="s2"&gt;"X-Verifier-Run: &lt;/span&gt;&lt;span class="nv"&gt;$RUN_ID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If your check is &lt;strong&gt;metric-based&lt;/strong&gt; (latency or error rate, the common case for AI-SRE alerts), same shape, swap the repro for a load test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;RUN_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;uuidgen&lt;span class="si"&gt;)&lt;/span&gt;

mirrord &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nt"&gt;--config-file&lt;/span&gt; .mirrord/verifier.json &lt;span class="nt"&gt;--key&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$RUN_ID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--target&lt;/span&gt; deploy/&amp;lt;your-service&amp;gt; &lt;span class="nt"&gt;--&lt;/span&gt; ./service/run-server.sh &amp;amp;

./loadtest.sh https://&amp;lt;your-service&amp;gt;.svc.cluster.local &lt;span class="nt"&gt;--header&lt;/span&gt; &lt;span class="s2"&gt;"X-Verifier-Run: &lt;/span&gt;&lt;span class="nv"&gt;$RUN_ID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; patched-metrics.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run that block once on the unpatched code and once on the patched one, diff &lt;code&gt;baseline-metrics.json&lt;/code&gt; vs &lt;code&gt;patched-metrics.json&lt;/code&gt;, and you have a verdict. The comparison is shaped by the alert: did the SLO condition the alert fires on still hold on the patched run? Plus a sanity check that the other signals didn't regress past tolerance. For an &lt;code&gt;error_rate &amp;gt; 5%&lt;/code&gt; alert, check the patched error rate is now under 5%, and that p50/p99 didn't grow past your tolerance. For a &lt;code&gt;p99 &amp;gt; 300ms&lt;/code&gt; alert, check p99 is now under 300ms, and that error rate didn't climb. (If your AI-SRE outputs a written report rather than ready-to-apply diffs, you'll also want a small bridge step; Part II walks through one.)&lt;/p&gt;

&lt;p&gt;An immediate verdict, against your real cluster. The verifier can also &lt;em&gt;reject&lt;/em&gt;. Sometimes the AI's fix moves the metric the alert fires on without moving it enough to clear the SLO, or improves one signal and regresses another. You want to find that out here, not in production.&lt;/p&gt;

&lt;h3&gt;
  
  
  Wiring it into your AI-SRE
&lt;/h3&gt;

&lt;p&gt;The integration is light. Most AI-SREs already run a hook when they finalize a suggestion (the part that posts the PR comment or the Slack message). In that hook:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Apply the generated patch to a checked-out copy of the service.&lt;/li&gt;
&lt;li&gt;Run the verification block above. If it doesn't pass, post the comparison back to the AI-SRE so it can try again (per the diagram). The on-call human keeps working the original alert; they just don't review a fix that didn't survive verification.&lt;/li&gt;
&lt;li&gt;If it passes, post the verdict and the diff. A human (or your auto-merge policy) takes it from there.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;One caveat on the REJECT-and-retry loop. Today's vendor AI-SREs (HolmesGPT, Resolve AI, incident.io's Investigator) are single-shot. To actually close the loop, put the retry in the small wrapper that turns the AI-SRE's recommendation into a code patch (covered in Part II): feed it the failed run's verdict and ask for another approach. Homegrown AI-SREs can put the retry wherever.&lt;/p&gt;

&lt;p&gt;In a noisy incident where your AI proposes ten patches, you spend ten quick verification runs and only the survivors reach a human.&lt;/p&gt;

&lt;h3&gt;
  
  
  Bonus: hand humans a live preview (mirrord Enterprise)
&lt;/h3&gt;

&lt;p&gt;When verification passes and a human still wants to click around before merging, mirrord preview environments (an Enterprise-tier operator feature) give them an isolated pod from the patched image, reachable at the same service URL with a session header.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mirrord preview start &lt;span class="nt"&gt;--key&lt;/span&gt; fix-&lt;span class="nv"&gt;$INCIDENT_ID&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--image&lt;/span&gt; registry.example.com/&amp;lt;your-service&amp;gt;:fix-&lt;span class="nv"&gt;$INCIDENT_ID&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--target&lt;/span&gt; deployment/&amp;lt;your-service&amp;gt; &lt;span class="nt"&gt;--ttl&lt;/span&gt; 30

curl &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s1"&gt;'baggage: mirrord-session=fix-$INCIDENT_ID'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  https://&amp;lt;your-service&amp;gt;.prod.svc/...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Other traffic keeps hitting the unpatched pods; the header is what routes you to the candidate. After the TTL the operator tears it down. See the &lt;a href="https://metalbear.com/mirrord/docs/use-cases/preview-environments" rel="noopener noreferrer"&gt;preview environments docs&lt;/a&gt; for details.&lt;/p&gt;




&lt;p&gt;That's the recipe. &lt;strong&gt;Part II&lt;/strong&gt; (coming next week) shows the loop running end-to-end against HolmesGPT, an OSS AI-SRE that ships standalone. Two planted bugs in a real GKE cluster: one the loop signs off on, one the loop correctly rejects.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why not just stage a copy of the service?
&lt;/h2&gt;

&lt;p&gt;You could run a copy of &lt;code&gt;checkout&lt;/code&gt; in a separate namespace, point it at the real &lt;code&gt;pricing&lt;/code&gt;, and call that verification. The reasons &lt;code&gt;mirrord exec&lt;/code&gt; is a better fit specifically for &lt;em&gt;AI-SRE auto-verification&lt;/em&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Identity matches prod.&lt;/strong&gt; &lt;code&gt;mirrord exec&lt;/code&gt; inherits the target pod's ServiceAccount, OIDC token, and service-mesh identity. A staged copy gets its own. The bugs AI-SREs handle often involve auth, mTLS, or NetworkPolicy interactions where identity is the variable; if your verification runs as a different principal than prod, you can pass on a fix that prod would still reject.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Env, secrets, mounts inherited automatically.&lt;/strong&gt; mirrord pulls the target pod's environment variables, mounted ConfigMaps and Secrets, and filesystem. Staged copies need a parallel set kept in sync; drift = false verdict. AI-SRE patches frequently touch env-shaped behavior (a missing flag, a stale config), which is exactly the drift class you want verification to catch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Seconds, not minutes.&lt;/strong&gt; &lt;code&gt;mirrord exec&lt;/code&gt; runs from a checked-out working copy in a few seconds. The staged-namespace approach is build image, push, &lt;code&gt;kubectl apply&lt;/code&gt;, wait-ready, port-forward, several minutes per try. In a noisy incident where the AI tries ten patches, that's the difference between "all ten verified in under five minutes" and "one verification per coffee."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Parallel runs are free.&lt;/strong&gt; Each &lt;code&gt;mirrord exec&lt;/code&gt; is its own process. Ten candidate patches can verify concurrently. The staged-copy approach shares a Deployment, Service, and Ingress, so concurrent runs collide unless you also templatize per-run namespaces.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;Setup against any cluster you can &lt;code&gt;kubectl&lt;/code&gt; into:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# install the CLI&lt;/span&gt;
brew &lt;span class="nb"&gt;install &lt;/span&gt;metalbear-co/mirrord/mirrord

&lt;span class="c"&gt;# install the operator in your cluster (Prerequisites above)&lt;/span&gt;

&lt;span class="c"&gt;# pick any deployment and confirm the loop works&lt;/span&gt;
mirrord &lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="nt"&gt;--target&lt;/span&gt; deploy/&amp;lt;your-service&amp;gt; &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="nb"&gt;printenv&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-10&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If that prints env vars from the target pod (not your local shell's), the verification primitive is live. Wire your AI-SRE's post-suggestion hook to the pattern in &lt;em&gt;Automated verification&lt;/em&gt; above and let the next alert prove itself. Then read Part II (coming next week) for the full walkthrough against HolmesGPT.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;mirrord: &lt;a href="https://metalbear.com/mirrord" rel="noopener noreferrer"&gt;https://metalbear.com/mirrord&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Questions: &lt;a href="mailto:hi@metalbear.com"&gt;hi@metalbear.com&lt;/a&gt; · &lt;a href="https://metalbear.com/slack" rel="noopener noreferrer"&gt;https://metalbear.com/slack&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>sre</category>
      <category>kubernetes</category>
      <category>devops</category>
    </item>
    <item>
      <title>A Beginner's Guide to Helm</title>
      <dc:creator>Arsh Sharma</dc:creator>
      <pubDate>Wed, 03 Jun 2026 12:25:24 +0000</pubDate>
      <link>https://dev.to/metalbear/a-beginners-guide-to-helm-5fj0</link>
      <guid>https://dev.to/metalbear/a-beginners-guide-to-helm-5fj0</guid>
      <description>&lt;p&gt;The &lt;a href="https://www.cncf.io/wp-content/uploads/2026/01/CNCF_Annual_Survey_Report_final.pdf" rel="noopener noreferrer"&gt;CNCF Annual Survey&lt;/a&gt; published in January 2026 showed that Helm is the second most used CNCF project, second only to Kubernetes itself. The reason is simple: once you start working with Kubernetes, YAML files accumulate fast and managing them keeps getting harder. A single microservice can require a Deployment, a Service, a ConfigMap, and a Secret, and that's before accounting for the different sets you’ll need for your development, staging, and production environments. Helm fixes this by packaging all of your manifests into a single, versioned unit that can be installed, upgraded, and rolled back easily.&lt;/p&gt;

&lt;p&gt;In this guide, we'll cover what Helm is, some key concepts, and the main Helm commands you need to know to deploy and manage manifests using it. You’ll walk away with a complete understanding of how to use Helm so you can simplify the YAML mess you might be in.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Prerequisite: &lt;a href="https://metalbear.com/blog/kubernetes-guide/" rel="noopener noreferrer"&gt;Basic familiarity with Kubernetes&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What is Helm in Kubernetes?
&lt;/h2&gt;

&lt;p&gt;Helm is a package manager for Kubernetes. Just as &lt;code&gt;apt&lt;/code&gt; manages software packages on Linux or &lt;code&gt;brew&lt;/code&gt; does on macOS, Helm manages Kubernetes applications and lets you install, upgrade, and roll them back as versioned, reusable units called charts.&lt;/p&gt;

&lt;p&gt;The way Helm solves the YAML management problem is easiest to understand through a programming analogy. Bare Kubernetes manifests work fine for a single, static deployment. But the moment you need multiple similar deployments (dev, staging, prod, which may differ only by a few values), you're essentially copy-pasting. Helm solves this the same way classes solve it in &lt;a href="https://developer.mozilla.org/en-US/docs/Learn_web_development/Extensions/Advanced_JavaScript_objects/Object-oriented_programming" rel="noopener noreferrer"&gt;object-oriented programming:&lt;/a&gt; you define the boilerplate once in a chart, which can include things like the Deployment template, the Service definition, the ConfigMap structure, and then each Helm chart deployment only specifies what's different: the image, the environment variables, the command being run in the pods. Someone installing your chart can also supply their own values file and get a correctly configured deployment for their use case without touching the original templates.&lt;/p&gt;

&lt;p&gt;Like Kubernetes, Helm is declarative. You describe which chart to install and what values to pass in and Helm figures out how to get the cluster there and keeps it in that state.&lt;/p&gt;

&lt;p&gt;Helm is built around three core concepts: Charts, Releases, and Repositories. We'll look at each of these next.&lt;/p&gt;

&lt;h2&gt;
  
  
  Key Concepts
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Charts
&lt;/h3&gt;

&lt;p&gt;A chart is a self-contained package of Kubernetes resource manifests. It defines everything needed to deploy an application (the Deployment, the Service, the ConfigMap, etc.) along with default values for any configurable settings. When you install a chart, Helm uses those templates and values to generate the actual Kubernetes manifests and applies them to your cluster.&lt;/p&gt;

&lt;p&gt;Charts can also declare dependencies on other charts, called subcharts. This makes it possible to build composable application definitions, for example, a chart for a web application might pull in a chart for PostgreSQL and another for Redis rather than having manifests for those dependencies within it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Releases
&lt;/h3&gt;

&lt;p&gt;Each time you install a chart, Helm creates a release which is a named instance of that chart running in your cluster. If you install the same chart three times under different names, you get three independent releases, each with its own configuration, its own Kubernetes resources, and its own revision history.&lt;/p&gt;

&lt;p&gt;It's important to understand that releases are what Helm actually manages. Helm doesn't think in terms of individual deployments or Helm charts when it comes to a Kubernetes cluster. Instead, it tracks releases and uses them to know what's deployed, what version it's at, and how to upgrade or roll back.&lt;/p&gt;

&lt;h3&gt;
  
  
  Repositories
&lt;/h3&gt;

&lt;p&gt;A repository is a collection of charts hosted at an accessible URL. You add repositories to your local Helm installation and then install charts from them by name. The most widely used public index is &lt;a href="https://artifacthub.io" rel="noopener noreferrer"&gt;Artifact Hub&lt;/a&gt;, which aggregates charts from many sources. Most projects that run on Kubernetes publish an official Helm chart there, so in practice you'll rarely need to write a chart from scratch to deploy well-known software like nginx, PostgreSQL, or Prometheus. For example, we also publish a chart for the mirrord Kubernetes operator, which you can find on &lt;a href="https://artifacthub.io/packages/helm/metalbear/mirrord-operator" rel="noopener noreferrer"&gt;Artifact Hub here&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's Inside a Helm Chart
&lt;/h2&gt;

&lt;p&gt;When you clone or create a Helm chart (using &lt;code&gt;helm create chart-name&lt;/code&gt;), you get a directory with a specific structure. Here's what a typical chart looks like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;my-chart/
├── Chart.yaml
├── values.yaml
├── charts/
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ...
    └── NOTES.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;Chart.yaml&lt;/code&gt; is the chart's metadata file. It declares the chart's name, version, and description, and optionally lists dependencies on other charts.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;v2&lt;/span&gt;
&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;my-chart&lt;/span&gt;
&lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;A Helm chart for my application&lt;/span&gt;
&lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;0.1.0&lt;/span&gt;
&lt;span class="na"&gt;appVersion&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1.0.0"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;apiVersion&lt;/code&gt; here refers to the Helm chart API version, not the Kubernetes API. &lt;code&gt;v2&lt;/code&gt; means the chart requires at least Helm 3 (don’t ask me why they couldn’t have made it simple by just having v3 for Helm 3). If you come across a chart with &lt;code&gt;v1&lt;/code&gt;, it was written for Helm 2, which had a different chart format but Helm 3 and later versions still support v1 for backward compatibility.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;version&lt;/code&gt; is the version of the chart itself. This follows &lt;a href="https://semver.org/" rel="noopener noreferrer"&gt;semantic versioning&lt;/a&gt; and should be bumped whenever you change the chart by doing things like adding a new template, changing a default value, or updating dependencies. It's independent of what the chart deploys.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;appVersion&lt;/code&gt; is the version of the application the chart deploys. This is purely for documentation and nothing else since Helm doesn't use it to pull images or make any decisions. You'd typically set it to the container image tag your chart defaults to, so users can tell at a glance what version of the application they're getting. You can update &lt;code&gt;appVersion&lt;/code&gt; without changing &lt;code&gt;version&lt;/code&gt;, and vice versa.&lt;/p&gt;

&lt;p&gt;A good way to think about it is that &lt;code&gt;appVersion&lt;/code&gt; tracks the version of the application being deployed, while &lt;code&gt;version&lt;/code&gt; tracks releases of the chart itself. If your application is upgraded from 1.0.0 to 1.1.0, you'd update &lt;code&gt;appVersion&lt;/code&gt; to reflect that change. If the chart's default image tag is also updated, you'd typically bump the chart &lt;code&gt;version&lt;/code&gt; as well, since the chart itself has changed. But if you only modify the chart (for example by adding a new template, changing default values, or updating dependencies) while continuing to deploy the same application version, you'd bump &lt;code&gt;version&lt;/code&gt; and leave &lt;code&gt;appVersion&lt;/code&gt; unchanged.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;values.yaml&lt;/code&gt; contains the default configuration for the chart. These are the values users can override at install time. For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;repository&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;nginx&lt;/span&gt;
  &lt;span class="na"&gt;tag&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;1.25"&lt;/span&gt;
&lt;span class="na"&gt;replicaCount&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;templates/&lt;/code&gt; is where the Kubernetes manifests live. Unlike regular YAML, these files use &lt;a href="https://docs.gomplate.ca/syntax/" rel="noopener noreferrer"&gt;Go template syntax&lt;/a&gt;, which lets Helm inject values into them at render time. A template referencing the values above would look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;replicas&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{{&lt;/span&gt; &lt;span class="nv"&gt;.Values.replicaCount&lt;/span&gt; &lt;span class="pi"&gt;}}&lt;/span&gt;
  &lt;span class="na"&gt;template&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;containers&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;app&lt;/span&gt;
          &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;{{&lt;/span&gt; &lt;span class="nv"&gt;.Values.image.repository&lt;/span&gt; &lt;span class="pi"&gt;}}&lt;/span&gt;&lt;span class="s"&gt;:{{ .Values.image.tag }}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When someone installs this chart and passes &lt;code&gt;--set image.tag=1.26&lt;/code&gt;, Helm substitutes that value before applying the manifest to the cluster. This is the mechanism that makes a single chart work across environments without touching the templates themselves.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;charts/&lt;/code&gt; is where subchart dependencies are bundled. If your chart depends on PostgreSQL, for example, the PostgreSQL chart will live in this folder.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;NOTES.txt&lt;/code&gt; is a template file that Helm renders and prints to the terminal after a successful install. Charts typically use it to show how to connect to or use what was just deployed. This is what a simple &lt;code&gt;NOTES.txt&lt;/code&gt; could look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Thanks for installing {{ .Chart.Name }}.

To get more info, try:

  $ helm status {{ .Release.Name }}
  $ helm get all {{ .Release.Name }}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Creating Your Own Chart
&lt;/h3&gt;

&lt;p&gt;Running &lt;code&gt;helm create my-chart&lt;/code&gt; scaffolds the directory structure we just walked through, with sensible defaults already filled in. The generated &lt;code&gt;values.yaml&lt;/code&gt; comes pre-populated with common settings like &lt;code&gt;replicaCount&lt;/code&gt;, &lt;code&gt;image.repository&lt;/code&gt;, and &lt;code&gt;image.tag&lt;/code&gt;, and &lt;code&gt;templates/&lt;/code&gt; includes a Deployment, Service, Ingress, and a few other Kubernetes resources you might need.&lt;/p&gt;

&lt;p&gt;The first thing to do after generating the chart is update &lt;code&gt;values.yaml&lt;/code&gt; to reflect your application. Change the &lt;code&gt;image.repository&lt;/code&gt; to point to your container image, set a default &lt;code&gt;tag&lt;/code&gt;, and remove any values that don't apply. The generated defaults are based on a generic nginx deployment, so most of them will need adjusting.&lt;/p&gt;

&lt;p&gt;The templates themselves often need less work. The generated Deployment and Service are wired up to the values in &lt;code&gt;values.yaml&lt;/code&gt; using the &lt;code&gt;{{ .Values.* }}&lt;/code&gt; syntax we saw earlier, so once your values are correct, the templates usually render correctly without modification. If your application has specific Kubernetes resources it needs that aren’t present or configured in the &lt;code&gt;templates/&lt;/code&gt; directory already, then you'd add those or modify the existing relevant template.&lt;/p&gt;

&lt;p&gt;Once you've made your changes, run &lt;code&gt;helm lint my-chart&lt;/code&gt; to catch mistakes like invalid YAML, missing required fields, broken template syntax. Then run &lt;code&gt;helm template my-chart&lt;/code&gt; to render the templates locally and review the exact Kubernetes manifests that would be applied to the cluster. This is the step where you verify your values are flowing into the right places before anything touches the cluster.&lt;/p&gt;

&lt;h2&gt;
  
  
  Common Helm Commands
&lt;/h2&gt;

&lt;p&gt;Let's now walk through some common commands you’ll find yourself using when working with Helm. Before you can install a chart, you need to add the repository it lives in. &lt;code&gt;helm repo add&lt;/code&gt; registers a repository under a local name:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm repo add metalbear https://metalbear-co.github.io/charts
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once added, run &lt;code&gt;helm repo update&lt;/code&gt; to pull the latest chart index from all registered repositories. Charts are versioned and updated frequently, so it's worth running this before installing. To find a chart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm search repo mirrord-operator
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This returns matching charts with their versions and descriptions. You can also browse &lt;a href="https://artifacthub.io" rel="noopener noreferrer"&gt;Artifact Hub&lt;/a&gt; to discover charts. Each listing there shows the &lt;code&gt;helm repo add&lt;/code&gt; command you need.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;helm install&lt;/code&gt; deploys a chart as a named release:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm &lt;span class="nb"&gt;install &lt;/span&gt;mirrord-operator metalbear/mirrord-operator
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Helm renders the chart's templates, applies the resulting manifests to the cluster, and begins tracking the deployment as a release named &lt;code&gt;mirrord-operator&lt;/code&gt;. It also prints the chart's &lt;code&gt;NOTES.txt&lt;/code&gt; which typically has instructions for what to do after installing the chart.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Note: The chart used in the example above installs the &lt;a href="https://metalbear.com/mirrord/docs/getting-started/installing-mirrord/operator" rel="noopener noreferrer"&gt;mirrord operator&lt;/a&gt; which allows multiple developers to develop against the same cluster when using &lt;a href="https://metalbear.com/mirrord/" rel="noopener noreferrer"&gt;mirrord&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;To see all releases currently running in the cluster:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm list
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To inspect a specific release and see its status and revision number:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm status mirrord-operator
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Customizing with Values
&lt;/h3&gt;

&lt;p&gt;By default, &lt;code&gt;helm install&lt;/code&gt; uses the values defined in the chart's &lt;code&gt;values.yaml&lt;/code&gt;. You can override them in two ways. For quick changes, use &lt;code&gt;--set&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm &lt;span class="nb"&gt;install &lt;/span&gt;mirrord-operator metalbear/mirrord-operator &lt;span class="nt"&gt;--set&lt;/span&gt; license.key&lt;span class="o"&gt;=&lt;/span&gt;ABC-DEF-GHI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For environment-specific configuration, a values file is easier to manage. Create a &lt;code&gt;custom-values.yaml&lt;/code&gt; with only the values you want to override, then pass it with &lt;code&gt;-f&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm &lt;span class="nb"&gt;install &lt;/span&gt;mirrord-operator metalbear/mirrord-operator &lt;span class="nt"&gt;-f&lt;/span&gt; values.yaml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When both are used together, &lt;code&gt;--set&lt;/code&gt; takes precedence over &lt;code&gt;-f&lt;/code&gt;, which takes precedence over the chart's &lt;code&gt;values.yaml&lt;/code&gt; defaults. To see all the values a chart allows you to configure you can run this command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm show values metalbear/mirrord-operator
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When you want to update a release to a newer chart version and/or change its values use &lt;code&gt;helm upgrade&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm upgrade mirrord-operator metalbear/mirrord-operator &lt;span class="nt"&gt;--set&lt;/span&gt; limits.cpu&lt;span class="o"&gt;=&lt;/span&gt;400m
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Helm applies only what changed, creates a new revision, and preserves the previous one. To see the full revision history:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm &lt;span class="nb"&gt;history &lt;/span&gt;mirrord-operator
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If an upgrade causes problems, &lt;code&gt;helm rollback&lt;/code&gt; reverts a release to an earlier revision:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm rollback mirrord-operator 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's important to understand that a rollback creates a new revision rather than deleting the failed one which means the history stays intact so you can always see what happened.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;helm uninstall&lt;/code&gt; removes all Kubernetes resources that belong to a release:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm uninstall mirrord-operator
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;By default, Helm also deletes the release's history. To keep it for auditing or future rollbacks, pass &lt;code&gt;--keep-history&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Frequently Asked Helm Questions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Do I have to use Helm with Kubernetes?
&lt;/h3&gt;

&lt;p&gt;You don't strictly need Helm. For a simple application that doesn't change much, managing it with raw &lt;code&gt;kubectl apply&lt;/code&gt; and plain YAML manifests works fine. Where Helm becomes worth it is when you're managing multiple environments, deploying the same application more than once, or distributing your application for others to run. Then maintaining separate YAML files per environment costs more in terms of time and effort than just learning Helm.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do I deploy a Helm Chart on Kubernetes?
&lt;/h3&gt;

&lt;p&gt;First, add the repository the chart lives in and update your local index. Then install the chart with &lt;code&gt;helm install&lt;/code&gt;, giving the release a name and pointing to the chart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;helm repo add metalbear https://metalbear-co.github.io/charts
helm repo update
helm &lt;span class="nb"&gt;install &lt;/span&gt;mirrord-operator metalbear/mirrord-operator
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the chart requires custom configuration, pass values with &lt;code&gt;--set&lt;/code&gt; for quick overrides or &lt;code&gt;-f your-values.yaml&lt;/code&gt; for a full config file. After installing, &lt;code&gt;helm list&lt;/code&gt; shows all running releases and &lt;code&gt;helm status &amp;lt;release-name&amp;gt;&lt;/code&gt; gives details on a specific one. The Common Helm Commands section above covers all of this in detail.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next Steps
&lt;/h2&gt;

&lt;p&gt;In this guide, we covered the fundamentals of Helm: charts, releases, and repositories, how chart structure works, and the commands you'll use day to day. These basics are enough to install and manage existing charts and start writing your own charts. If you want to go deeper from here, working toward a Helm certification like &lt;a href="https://training.linuxfoundation.org/skillcred/helm/" rel="noopener noreferrer"&gt;Developing Helm Charts (SC104)&lt;/a&gt; can be a good way to continue learning Helm. For this and other topics the &lt;a href="https://helm.sh/docs/" rel="noopener noreferrer"&gt;official Helm documentation&lt;/a&gt; is the best place to go.&lt;/p&gt;

&lt;p&gt;Or if you don’t want to prepare for a certification but want to learn how to make managing multiple charts across multiple environments easier, &lt;a href="https://helmfile.readthedocs.io/" rel="noopener noreferrer"&gt;Helmfile&lt;/a&gt; is the natural next tool to look at. It lets you declaratively define a set of Helm releases as a single unit, with per-environment values and deployment ordering.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>containers</category>
      <category>beginners</category>
    </item>
  </channel>
</rss>
