DEV Community: Arsh Sharma

Best Microservices Testing Tools for Kubernetes

Arsh Sharma — Mon, 02 Nov 2026 12:31:24 +0000

TLDR: There is no single best tool for microservices testing. You need a stack that covers several layers: a test environment (mirrord, Signadot, Telepresence, Okteto, Tilt), test orchestration (Testkube), API testing (Postman), service mocking (WireMock), contract testing (Pact) and load testing (k6). The environment layer matters most, because it decides whether every other tool's results can be trusted or not.

When teams ask us about microservices testing, the conversation usually starts with tools, when it should start with environments. Testing a monolith is mostly a question of writing good tests. Testing microservices on Kubernetes is a question of where those tests run, because a microservice can't truly be verified without the services, databases and queues it talks to. Every common workaround has a cost:

Mocks drift away from how the real services behave.
A shared staging environment turns into a queue where one broken deploy blocks everyone.
Spinning up a full ephemeral environment for every pull request is slow and gets expensive quickly.

And with 59% of organizations reporting that much or nearly all of their development and deployment is now cloud native (CNCF Annual Cloud Native Survey 2025), this is a problem most engineering teams now have, not a niche one.

It hits platform and DevOps engineers who own the clusters, backend developers who need fast feedback, and engineering leaders who pay for all of the above. AI coding agents make it worse. JetBrains’ 2026 research found that more than half of all developers now write less than 20% of their code manually, and one in five writes absolutely no code without AI help. Agents produce changes faster than teams can validate them.

At MetalBear we build mirrord, which connects AI agents and developers to a live cluster so they get real-world context and a real feedback loop, so a lot of our time goes into working through exactly this problem with platform teams at companies like monday.com, Podium and SurveyMonkey. Many of the tools below come up in those conversations, either as alternatives teams evaluated or as other parts of their testing stack, which is where most of the comparisons in this guide come from.

In this guide, we'll compare 10 tools grouped by the layer of testing they cover, show them side by side in one table, list a few more worth knowing, and then look at how to combine them into a stack that fits your team.

Microservices testing tools compared at a glance

The tools below appear in the same order as in the rest of this article, grouped by category, starting with test environments.

Tool	Category	Isolation model	Cost	Best fit
mirrord	Test environments	Per-session isolation in one shared cluster (traffic filters, queue splitting, DB branching)	Open source free; Team $50/seat/mo (20% less billed annually); Enterprise custom	Many developers, CI runs and AI agents testing on one staging cluster at the same time
Signadot	Test environments	Sandboxes of changed services in a shared cluster, routed by request headers	Starter free; Business $250/mo; Enterprise custom	Isolated sandboxes per pull request
Telepresence	Test environments	Entire local machine connected to a remote cluster through an in-cluster Traffic Manager and Traffic Agents	Free, open source	Individuals or teams that want a fully open-source tool with no vendor behind it
Okteto	Test environments	Full environment per developer or per pull request in its own namespace	Starter free (5 seats); Scale and Enterprise custom	Personal ephemeral environments per developer
Tilt	Test environments	Full stack you define, usually on a local cluster	Free, open source	Local Kubernetes inner loop
Testkube	Test orchestration	In-cluster runner; tests run as Kubernetes workloads against whichever environment you target	Open-source agent free; Pro $700/mo per runner agent; Enterprise custom	Platform teams that want all test types run and tracked in one place
Postman	API testing	Desktop or web client, plus a CLI in CI, calling whichever environment you point it at	Free; Solo $9/mo; Team $19/user/mo (billed annually); Enterprise custom	Teams that want shared API tests that both developers and QA work with
WireMock	Service mocking	Mock services (in-process, standalone, Docker or cloud) standing in for a dependency	Open source free; Cloud free tier (1,000 calls/mo); Enterprise quote	Third-party APIs that are rate-limited or expensive to call
Pact	Contract testing	Consumer tests run against a Pact mock; providers verify recorded contracts in their own build; no shared environment needed	Open source free; PactFlow Starter free, Team from $115.42/mo (billed yearly), Enterprise custom	Organizations where many teams release services independently
k6	Performance testing	Load generator run locally, in CI, in-cluster via the k6 Operator, or in Grafana Cloud	Open source free; Grafana Cloud k6 includes 500 VUh/mo free, then from $0.15/VUh	Teams that want load tests as code, with pass/fail gates in CI

Pricing checked on each vendor's official pricing page on October 14, 2026.

What does a microservices testing stack need to cover?

The best tools for testing microservices architecture work as a stack, because no single tool covers every layer of microservices testing. A reliable stack combines several tools, and the environment they run in determines whether their results can be trusted.

These are the six layers we'll use throughout this article:

Test environment: where the service under test runs, whether its dependencies are real, mocked or copied, and how production-like the whole setup is.
Test orchestration: running and scheduling tests inside the cluster and collecting the results in one place.
API and integration testing: verifying a service's endpoints and its interactions with other services.
Service mocking: replacing dependencies that are unavailable, unstable or expensive to call, also known as service virtualization.
Contract testing: catching breaking changes between consumer and provider services before they're deployed.
Performance testing: checking how individual services and chains of services behave under load.

End-to-end and browser frameworks like Playwright, Cypress and Selenium run on top of these layers. They're important, but they depend on an environment to point at, so we aren't covering them as separate entries here. Unit testing also isn't covered, since it doesn't change much between a monolith and microservices.

Why treat the environment as the foundation? Because every other layer inherits its fidelity. An API test that passes against a mock tells you the code works against the mock, not that it works with the real services. A shared staging environment gives you real dependencies, but two people testing at once can overwrite each other's data or consume each other's queue messages, which leads to queuing up for your turn and long wait times as every change is tested independently. Duplicating the environment per pull request solves the conflicts but multiplies cloud cost and adds minutes of spin-up time to every change. And flows that go through message queues are the hardest to test before production: in a shared environment, a test consumer would pull messages meant for other consumers, and giving every test run its own broker is a lot of setup.

Here's how the 10 tools map onto those layers:

How we selected and ranked these tools

We selected and ordered the tools using the same criteria for each, so the list can be compared fairly.

Selection criteria: fit for Kubernetes-based microservices, how faithfully the tool reflects real dependencies, CI/CD readiness, and cost and maintenance effort.
Order of the list: test environments first, then test orchestration, then tools for individual test types. Environments come first because they determine whether the results of everything else can be trusted. The category label under each heading tells you which layer a tool belongs to.
Sources: each tool's official documentation and pricing page, last checked October 14, 2026.
Transparency: MetalBear builds mirrord. Every tool on this list, mirrord included, is described using the same criteria and only verifiable facts.

1. mirrord: best for testing against a real cluster without deploying

Category: Test environments

mirrord lets you run a service locally, in an agent's sandbox, or in a CI runner, as if it were running inside your Kubernetes cluster. It's best for teams that want to test against the real services, databases and queues in a shared cluster, without deploying code or maintaining copies of the environment. Developers use it for local development, but the same approach also runs in CI, powers preview environments, and gives AI agents a place to verify their own changes.

How mirrord works

When you start a mirrord session, mirrord connects your local process to a target in the cluster, like a deployment or a pod. Your process receives a copy of that target's incoming traffic (or takes it over, in steal mode), makes its outgoing connections and DNS lookups through the target pod, so cluster-internal databases and services are reachable, and reads the same environment variables and files as the pod. As far as your code is concerned, it's running in the cluster. Nothing gets built, pushed or deployed.

# Run your service in the context of the deployed checkout service
mirrord exec --target deployment/checkout -- npm run dev

The diagram below, from the mirrord docs, shows a mirrord session: the local process gets the target's incoming traffic, and its outgoing traffic, database access and file reads go through the cluster.

How it supports microservices testing

Because your service talks to its real dependencies, integration and end-to-end tests run against actual behavior instead of mocks. Developers do this from their IDE (VS Code, Cursor, JetBrains, etc.) or a terminal. In CI, mirrord for CI starts the service under test inside the runner and connects it to your staging cluster, so you can run your existing test suite against real dependencies without deploying a fresh environment for every run:

mirrord ci start --target deployment/checkout -- npm start
npm run test:e2e

mirrord ci works with the open-source version. If the mirrord Operator is installed in your cluster (which the team features below need), CI runs authenticate with a CI API key, which is part of the Enterprise plan and keeps CI sessions from taking up developer seats.

This is also what makes mirrord useful for AI coding agents. An agent can run the code it just wrote against real services, see what breaks, and iterate, in seconds rather than waiting on a full CI deploy cycle. MetalBear maintains an open-source skills plugin that teaches agents like Claude Code and Cursor how to configure and run mirrord.

Isolation for teams sharing one cluster

The obvious question with a shared cluster is: what stops people from stepping on each other? mirrord handles traffic isolation, message isolation and data isolation per session, rather than per environment:

HTTP traffic filtering routes only matching requests (for example, ones carrying your session header, or a specific path) to your local process, while all other traffic keeps going to the service in the cluster. With traffic mirroring, your process gets a copy of the requests instead, and the service in the cluster keeps handling the originals.
Queue splitting sends the messages that match your filter to your local consumer and leaves the rest for the shared consumer, so the two don't compete for the same messages. You can also mirror matching messages instead, so both your local consumer and the deployed one process a copy. It supports Kafka, SQS, RabbitMQ, Google Cloud Pub/Sub, Azure Service Bus, NATS, Redis Pub/Sub, BullMQ and Temporal.
Database branching gives each session an isolated branch of the database, so writes and migrations don't affect anyone else's data. It supports MySQL, MariaDB, PostgreSQL, MSSQL, MongoDB, Redis, DynamoDB, ClickHouse, CockroachDB, Google Spanner, S3 and turbopuffer, plus a generic option for any other service you run from your own image.

Beyond isolation, the Operator adds features for running mirrord across a team. Policies let platform teams limit which mirrord features can be used on which targets, for example blocking traffic stealing without an HTTP filter. Profiles give everyone shared, reusable mirrord configurations. Session management shows which sessions are active on which targets and lets you stop them. The admin dashboard shows who uses mirrord, on which targets and in which CI pipelines, and how much time it saves.

Preview environments

Local and CI sessions end when the process stops. A preview environment keeps a change running in the cluster so someone else can try it: a product manager, a QA engineer, a reviewer, or an AI agent checking its own work. A preview runs only the changed service, as a pod built from the existing workload's spec with your new image. Requests carrying the preview's key go to it, everything else keeps going to the existing workload, and the preview talks to the real databases, queues and upstream services in the cluster. You can share a preview as a link, and it stays up for a time-to-live you set, independent of anyone's machine.

That's the main difference from other preview environment tools that build a full copy of the environment for every pull request: nothing else gets duplicated. With idle mode, a preview also scales down to zero pods when it gets no traffic and starts again when a matching request or queue message arrives. A preview for every open pull request costs almost nothing while nobody is using it.

Previews also change how you review work you hand to an AI coding agent. With mirrord set up in its environment, the agent picks up a ticket, makes the change, builds and pushes an image, and runs mirrord preview start with the ticket ID as the key. Instead of judging the change from the diff alone, you open the preview, try it yourself against the cluster's real dependencies, and then decide whether it's ready to merge.

Chaos testing

Testing against real dependencies also means testing what happens when they misbehave. With chaos testing, you add rules to your session that can slow down or break your service's outgoing connections to specific hosts: added latency (with jitter), or connection resets, timeouts and refusals, applied to every matching connection or a percentage of them. The rules only affect your session, so you can see how your code handles a slow payment API or a dropped database connection without disrupting anyone else on the cluster.

We didn't find built-in latency or connection-failure injection in the docs of the other environment tools in this list. The closest is Signadot's Override middleware, which can return canned or error responses for requests to the sandboxed workload.

Here's what each plan includes, per the pricing page and docs:

Open source (free): the CLI and IDE extensions, mirroring and stealing traffic, HTTP filtering, chaos testing, and mirrord ci.
Team: adds the mirrord Operator in the cluster, which enables concurrent sessions against the same target, queue splitting, database branching, policies, RBAC and usage monitoring.
Enterprise: everything in Team, plus preview environments, CI API keys, high availability and support for air-gapped clusters.

Because isolation happens per session rather than per environment, adding another developer, CI runner or agent doesn't add another copy of your infrastructure, so cloud cost stays flat as the number of concurrent testers grows.

Real-world results

monday.com moved from per-developer cloud environments to one shared environment. Environment setup dropped from 30 minutes to 10 seconds, onboarding from 3–5 days to about an hour, and retiring the per-developer environments saves around $450 per developer per month. 350+ engineers now work on one cluster.
CoLab cut the time to get a change running in the cloud from over 15 minutes to seconds, a reduction in dev cycle time of up to 98%.
At Daylight Security, Claude Code verifies its own changes in a mirrord-powered preview environment against real pre-prod services. Each verification takes about 5 seconds, compared to the 5+ minutes a CI loop would take.

Trade-offs

mirrord isn't a load testing tool, and it doesn't replace contract tests or unit tests. It assumes you already have a cluster (usually staging) that's in reasonably good shape, because that's what your code runs against. Filtering traffic for several services in a call chain also requires that your services propagate the routing header between them. The team features described above need the operator installed in the cluster, which is a conversation to have with your platform team.

Pricing and licensing

Per the pricing page, the open-source version is free. The Team plan costs $50 per seat per month billed monthly, or 20% less billed annually, with no credit card required to start. Seats are counted by monthly activity. Enterprise pricing is custom and covers preview environments, CI API keys, high availability, support for air-gapped clusters and dedicated support.

See also: the mirrord vs. Signadot and mirrord vs. Telepresence comparisons, and the quickstart.

2. Signadot: best for isolated sandboxes per pull request

Category: Test environments

Signadot creates lightweight sandboxes inside an existing Kubernetes cluster. It's best for teams that want an isolated environment for every pull request without duplicating the whole stack.

What it does

A sandbox deploys only the services you changed. Everything else is served by the shared baseline environment, meaning the unchanged services already running in the cluster. According to Signadot's documentation, requests are routed to the sandbox or the baseline through header propagation (it names OpenTelemetry and B3), using its built-in routing or a service mesh like Istio or Linkerd. Signadot installs an operator in your cluster that keeps a secure tunnel open to the Signadot Control Plane, a service Signadot runs outside your cluster. The control plane serves the API for creating and managing sandboxes, and requests to the preview URLs it hosts are proxied through it into your cluster.

How it supports testing

You can run your existing test suites against a sandbox, create sandboxes per pull request from CI, and use Signadot's Smart Tests to compare a sandbox's behavior against the baseline. Signadot also supports local development against a sandbox.

When it fits, and when it doesn't

Signadot fits teams with many services who want PR-level isolation and are comfortable with their cluster keeping a tunnel open to a control plane that Signadot runs, rather than running every component inside their own infrastructure. Like any tool that routes by request headers (mirrord included), it depends on every service in a call chain forwarding the routing header. Its plans also count sandboxes per month, so teams that open many short-lived PRs should check the limits against their volume.

Pricing

Per Signadot's pricing page: Starter is free (50 sandboxes and 100 test invocations per month, one cluster). Business costs $250 per month (100 sandboxes per month, three clusters, extras billed in blocks). Enterprise is custom.

3. Telepresence: best for connecting a local service to a remote cluster

Category: Test environments

Telepresence is an open-source tool that lets you run a service on your machine while the cluster's network and services stay reachable. It's best for individual developers who want to debug a service locally against a remote cluster, or for teams that want a completely open-source, vendor-free tool and don't need commercial support.

What it does

Telepresence connects your workstation to the cluster network and can intercept traffic meant for a workload and route it to your machine, or send a copy of it ("wiretap") while the original keeps flowing. Intercepts can be filtered by header or path.

What gets installed

Per the Telepresence architecture docs, a cluster administrator installs the Traffic Manager, which coordinates connections and tracks active sessions. By default, a Traffic Agent is added to each workload you intercept, usually as a sidecar container, or optionally as a node-hosted agent. The pod restarts when the sidecar is first injected, and the sidecar stays in the pods after you disconnect until it's uninstalled. On the workstation, Telepresence runs a user daemon and a root daemon, which sets up a virtual network device and DNS for the whole machine. A Docker mode runs this inside a container instead.

Where it fits

Telepresence describes itself as "an open source tool for Kubernetes application developers," and its documentation focuses on local development and debugging against a remote cluster. The documentation doesn't describe a way to isolate databases between developers. For queues, it provides an API that consumers of an intercepted workload can call with a message's headers: the consumer in the cluster leaves messages that match an active intercept on the queue, and your local consumer takes them. That check has to be added to the consumer's code. It also documents a replace mode that takes the application container out of the cluster pod while you're attached, so only your local copy consumes messages and none get processed twice.

Licensing

Telepresence is open source under the Apache 2.0 license and a CNCF Sandbox project, originally built by Ambassador Labs. It's community-supported.

4. Okteto: best for ephemeral environments per developer or pull request

Category: Test environments

Okteto is a platform for development, preview and testing environments that run inside Kubernetes. It's best for teams that want each developer or pull request to get its own complete copy of the application.

What it does

Okteto gives developers remote ephemeral development environments with local-to-cloud code sync and remote debugging. For each pull request, it creates a preview environment defined in an Okteto manifest, triggered from CI tools like GitHub Actions or GitLab CI. It also offers testing environments for running test suites in the cluster.

How it supports microservices testing

By default, each environment is a full copy of the stack in its own namespace, with every service plus its own databases and queues. That makes isolation simple, since nothing is shared and nothing collides, but cloud cost grows with every open PR and active developer, and creating an environment takes as long as deploying the whole stack. Data in those environments typically comes from seed data rather than the real shared databases.

Okteto's Divert feature narrows this: you deploy only the services you change, and requests for the rest are routed to a shared environment using the baggage header, as long as your services propagate it. That shared environment has to be deployed by Okteto, though, so unless your staging already runs on Okteto, you first have to redeploy or duplicate your services there.

Pricing

Per Okteto's pricing page, the Starter plan is free for 5 seats. Scale and Enterprise pricing isn't published. Okteto is self-hosted on your own cluster, so you also pay the cloud cost of the environments it creates. The Okteto CLI is open source.

5. Tilt: best for local Kubernetes inner-loop development

Category: Test environments

Tilt is an open-source tool, maintained by Docker, that automates the inner loop of Kubernetes development. It's best for developers who want fast rebuilds against a local cluster.

What it does

You describe your services in a Tiltfile, run tilt up, and Tilt watches your files and, by default, rebuilds the container image and redeploys it on every change. If you configure Live Update, Tilt syncs changed files straight into the running containers instead, which skips the image rebuild. The Tilt UI shows logs and status for every resource in one dashboard.

Where it fits

Tilt is great when the whole system you need fits on your machine, and that's the setup its docs steer you toward. Its guide to choosing a dev cluster covers local clusters (kind, Docker Desktop, MicroK8s, Minikube, k3d and Rancher Desktop) first, and treats remote clusters as the advanced option. Tests you run against a Tilt environment hit real containers of your services, but only the ones you've defined, often with local databases and seed data.

Where it stops being enough

The ceiling is your machine. The same guide points out that some local cluster options, like Docker Desktop and Minikube, are heavy on local resources. Tilt can also work against a remote dev cluster, where developers can share common services like a dev database, but the same guide says "we only recommend remote clusters for large engineering teams where a dedicated dev infrastructure team can maintain your dev cluster," or for debugging "something that only reproduces in a complete cluster." So moving past the local machine means taking on a dev cluster that someone has to run and maintain.

Licensing and pricing

Tilt is free and open source under the Apache 2.0 license.

6. Testkube: best for running any test framework inside the cluster

Category: Test orchestration

Testkube runs your tests as Kubernetes-native workloads inside the cluster. It's best for platform teams that want one place to run, schedule and see the results of every kind of test.

What it does

In most pipelines, the CI system (GitHub Actions or GitLab CI, for example) runs integration, API, end-to-end and load tests on its own runner machines, which reach the cluster over the network. Testkube runs those same tests as workflows inside the cluster instead, with scheduling, Kubernetes event triggers, CI/CD integration, and logs and artifacts collected centrally.

Supported frameworks

Per Testkube's documentation, examples cover k6, JMeter, Gatling, Locust and Artillery for load testing; Cypress, Playwright, Selenium and Robot Framework for end-to-end tests; Postman, SoapUI, REST Assured and cURL for API testing; and Pytest, Maven, Gradle, NUnit and Testcontainers for unit and integration tests.

What it doesn't do

Testkube orchestrates tests, but it doesn't provide the environment they run against. It complements environment tools: you still need to decide whether your tests hit staging, a per-PR copy or a service connected through something like mirrord.

Pricing

Per Testkube's pricing page, the open-source agent is free with community support. Pro costs $700 per month per runner agent and adds a hosted control plane, parallel execution, analytics and flakiness detection. Enterprise is custom. AI add-ons are priced separately.

7. Postman: best for API testing of microservices

Category: API testing

Postman is an API platform for building, testing and documenting APIs. It's best for teams that want shared, versioned API tests that both developers and QA can work with.

What it does for microservices

You build requests and group them into collections, define environments with variables (for example, local, staging and production base URLs), and write test scripts that assert on responses. The Collection Runner runs every request in a collection in the order you choose and logs the test results for each one, and the Postman CLI runs collections the same way in CI. Postman can also host mock servers.

Where it fits

Postman sits at the API and integration layer, and it's strongest as a collaboration tool: a shared collection becomes the living reference for how a service's API is supposed to behave. When teams shortlist the best API testing tools for microservices, Postman usually sits next to code-first options like REST Assured (for Java) and Karate, which keep tests in the repository next to the service.

Limitations

Postman tests whatever environment you point it at, so how much a passing collection tells you depends on that environment. Large collections also become code you have to maintain, with the scripts living inside Postman rather than next to the service in your repository by default. Postman can send HTTP, GraphQL, gRPC, WebSocket, Socket.IO and MQTT requests, but its docs describe collection runs for HTTP, GraphQL and gRPC tests only, and GraphQL and gRPC collection runs need a paid plan.

Pricing

Per Postman's pricing page, checked October 14, 2026: Free (1 user), Solo at $9 per month and Team at $19 per user per month, both billed annually, plus a custom Enterprise plan. The Collection Runner and mock servers are included on every plan, with GraphQL and gRPC collection runs limited to paid plans.

8. WireMock: best for mocking unavailable dependencies

Category: Service mocking

WireMock is an open-source tool for simulating HTTP APIs, a practice also called service virtualization. It's best for teams that need to test against dependencies that are unavailable, unstable, rate-limited or expensive to call.

What it does

WireMock stubs HTTP endpoints with the responses you define. It supports stateful behavior, fault injection (malformed responses, dropped connections), artificial delays, and record-and-replay, where it proxies a real API and saves the responses as stubs. It runs embedded in a test, as a standalone process or as a Docker container.

When to mock, and when not to

Mocking is the right choice for third-party APIs you can't call freely (payment providers, for example), for dependencies that don't exist yet, and for failure scenarios that are hard to trigger on demand. For your own internal services, testing against the real thing usually catches more, because a stub only knows about the behavior you told it about.

The maintenance cost

Every stub is a copy of another service's behavior at one point in time. When that service changes, the stub doesn't, and tests keep passing against behavior that no longer exists. In our experience, this drift is the most common reason teams with large mock suites still get surprised when they ship to production. Pairing mocks with contract tests (see Pact below) helps keep them honest.

Pricing and licensing

WireMock is open source under the Apache 2.0 license. Per the WireMock Cloud pricing page, the hosted Free plan includes 1,000 mock API calls per month for a single user. Enterprise pricing is quote-only and adds stateful mocking, team management and Kubernetes self-hosting.

9. Pact: best for contract testing between services

Category: Contract testing

Pact is an open-source framework for consumer-driven contract testing. It's best for organizations with many services that are deployed independently by different teams. If you're comparing the best tools for contract testing microservices, the usual alternatives are Spring Cloud Contract for JVM teams, and spec-driven tools like Specmatic and Microcks, which turn OpenAPI and AsyncAPI specs into contract tests and mocks.

What it does

The consumer of an API writes tests describing the requests it sends and the responses it expects. Those tests run against a Pact mock and produce a contract. The provider then verifies that contract against its real implementation in its own build. If the provider changes in a way that breaks a consumer, the provider's build fails, before anything is deployed.

Pact Broker, PactFlow and can-i-deploy

Contracts are shared through the Pact Broker (self-hosted) or PactFlow (the managed version). The can-i-deploy command checks the broker to tell you whether a specific version of a service is compatible with everything already in the target environment. Per the Pact docs, official implementations exist for JavaScript, the JVM, .NET, Go, Python, Swift, Scala, PHP, Ruby, Rust and C++.

When it pays off, and the limitations

Pact pays off when many teams release independently, so a provider team can't check every change by hand with each of the teams that consume its API. Those consumer teams need to be known and willing to write Pact tests too, which is why Pact's docs list public APIs as a poor fit. The cost is coordination: consumer and provider teams both need to adopt it, and there's a learning curve around provider states and the broker workflow. Contract tests also only check the shape of interactions, not whether the full flow works end to end.

Licensing and pricing

Pact is open source, under the MIT or Apache 2.0 license depending on the language implementation (Pact JVM, for example, uses Apache 2.0), and the Pact Broker can be self-hosted for free. Per PactFlow's pricing page, the Starter plan is free (2 integrations), Team starts at $115.42 per month billed yearly, and Enterprise is custom.

10. k6: best load testing tool for microservices

Category: Performance testing

k6 is an open-source load testing tool from Grafana Labs. It's best for teams that want performance tests written as code and enforced in CI. The best load testing tools for microservices usually come down to a shortlist of k6, Gatling and Apache JMeter. k6 stands out for its JavaScript scripting and its Kubernetes operator.

What it does

You write load tests in JavaScript, define thresholds (for example, p95 latency under 300 ms or an error rate under 1%) that act as pass/fail criteria, and stream results to tools like Prometheus or Grafana. Because thresholds fail the run, k6 fits naturally as a gate in a CI pipeline.

Running k6 in Kubernetes

The k6 Operator adds a TestRun custom resource. You set a parallelism value, and the operator starts that many k6 pods and splits the workload between them. Testkube can also run k6 tests in-cluster.

Limitations

k6's documentation notes that a single machine may not generate enough load for very large tests, which is what distributed runs are for. Results also only reflect the environment you test: load testing a scaled-down staging cluster tells you about staging, not production.

Licensing and pricing

k6 is open source under the AGPL-3.0 license. Per Grafana's pricing page, Grafana Cloud k6 includes 500 virtual user hours (VUh) per month for free, with additional usage starting at $0.15 per VUh. The Pro plan has a $19 monthly platform fee.

Other microservices testing tools worth knowing

The 10 tools above cover each layer of the stack, but they aren't the only options. These are worth a look if one of the tools above doesn't fit your setup.

Tool	Category	What it is
Release	Test environments	Environments-as-a-service platform that creates full-stack ephemeral environments per pull request
Bunnyshell	Test environments	Environments-as-a-service platform for ephemeral preview and development environments
vCluster	Test environments	Open-source virtual Kubernetes clusters, each with its own API server, running inside a namespace of a host cluster
Microcks	Service mocking, contract testing	CNCF Incubating open-source tool for API mocking and contract testing across OpenAPI, AsyncAPI, gRPC and GraphQL
Specmatic	Contract testing	Turns OpenAPI and AsyncAPI specs into executable contract tests and service virtualization
Gatling	Performance testing	Open-source load testing tool with tests written as code
Apache JMeter	Performance testing	Long-standing open-source load testing tool with a GUI test plan editor and broad protocol support
Playwright	End-to-end testing	Open-source browser automation framework from Microsoft for end-to-end tests across Chromium, Firefox and WebKit
Testcontainers	Integration testing	Open-source library that starts throwaway Docker containers (databases, Kafka and more) from your test code

Which combination of tools fits your team?

The right combination depends on your team size, the number of services, how many teams share your staging environment, and how independently your teams release. Here are four common profiles.

A small team with a few services and no shared staging pain yet. Tilt for the inner loop, Postman for API tests and k6 for a basic load test in CI is plenty. If you're running everything locally without trouble, you don't need a shared-cluster tool yet. Revisit when laptops start struggling or the service count grows.

A platform team supporting many developers on a shared staging cluster. This is where environment isolation matters most. mirrord lets everyone test against the same cluster with traffic filtering, queue splitting and database branching, without per-developer environments. Signadot is the closest alternative, with sandboxes for both local development and pull requests. The difference is in how databases and queues get isolated. Signadot does it through resource plugins (it publishes examples for MariaDB, PostgreSQL and SQS) or routing-key checks in consumer code, while mirrord has database branching and queue splitting built in, configured in mirrord rather than in each consumer's code. Okteto fits if you'd rather give each developer and PR a full copy and can absorb the cloud cost. Add Testkube if you want one place to run and track all your test suites in-cluster.

A team using AI coding agents that generate changes faster than the pipeline can validate them. The bottleneck here is verification, and every agent needs a real place to test its work. Per-PR environment copies get expensive quickly when each agent spins one up for every change. mirrord's per-session isolation lets many agents test against one cluster concurrently without adding infrastructure for each one, and the mirrord skills plugin helps agents set it up. Keep Pact in place so agents can't silently break an API that another team depends on.

A larger organization with independently released services owned by different teams. Pact with can-i-deploy keeps teams from breaking each other's APIs. Pair it with an environment tool for integration testing (mirrord or Signadot on a shared cluster), WireMock for third-party dependencies, and k6 for performance budgets on critical paths.

Whichever profile you fit, start with one layer, usually the environment, since it affects everything else. Run a pilot on one service, measure feedback time and flaky test rate before and after, and only then roll it out further.

Conclusion

The best tools for microservices testing work as a layered stack, and the fidelity of the environment decides how far you can trust the results of every other tool in it.

To recap: mirrord, Signadot, Telepresence, Okteto and Tilt cover the environment layer, each with a different isolation model. Testkube orchestrates tests inside the cluster. Postman covers API testing, WireMock handles dependencies you can't call, Pact catches breaking changes between teams, and k6 tells you how services behave under load.

For teams that want to test against real dependencies without deploying or duplicating environments, mirrord is a strong option. monday.com uses it to run 350+ engineers on a single cluster instead of hundreds of per-developer environments, and teams like Daylight use it to let AI agents verify their own changes in seconds.

Try mirrord for free or book a demo, and check out the quickstart guide to run your first session in a few minutes.

Read this super detailed blog to learn how to evaluate preview environment tools!

Arsh Sharma — Wed, 21 Oct 2026 14:02:18 +0000

Jake Page for MetalBear

Oct 21

Best Preview Environment Tools for Kubernetes

#kubernetes #ai #agents #software

11 min read

learn how engineering teams are actually using AI!

Arsh Sharma — Thu, 08 Oct 2026 10:42:46 +0000

Eyal Bukchin for MetalBear

Oct 8

How Our Engineering Team Uses AI, Part II: Meat Proxies

#ai #engineering #productivity #programming

7 min read

[Boost]

Arsh Sharma — Thu, 08 Oct 2026 10:32:41 +0000

Eyal Bukchin for MetalBear

Oct 7

Build vs. Buy Your Kubernetes Development Environment

#kubernetes #devops #platformengineering #productivity

11 min read

Read how monday.com is testing AI agents!

Arsh Sharma — Mon, 21 Sep 2026 11:09:15 +0000

Arsh Sharma for MetalBear

Sep 21

How monday.com Runs Agent Evals Against Real Dependencies: Webinar Recap

#ai #testing #llm #agents

7 min read

How monday.com Runs Agent Evals Against Real Dependencies: Webinar Recap

Arsh Sharma — Mon, 21 Sep 2026 11:08:48 +0000

An agent eval suite's outcome can only be trustworthy if it's operating in an environment similar to production. You can have the best grading logic in the world, but if the agent is calling mocked databases and fake APIs, you're not testing how it behaves in the real world, you're testing how it behaves in the mocks you built. That gap is what we dug into on September 15, when we sat down with Dor Cohen, AI Engineering Director at monday.com, for an interview and demo on how his team architected running agent evals against a live staging cluster instead of mocks.

If you missed the session, here's the recap: what agent evals actually need to check, why mocks fall short compared to the agent calling real systems, and how monday.com built their agent evals infrastructure.

What are agent evals and why you need them

Testing an agent isn't the same as testing a model. An agent is a package: the model, the tools it calls, and the dependencies behind those tools. Unlike a unit test, where you know exactly what output to expect, an agent is non-deterministic, as it can pick a different tool or path on every run. So the eval has to check the full trajectory it takes to get there, not just whether the final answer sounded right.

Dor shared an example of why testing the full chain matters. They had an agent that was asked to retrieve 600 items. It called the right tools, formatted a confident, well-structured answer, and returned a response that read as correct. Except it had only processed about 500 of them. Every individual tool call worked. The final answer looked fine, but the task wasn't actually done. An eval that only grades the final answer would have passed it but an eval that checks trajectory and completeness catches it.

Where mocks fall short

One very important part in the agent eval chain we discussed above is the dependencies your agent calls. The default way to run evals is against mocks because they're lightweight, fast to set up, and isolated, and for a lot of testing that's exactly what you want. But for an agent that's calling real databases, services, and APIs in production, mocks introduce three problems.

They drift: a mock has to be kept in sync with your production environment by hand, and it never is.
The data isn't real either, synthetic rows present in mocks are cleaner and thinner than what actually lives in your database, so they can't expose the edge cases production will.
And mocks can't represent state. When an agent reads data and takes a downstream action, a mock either forgets the write happened or fakes it, so you can't actually tell what the agent left behind.

Staging doesn't have any of those problems by construction. It stays current with production because it's the last stop before it. It runs on real data, at close to production scale, with all the variety and mess that comes with that. And because the agent's actions actually land, you can check the real end state, not a simulated one. Fidelity to production is what makes or breaks the quality of your agent evals.

The objection to running evals against staging (or other pre-prod environments) is reasonable, though: it's not lightweight, not isolated, and not easy to wire up. That's the problem monday.com solved with mirrord.

How mirrord fits into the pipeline

mirrord connects a process running locally, in a CI job, or in an eval runner, to a real Kubernetes cluster, so it behaves as if it's deployed there: real traffic mirrored or stolen from a live pod, real env vars and secrets imported straight from the target, real reads and writes on the remote filesystem. DB branching and queue splitting isolate each eval session's writes, so multiple runs, from different engineers or different CI jobs, can hit the same cluster at the same time without stepping on each other's data.

There's no separate deploy step as well, you just wrap your existing eval command and run it with mirrord:

mirrord exec --target deployment/orders-api \
    -- pytest evals/agent_suite.py

And in CI you can just use mirrord ci start instead of mirrord exec, with the same arguments. When a model, a prompt, or a tool changes, your CI job runs the suite against real dependencies before the change reaches a merge gate.

Inside monday.com's agent evals pipeline

monday.com runs 700+ microservices behind its agents, with the usual sprawl that comes with that scale: authentication, feature flags, permission layers, and a long list of third-party integrations. Mocking these dependencies or setting them up each time in an ephemeral environment would've been a huge investment for them in terms of both time and money. Their estimate for a dedicated evaluation environment came out to about a month of setup, plus ongoing maintenance across 700+ services just to keep it in sync. Obviously, they ended up going the more practical route of pointing their evals at staging with mirrord, which their developers were already using for local development.

The difference shows up in the day-to-day now: agent eval suites run in minutes instead of hours, the added infrastructure cost is close to zero since monday.com already pays for staging, and they get an output that they can actually trust.

Catching a regression before it ships

monday.com treats a model upgrade the same way it treats any other version change to the agent: it runs the full eval suite against the new model before switching. When the team tested Sonnet 5 against their existing default model, goal completion dropped by about 10%, and agent correctness dropped by about the same. The suite failed to cross the pass threshold.

This was great for the team as it gave them a clear signal that the upgrade wasn't safe to ship as-is, pointed to specific failure points in tool selection and prompt adherence, and gave them a concrete direction for what to adjust before trying again. Without a suite running against real dependencies, this is the kind of regression that tends to surface only after a customer hits it.

Three ways monday.com actually runs AI agent evals

Dor shared that their eval pipeline isn't just limited to CI pipelines, it's the same underlying infrastructure triggered from three different places, all producing consistent results through mirrord and LangSmith.

From Claude Code, a developer working locally can invoke an offline evals skill right after making a change. It asks for the eval scope, all cases or a specific category, connects to dependencies living in staging through mirrord, and runs the suite. Results land in LangSmith with input, reference output, actual output, and a score per metric.
From Slack, monday.com has an internal Claude-based evaluation agent, nicknamed "Eevee," that can be asked to run evals with specific parameters, like swapping in a different model for a test run. It uses the same infrastructure, produces the same LangSmith output, and posts a summary back in Slack. This is actually how the team caught the ~10% drop in goal completion during the Sonnet 5 test.
From CI, eval results show up alongside regular checks on the pull request, each with a score and a mode: observed, which is informational, or blocked, which prevents the merge, depending on the severity of the change and the product it's touching.

monday.com runs these "offline evals" alongside "online" ones. Offline evals run against their staging environment with controlled datasets, gating changes before they ship. Online evals grade live agent behavior continuously after it is shipped to production. When something surfaces in production that the offline dataset didn't cover, it gets pulled into that dataset for future regression testing, closing the loop between what ships and what gets tested next.

Metrics to track for agent evals

monday.com's base eval pipelines track six metrics:

Goal completion: did the agent finish the whole task, not just appear to
Agent correctness: was the answer factually and logically right
Tool precision: did it pick the correct tools for the task and call them in the correct sequence
Trajectory: was the path taken the expected one
Groundedness: are the agent's claims backed by evidence it actually had
Intercompleteness: did it gather everything it needed before answering

Some of these are fully deterministic, like whether the required tool got called. Others need an LLM-as-judge, for things like whether the agent understood the goal or stayed in scope. And individual product teams at monday.com add their own domain-specific evaluators on top of the base set, because what counts as a passing score depends entirely on the product and the use case.

Takeaways

A few things worth carrying away from the session:

Agent evals should test more than just the final output: tools picked, dependencies used, trajectory taken, and completeness all matter.
Mocks drift, and synthetic test data can't replicate the variety and edge cases that live in production.
Most teams already have the right environment for running agent evals properly. Staging is current with production by definition. Building a dedicated eval environment from scratch is usually more expensive and slower than pointing your suite at the one you already run.
Online and offline evals reinforce each other. Edge cases from production belong in your offline dataset, so the two tracks compound over time instead of drifting apart.

If you'd like to see the full webinar recording including the demo Dor showed, you can get it here. And if you want to set up a 1:1 session with one of our engineers who'll walk you through how to connect your agent evals to real dependencies with mirrord, you can book a demo with us.

How To Reduce Token Costs of AI Coding Agents

Arsh Sharma — Thu, 13 Aug 2026 11:01:59 +0000

Costs related to AI coding tools are projected to only go up from what they are today. We've seen this before with Anthropic adding weekly rate limits on top of Claude Pro and Max's existing five-hour windows, with little warning. So the question on everyone's minds these days is how do you get more work out of the same AI budget.

One answer is you have your AI agents burn less tokens. But token burn isn't one problem. An agent stuck looping on a bug it can't see and an agent reloading the same instructions on every single prompt look identical on your bill: a long session, a high token count. But the causes are completely different, and so are the fixes.

In this post, we'll go through five specific ways this happens:

Agents reading half your repo to change three files
Agents doing mechanical work on your most expensive model
Agents guessing at what a live dependency looks like, or trusting a mock that drifted away from it
Agents looping without any way to tell if their fix actually worked
Agents loading instructions on every prompt instead of only when needed

For #3 and #4 we'll show how mirrord, our Kubernetes development platform, closes the gap: it connects code to a real cluster without deploying it, whether that code runs on a laptop, in CI, or in an agent's sandbox.

Reading half the repo to change three files

Point an agent at a large codebase with a vague instruction and it will go exploring, and that exploration is usually the single largest line item in a session: file after file pulled into context, most of which never ends up informing the change. What makes it expensive isn't the reading itself so much as the compounding, because a file read on turn three is still being re-sent on turn forty.

The easiest fix to this is just being specific in your prompts. If you already know the change lives in three files, just say so. If the search genuinely is the task, hand it to a subagent, which does the fan-out reading and returns a conclusion, so the forty files it opened never enter your main context at all. And when you move on to something unrelated, start a fresh session rather than continuing the old one, because a long-running session carries its full history into every new request whether or not any of it is still relevant.

Paying premium model rates for mechanical work

Not every task needs your best model. Renaming a symbol across a repo, writing a migration from a schema you've already specified, reformatting test fixtures, drafting a commit message: these are all mechanical, and a smaller model does them at a fraction of the cost.

The mistake is treating model choice as a per-project setting rather than a per-task one. Reach for the expensive model when the work is genuinely architectural, when you're debugging something subtle or designing an interface or deciding how a change should be structured, and drop down for everything downstream of that decision. Most agent tools let you switch mid-session, and most people never utilize that.

Guessing at environment shape vs. reading it live

An AI agent writing code for a microservice doesn't get to see the cluster where the entire application is deployed. It only sees your repo, whatever context you gave it, and patterns from its training data, and it fills every other gap with an assumption: what response a downstream service returns, what a queue message actually looks like, what a config value resolves to once it's deployed. Some of those assumptions are wrong, and the agent has no way to know which ones until the code runs against the real thing.

Say you ask an agent to add a consumer for order events, and it writes the handler expecting a flat total_cents field, because that's what the schema doc in the repo still says. In staging, that event has looked like {"pricing": {"total": ...}} for the last two months, ever since a migration that never made it back into the docs. The agent doesn't find out until it runs the code and the field comes back None, and at that point fixing it means re-deriving the real shape from the failure and rewriting the parsing logic from scratch. Every downstream call, config value, queue, or database the code touches carries the same risk, and each wrong guess costs a whole new bag of tokens.

The usual answer to this is a mock, and mocks make it worse rather than better, because now the agent isn't missing information, it's being handed information that's confidently wrong. A mock is a snapshot of what a dependency looked like the day someone created it, and nothing keeps that snapshot in sync automatically, so the gap only ever grows. Say the real order queue gains a required currency field, and the agent knows about it from a recent PR, so it writes a consumer that correctly expects the field. Locally it fails, because the mock is still emitting the old shape. The agent has no way to know the mock is the outdated one, so it assumes its own code is wrong and starts "fixing" it, loosening the field to optional, adding fallback logic, trying variation after variation, none of which touch the actual bug because there isn't one. The bug is the mock. The agent just burned a dozen turns proving that correct code doesn't satisfy an incorrect test.

The fix for both is to stop guessing. With mirrord connected to your cluster, an agent can read the real shape of a dependency before it writes a single line: what's actually on the queue right now, what a service returns today, or what a ConfigMap resolves to in the running pod. That collapses the guess-then-fail-then-regenerate cycle most AI-generated code goes through into just one or two passes, because the first draft is already written against what's true instead of what's assumed. And there's no need to maintain mocks that drift, because you can get a response from the dependencies running in the cluster directly.

No integration feedback

Without a way to verify its own work, an agent can make a change, declare it done, and open a PR with a bug baked in that only shows up once the code touches the real dependencies in the cluster. Nobody catches it at write time. It gets caught in review instead after a CI run or a staging deploy, which means going back, explaining what's actually wrong, and waiting on another round of fixes, and possibly another after that if the first fix is incomplete. Every one of those review-fix-review rounds costs tokens the agent wouldn't have spent if it had caught the bug itself the moment it introduced it.

Give the agent the same tests your team already runs in CI, but let it run them itself, against a real environment, right after it makes a change. Then it catches that kind of bug in the same turn it introduced it, not two review cycles later. We've walked through exactly this kind of case in detail in our previous blog How to prevent token burn using mirrord with e2e tests.

Context loaded on every prompt vs. on-demand skills

Most AI coding agents support an instructions file like AGENTS.md or CLAUDE.md that the agent reads in full on every single prompt in a session. It's the simplest way to tell an agent "always do X, never do Y," but that simplicity has a cost baked in. The file loads regardless of whether the current prompt has anything to do with it. If you've added a paragraph asking the agent to use mirrord to verify its code, one about your auth conventions, one about deployment rules, the agent is now paying to re-read all three on every request, including one that touches none of them.

Agent Skills fix this by making the loading conditional. Each skill ships as its own file with its own description, and the model only pulls it into context when a prompt actually matches what that skill covers. We packaged mirrord's own instructions this way: six Agent Skills covering commonly used mirrord use cases, installed in one command. Ask an agent to set up queue splitting and only the mirrord-kafka skill loads. Ask it to fix a typo in your frontend and none of the six load because nothing in that prompt matches any of them. The rule of thumb is that your always-loaded file should only hold what's true for every prompt, and everything else belongs in a skill.

Token costs are going up, but your bill doesn't have to

The first two scenarios trace back to the same root cause: an agent that can't see what's actually true about your cluster, so it guesses at a dependency's shape, gets fooled by a mock that no longer matches reality, or ships a change with no way to check it actually works. mirrord closes that gap by connecting the agent's locally running code to your real cluster. The agent now knows what's actually on the queue, what a service returns, what the values of the environment variables the process inherits in the cluster are, so it is testing against reality instead of an assumption.

The other three, on-demand skills and tighter context scoping and matching the model to the task, are solving different problems, but the same principle applies to all of them: only pay for what the current prompt actually needs, and not a wasted guess, an irrelevant paragraph of instructions, or forty files the agent read once and re-sends forever.

If your AI agents are writing code for Kubernetes services, give mirrord a try. Our guide on using mirrord with AI agents walks through the setup for each of these workflows.

How Prathyusha Ramanjaneyulu at Celonis Eliminated Dev Environment Overhead for 600 Engineers

Arsh Sharma — Wed, 01 Jul 2026 13:05:29 +0000

I recently spoke with Prathyusha Ramanjaneyulu, a Senior Software Engineer at Celonis. She owns the local development component of the organization's internal developer tooling and is responsible for maintaining high velocity for over 600 developers. The conversation was about a problem that comes up at nearly every large engineering org: how do you give hundreds of developers a way to test their code against a real environment without having them being slowed down by CI and staging deployments? She described environments that took weeks to set up, 3+ hours a week lost to VM maintenance, and teams blocked whenever someone pushed a bad change. This is a recap of that conversation. I'll share the approaches she was using, the issues she faced with those, and how mirrord helped her finally solve this problem.

The problems she told me are ones I think many engineers at large companies will relate to and can learn from.

The Nature of the Bottleneck at Scale

The engineering org she works with has around 600 developers building applications that have a microservices architecture and are deployed on Kubernetes. The nature of that setup means that when an engineer is working on a single service, their code doesn't run in isolation but it depends on a web of other services, databases, and queues. Testing even a small change means the service needs to interact with those dependencies in a way that reflects what actually happens in production.

What she had tried

Before landing on a solution that worked, she went through two approaches.

Approach 1: Run everything locally

Every engineer would clone, compile, and run all the services their code depended on locally, which in an org of that size meant a lot of repos and a lot of compute. And even after spending the time to do all that, the machines were so resource-constrained while running all of it that engineers couldn't do much else at the same time. And because each engineer was running services they hadn't written, they'd inevitably run into issues they had no context to debug. This led to them spending time on problems that had nothing to do with the code they were actually trying to test.

Approach 2: Shared VM + some local services

The second approach moved some services to a shared virtual machine (VM) while keeping others running locally. That got the initial environment setup time down from the weeks required by the all-local approach to roughly a day and a half, which was meaningful progress. But it introduced a different set of problems. Engineers were spending 3+ hours a week on VM maintenance and things like debugging connectivity issues. This was time that had nothing to do with writing code. And because the VM was shared, when one engineer's change broke something, everyone working against that environment was blocked until it was resolved. The bottleneck had moved, but it hadn't gone away.

The Selection Criteria for Scaling Local Development

As the owner of the Local Development component, Prathyusha is tasked with maintaining high engineering velocity for 600 developers. She mentioned her team maintains a proactive watch over the cloud-native ecosystem; it was through this strategic evaluation that she and her team identified both Telepresence and mirrord as potential solutions to address their bottlenecks.

Both tools solve the same core problem: connecting a local development environment to a remote Kubernetes cluster. Here's the reasoning she gave me for landing on mirrord.

Philosophy. Telepresence's model is "your workstation becomes part of the cluster network." mirrord's model is "your local process impersonates a process inside a pod." With Telepresence, a broader portion of your machine inherits cluster connectivity and behavior. With mirrord, only the specific process being run is affected. That narrower scope reduces blast radius and configuration complexity.
Operational overhead. Telepresence requires managing daemon processes and per-user configuration across the team, which adds ongoing maintenance work for the platform team. mirrord is plug-and-play. Engineers pick it up and use it without her having to maintain it.
Security. Telepresence relies on Preview URLs, which are public endpoints, to enable collaboration. mirrord operates entirely within the private network, so there's no risk of accidentally exposing data through a public endpoint.
Resource footprint. mirrord only consumes cluster resources during an active debug session. Nothing persists between sessions, which keeps the staging cluster clean and means the platform team isn't maintaining persistent dev-tooling components on top of everything else.

What changed

When Prathyusha walked me through the impact, I asked for the numbers first. She mentioned that with the all-local approach, environment setup took weeks per engineer. With the shared VM, that came down to about a day and a half. With mirrord, it takes around 15 minutes. That's a 144x speedup over the shared VM approach! She also mentioned how the time between an engineer writing code and knowing if it works went from an hour down to seconds.

But beyond the speed, she pointed to the elimination of an entire category of problem. With mirrord, testing locally became equivalent to testing in staging as engineers connect their local process directly to the real cluster, with real dependencies. That closed the environment gap that caused the "it works on my machine but not in staging" problem. She described it as a shift in the confidence of the engineers: if it works with mirrord, engineers don't have to wonder whether it'll work in staging or not.

The shared environment also solved the blocking problem that plagued the shared VM approach. With the VM, one engineer's bad change could bring the environment down for everyone. With mirrord, any number of engineers can test against the same staging cluster at the same time without affecting each other. mirrord handles the isolation through features like HTTP traffic filtering, queue splitting and DB branching which ensure each engineer's test traffic and changes don't affect other people using the environment.

What all of this added up to was engineers getting their time back.

No more 3+ hours a week wasted on VM maintenance and configuration.
No debugging of services they didn't write.
No waiting on the dev tools team to fix a broken shared environment.

To put the capacity impact in perspective: if mirrord saves each engineer just one hour per week, at 600 engineers that's 600 hours reclaimed every week. Over a year, that's 30,000 hours of engineering capacity redirected from infrastructure maintenance to product work.

Closing remarks

The thing that struck me most from talking with Prathyusha was how much of the improvement came from changing the model, not the tooling. At this scale, even small amounts of per-engineer overhead compound into a really big cost for the organization as a whole. But the solution wasn't faster local tooling or more compute. It was changing the model: instead of every engineer maintaining their own isolated environment, share one staging cluster and just have the DevOps team maintain that.

Great post on using AI SREs!

Arsh Sharma — Wed, 24 Jun 2026 12:48:05 +0000

Eyal Bukchin for MetalBear

Jun 24

Auto-verifying your AI-SRE's fixes (Part II): HolmesGPT end-to-end on a real cluster

#ai #sre #kubernetes #devops

5 min read

if you're already using AI SREs, check this out!

Arsh Sharma — Thu, 11 Jun 2026 11:08:51 +0000

Eyal Bukchin for MetalBear

Jun 11

Auto-verifying your AI-SRE's fixes against your real cluster, with mirrord

#ai #sre #kubernetes #devops

8 min read

A Beginner's Guide to Helm

Arsh Sharma — Wed, 03 Jun 2026 12:25:24 +0000

The CNCF Annual Survey published in January 2026 showed that Helm is the second most used CNCF project, second only to Kubernetes itself. The reason is simple: once you start working with Kubernetes, YAML files accumulate fast and managing them keeps getting harder. A single microservice can require a Deployment, a Service, a ConfigMap, and a Secret, and that's before accounting for the different sets you’ll need for your development, staging, and production environments. Helm fixes this by packaging all of your manifests into a single, versioned unit that can be installed, upgraded, and rolled back easily.

In this guide, we'll cover what Helm is, some key concepts, and the main Helm commands you need to know to deploy and manage manifests using it. You’ll walk away with a complete understanding of how to use Helm so you can simplify the YAML mess you might be in.

Prerequisite: Basic familiarity with Kubernetes

What is Helm in Kubernetes?

Helm is a package manager for Kubernetes. Just as apt manages software packages on Linux or brew does on macOS, Helm manages Kubernetes applications and lets you install, upgrade, and roll them back as versioned, reusable units called charts.

The way Helm solves the YAML management problem is easiest to understand through a programming analogy. Bare Kubernetes manifests work fine for a single, static deployment. But the moment you need multiple similar deployments (dev, staging, prod, which may differ only by a few values), you're essentially copy-pasting. Helm solves this the same way classes solve it in object-oriented programming: you define the boilerplate once in a chart, which can include things like the Deployment template, the Service definition, the ConfigMap structure, and then each Helm chart deployment only specifies what's different: the image, the environment variables, the command being run in the pods. Someone installing your chart can also supply their own values file and get a correctly configured deployment for their use case without touching the original templates.

Like Kubernetes, Helm is declarative. You describe which chart to install and what values to pass in and Helm figures out how to get the cluster there and keeps it in that state.

Helm is built around three core concepts: Charts, Releases, and Repositories. We'll look at each of these next.

Key Concepts

Charts

A chart is a self-contained package of Kubernetes resource manifests. It defines everything needed to deploy an application (the Deployment, the Service, the ConfigMap, etc.) along with default values for any configurable settings. When you install a chart, Helm uses those templates and values to generate the actual Kubernetes manifests and applies them to your cluster.

Charts can also declare dependencies on other charts, called subcharts. This makes it possible to build composable application definitions, for example, a chart for a web application might pull in a chart for PostgreSQL and another for Redis rather than having manifests for those dependencies within it.

Releases

Each time you install a chart, Helm creates a release which is a named instance of that chart running in your cluster. If you install the same chart three times under different names, you get three independent releases, each with its own configuration, its own Kubernetes resources, and its own revision history.

It's important to understand that releases are what Helm actually manages. Helm doesn't think in terms of individual deployments or Helm charts when it comes to a Kubernetes cluster. Instead, it tracks releases and uses them to know what's deployed, what version it's at, and how to upgrade or roll back.

Repositories

A repository is a collection of charts hosted at an accessible URL. You add repositories to your local Helm installation and then install charts from them by name. The most widely used public index is Artifact Hub, which aggregates charts from many sources. Most projects that run on Kubernetes publish an official Helm chart there, so in practice you'll rarely need to write a chart from scratch to deploy well-known software like nginx, PostgreSQL, or Prometheus. For example, we also publish a chart for the mirrord Kubernetes operator, which you can find on Artifact Hub here.

What's Inside a Helm Chart

When you clone or create a Helm chart (using helm create chart-name), you get a directory with a specific structure. Here's what a typical chart looks like:

my-chart/
├── Chart.yaml
├── values.yaml
├── charts/
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ...
    └── NOTES.txt

Chart.yaml is the chart's metadata file. It declares the chart's name, version, and description, and optionally lists dependencies on other charts.

apiVersion: v2
name: my-chart
description: A Helm chart for my application
version: 0.1.0
appVersion: "1.0.0"

apiVersion here refers to the Helm chart API version, not the Kubernetes API. v2 means the chart requires at least Helm 3 (don’t ask me why they couldn’t have made it simple by just having v3 for Helm 3). If you come across a chart with v1, it was written for Helm 2, which had a different chart format but Helm 3 and later versions still support v1 for backward compatibility.

version is the version of the chart itself. This follows semantic versioning and should be bumped whenever you change the chart by doing things like adding a new template, changing a default value, or updating dependencies. It's independent of what the chart deploys.

appVersion is the version of the application the chart deploys. This is purely for documentation and nothing else since Helm doesn't use it to pull images or make any decisions. You'd typically set it to the container image tag your chart defaults to, so users can tell at a glance what version of the application they're getting. You can update appVersion without changing version, and vice versa.

A good way to think about it is that appVersion tracks the version of the application being deployed, while version tracks releases of the chart itself. If your application is upgraded from 1.0.0 to 1.1.0, you'd update appVersion to reflect that change. If the chart's default image tag is also updated, you'd typically bump the chart version as well, since the chart itself has changed. But if you only modify the chart (for example by adding a new template, changing default values, or updating dependencies) while continuing to deploy the same application version, you'd bump version and leave appVersion unchanged.

values.yaml contains the default configuration for the chart. These are the values users can override at install time. For example:

image:
  repository: nginx
  tag: "1.25"
replicaCount: 1

templates/ is where the Kubernetes manifests live. Unlike regular YAML, these files use Go template syntax, which lets Helm inject values into them at render time. A template referencing the values above would look like this:

spec:
  replicas: {{ .Values.replicaCount }}
  template:
    spec:
      containers:
        - name: app
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}

When someone installs this chart and passes --set image.tag=1.26, Helm substitutes that value before applying the manifest to the cluster. This is the mechanism that makes a single chart work across environments without touching the templates themselves.

charts/ is where subchart dependencies are bundled. If your chart depends on PostgreSQL, for example, the PostgreSQL chart will live in this folder.

NOTES.txt is a template file that Helm renders and prints to the terminal after a successful install. Charts typically use it to show how to connect to or use what was just deployed. This is what a simple NOTES.txt could look like:

Thanks for installing {{ .Chart.Name }}.

To get more info, try:

  $ helm status {{ .Release.Name }}
  $ helm get all {{ .Release.Name }}

Creating Your Own Chart

Running helm create my-chart scaffolds the directory structure we just walked through, with sensible defaults already filled in. The generated values.yaml comes pre-populated with common settings like replicaCount, image.repository, and image.tag, and templates/ includes a Deployment, Service, Ingress, and a few other Kubernetes resources you might need.

The first thing to do after generating the chart is update values.yaml to reflect your application. Change the image.repository to point to your container image, set a default tag, and remove any values that don't apply. The generated defaults are based on a generic nginx deployment, so most of them will need adjusting.

The templates themselves often need less work. The generated Deployment and Service are wired up to the values in values.yaml using the {{ .Values.* }} syntax we saw earlier, so once your values are correct, the templates usually render correctly without modification. If your application has specific Kubernetes resources it needs that aren’t present or configured in the templates/ directory already, then you'd add those or modify the existing relevant template.

Once you've made your changes, run helm lint my-chart to catch mistakes like invalid YAML, missing required fields, broken template syntax. Then run helm template my-chart to render the templates locally and review the exact Kubernetes manifests that would be applied to the cluster. This is the step where you verify your values are flowing into the right places before anything touches the cluster.

Common Helm Commands

Let's now walk through some common commands you’ll find yourself using when working with Helm. Before you can install a chart, you need to add the repository it lives in. helm repo add registers a repository under a local name:

helm repo add metalbear https://metalbear-co.github.io/charts

Once added, run helm repo update to pull the latest chart index from all registered repositories. Charts are versioned and updated frequently, so it's worth running this before installing. To find a chart:

helm search repo mirrord-operator

This returns matching charts with their versions and descriptions. You can also browse Artifact Hub to discover charts. Each listing there shows the helm repo add command you need.

helm install deploys a chart as a named release:

helm install mirrord-operator metalbear/mirrord-operator

Helm renders the chart's templates, applies the resulting manifests to the cluster, and begins tracking the deployment as a release named mirrord-operator. It also prints the chart's NOTES.txt which typically has instructions for what to do after installing the chart.

Note: The chart used in the example above installs the mirrord operator which allows multiple developers to develop against the same cluster when using mirrord.

To see all releases currently running in the cluster:

helm list

To inspect a specific release and see its status and revision number:

helm status mirrord-operator

Customizing with Values

By default, helm install uses the values defined in the chart's values.yaml. You can override them in two ways. For quick changes, use --set:

helm install mirrord-operator metalbear/mirrord-operator --set license.key=ABC-DEF-GHI

For environment-specific configuration, a values file is easier to manage. Create a custom-values.yaml with only the values you want to override, then pass it with -f:

helm install mirrord-operator metalbear/mirrord-operator -f values.yaml

When both are used together, --set takes precedence over -f, which takes precedence over the chart's values.yaml defaults. To see all the values a chart allows you to configure you can run this command:

helm show values metalbear/mirrord-operator

When you want to update a release to a newer chart version and/or change its values use helm upgrade:

helm upgrade mirrord-operator metalbear/mirrord-operator --set limits.cpu=400m

Helm applies only what changed, creates a new revision, and preserves the previous one. To see the full revision history:

helm history mirrord-operator

If an upgrade causes problems, helm rollback reverts a release to an earlier revision:

helm rollback mirrord-operator 1

It's important to understand that a rollback creates a new revision rather than deleting the failed one which means the history stays intact so you can always see what happened.

helm uninstall removes all Kubernetes resources that belong to a release:

helm uninstall mirrord-operator

By default, Helm also deletes the release's history. To keep it for auditing or future rollbacks, pass --keep-history.

Frequently Asked Helm Questions

Do I have to use Helm with Kubernetes?

You don't strictly need Helm. For a simple application that doesn't change much, managing it with raw kubectl apply and plain YAML manifests works fine. Where Helm becomes worth it is when you're managing multiple environments, deploying the same application more than once, or distributing your application for others to run. Then maintaining separate YAML files per environment costs more in terms of time and effort than just learning Helm.

How do I deploy a Helm Chart on Kubernetes?

First, add the repository the chart lives in and update your local index. Then install the chart with helm install, giving the release a name and pointing to the chart:

helm repo add metalbear https://metalbear-co.github.io/charts
helm repo update
helm install mirrord-operator metalbear/mirrord-operator

If the chart requires custom configuration, pass values with --set for quick overrides or -f your-values.yaml for a full config file. After installing, helm list shows all running releases and helm status <release-name> gives details on a specific one. The Common Helm Commands section above covers all of this in detail.

Next Steps

In this guide, we covered the fundamentals of Helm: charts, releases, and repositories, how chart structure works, and the commands you'll use day to day. These basics are enough to install and manage existing charts and start writing your own charts. If you want to go deeper from here, working toward a Helm certification like Developing Helm Charts (SC104) can be a good way to continue learning Helm. For this and other topics the official Helm documentation is the best place to go.

Or if you don’t want to prepare for a certification but want to learn how to make managing multiple charts across multiple environments easier, Helmfile is the natural next tool to look at. It lets you declaratively define a set of Helm releases as a single unit, with per-environment values and deployment ordering.

Wrote a new blog!

Arsh Sharma — Thu, 04 Sep 2025 10:54:59 +0000

Arsh Sharma for MetalBear

Sep 4 '25

New Features We Find Exciting in the Kubernetes 1.34 Release

#kubernetes #devops #cloud #docker

8 min read