DEV Community: Bubunyo Nyavor

Building a Multi-Tenant Dev Environment

Bubunyo Nyavor — Wed, 22 Jul 2026 13:55:00 +0000

[This article was original posted on my blog, https://www.5error.com/multi-tenant-dev-env/]

Every engineering org that grows eventually hits the same bottleneck: testing changes in a shared dev environment becomes unreliable very quickly. Everyone deploying to the same environment means everyone stepping on each other's data. When the frequency of deployments ticks up, the problem gets worse. With AI, people are able to create features faster, and it will continue to get worse. After each feature, you typically want to test a flow end to end, but someone else's test just mutated the state you depended on while you were testing. Such an environment earns a reputation for being unstable, and nobody trusts it.

This will typically spark demand for isolated environments, and it has happened where I work in multiple ways, on different forums. The ask is always the same. How do I spin up a full copy of the system, mutate it however I want, throw it away, and get a fresh one, without affecting anyone else?

Some context on what "the system" means here. We run a microservice architecture on Kubernetes, with multiple teams owning several services, each with a single delegated task. We also depend on a Postgres database, Kafka, a secrets vault and most things you would find on a modern cloud-native platform. Let's call the whole running system a world. No service in it works alone: any critical flow cuts across many of them. One of many critical flows is straight-through order processing : everything that happens between a customer clicking buy and the platform confirming the stock is theirs. A single order touches a good number of services end to end. If that path breaks, the business feels it immediately. And when an engineer says they want to test their change properly, this is usually the flow they mean, which makes it exactly the thing a shared environment is worst at.

Testing that flow in isolation is worth a lot, today. The first reason is smoke tests. A test that runs the chain end to end for every pull request, from any of the services involved, massively increases confidence in the change you are about to unleash. Without isolation, that confidence comes slowly: you deploy through shared stages and watch your work crawl through the pipeline before you learn anything. An isolated world collapses that loop to seconds, and that is a powerful thing a platform engineer can hand to product engineers. The second reason is load testing. You want to bootstrap a world, seed it with a particular scenario, and run a load test that targets a specific use case. If you do not seed a version of the world, your results rely on whatever data happens to be in place, and that comes with problems of its own. Runs stop being comparable, because the data shifted underneath them, and the accumulated junk looks nothing like production, so the numbers describe the leftovers from previous runs, rather than the system.

But the reasons go beyond what we can do today. The most valuable version of this tool is not one that runs the system as it is, but one that lets you inject your own changes, your branch, your local image, into a running world and test against everything else. That is a major hassle today, and it is exactly what a disposable world makes cheap. And one reason only became obvious recently: agents. Long-running agentic workflows are blocked by the same shared environments humans are. An agent that wants to validate its own changes needs a human in the loop before it can touch anything shared, and that human is the bottleneck. Give the agent a disposable world and the verification loop closes without anyone babysitting it. Agents need sandboxes even more than humans do.

So I set myself a deliberately idealistic goal. I should be able to run multiple parallel worlds, each involving many microservices, side by side, ideally on my laptop, sharing infrastructure but with fully isolated data planes. Data generated in world A must never leak into world B.

This post is about the four main problems that stood between me and that goal, and how I solved them.

Why now

Among other internal factors, one thing changed fundamentally: the cost of building tools like this has collapsed, and AI is the reason. It also helps that I have a pretty handy AI budget that makes experiments like this possible.

Think about the traditional economics of a project like this. An isolated environment tool touches Kubernetes, databases, secrets infrastructure, and Kafka internals. Under the old model that is a multi-quarter effort. Design documents, reviews, prototypes, more reviews, and months of iteration before anyone can touch a working version. Projects like this rarely die because they are bad ideas. They die because the upfront cost never clears the bar against everything else competing for engineering capacity. That is why demand can sit unanswered for years.

AI changed the shape of that cost curve. I sat down with an AI assistant, had it help reason through each implementation one after the other, and iterated. Try an approach to TLS termination, watch it fail, understand why, try the next one, all within an afternoon instead of a sprint. I never left the model to run amok or reduced it to result delivery. I was constantly the human in the loop, stubbornly intentional and deliberately slow about design choices. The decisions remained mine while the drafts got cheap. The expensive part of building software has always been the iteration loop: the time between having an idea and learning whether it survives contact with reality. AI compresses that loop brutally. A few weeks of this, and I had a working system spanning four domains that would each have been a project on their own.

An interesting consequence is not that a tool got built faster. It is that a whole class of projects that used to be economically unjustifiable is now viable. If there is an internal tool your team has wanted for years, the math may have changed. It is worth rerunning the numbers.

But I digress.

The core

The naive approach is to duplicate everything per world, just like we have in our current setup. Every database, every broker, every secret store, cloned for each environment. That is slow, ridiculously expensive, and does not scale.

So I flipped it on its head. Share all the infrastructure, and make every layer multi-tenant. One Postgres. One Kafka. One secrets endpoint. Each world gets a partitioned slice. This is to be a fresh, self-contained stack, stood up once per machine. The infrastructure can be as simple as docker-compose; the tool's job is to provide the rails for bootstrapping isolated worlds on top.

There is one constraint I fought hard to keep: services must not know any of this is happening. No code changes. No tenant awareness. No forcing teams to adopt a particular client library. If a service needs modification to run in a world, the design has failed. Adoption dies the moment you put homework on someone else's desk.

To that end, there were four main problems I had to solve: manifests, databases, secrets, and Kafka.

Manifests

Production Kubernetes manifests are built for high availability. Autoscalers , PodDisruptionBudgets , service mesh sidecars, observability agents, generous resource requests. None of that matters for a throwaway dev world, and most of it actively fights you in a local dev environment.

So I strip manifests to the bare bones. Autoscalers and disruption budgets are removed. Replica counts are forced to one. Resource requests are set to cluster defaults, so nobody gets to ask for twelve CPUs. The mesh sidecar goes. The observability agent goes, for now, though I suspect putting logs and metrics back in a dev-friendly form will be a real quality of life win later.

I set this up as a transformation pipeline. Source manifests go in one end, an ordered series of transformations runs over them, and stripped-down dev manifests come out the other. That decision paid for itself. A pipeline is testable and tested, each transformation is composable, and when a later problem needs something injected into every manifest, the injection is just one more stage. It shows up again twice below.

What is left is a deployment, a service, and the external secrets the service depends on. Deploy that into a local kind cluster and it works out of the box.

Postgres

In production, each service effectively gets its own managed Postgres instance, with backups, failover, and authenticated access. For local dev you need none of it, and duplicating an instance per service per world is a non-starter.

The nice thing is that Postgres is already multi-tenant by design, and it gives you two units of tenancy to choose from. The obvious first stop is schemas : one database, one schema per world, table names stay stable, and search_path does the routing. It was the first thing I tried, but that did not work. Some services ship migrations that point explicitly at the public schema, and that assumption runs deep. Fixing it means changing services, and changing services is against the rules.

So I went one level up. The unit of tenancy became the database name, which no service ever references in code ... yet. It arrives through configuration, which makes it the one knob I can turn without anyone noticing. I run one Postgres instance, and every world gets its own set of databases, prefixed with the world name. World A gets world-a.service-x. World B gets world-b.service-x. Both live side by side.

Services already take their connection settings from environment variables: host, user, password, database. Injecting the prefixed database name per world is just another stage in the manifest pipeline from above, and requires zero code changes. Two worlds on one instance, and neither is any the wiser.

Secrets

This was one of the harder problems. But in the end, the solution was incredibly obvious. Services fetch secrets from the cloud provider's Secret Manager, either through the SDK or through Kubernetes External Secrets . Both paths hit the same public endpoint, over TLS, authenticated with cloud identity.

You cannot point that at localhost. Or can you?

The trick was a three-layer fake. First, DNS. Inside the local cluster, we rewrite DNS so the Secret Manager hostname resolves to a secrets emulator instead of the real cloud API.

Second, TLS. The SDK expects a valid TLS handshake, and the emulator does not speak TLS. So the setup creates its own root certificate authority, appends it to the trust store of every pod it ships, as part of the pipeline, and uses it to issue a certificate for the Secret Manager hostname. Every pod now trusts the fake endpoint, because we made it so.

Third, gRPC. Secret fetching happens over gRPC, so after terminating TLS you forward the gRPC traffic to the emulator.

All of this is nginx configuration.

The service calls the same hostname it calls in production, gets a valid handshake, speaks gRPC, presents the same credentials it always presents, and receives its secrets. It never learns that everything behind the hostname is fake. As a bonus, that internal CA became a reusable building block. Once you can issue certificates that every pod trusts, you can impersonate any TLS endpoint you need to mock. It shows up again in the next section.

Kafka

Kafka is fundamentally not multi-tenant. There is one global topic namespace per cluster. Create topic-a and everyone on the cluster sees topic-a.

The official Kafka guidance is to prefix all topics with a tenant identifier and enforce it with policies. Fine, except our services are not tenant aware, and by my own constraint they are not allowed to become tenant aware. So something has to sit between the service and Kafka and do the tenancy transparently.

I had four requirements for that something. A single shared broker, with no tearing infrastructure down and up per tenant. Support for the actual Kafka wire protocol, not just produce and consume but metadata, offset commits, and transactional IDs. Light enough to run alongside everything else. And licensing that does not require a procurement conversation for an experiment.

I evaluated what the market had. Kroxylicious was the most viable option I found. It is open source and works as designed, but it runs on the JVM and kept getting heavy next to everything else. Conduktor Gateway is easier to use, but the features I actually needed sit behind an enterprise license, and an experiment is not the time for that conversation. The third option, making services tenant aware, is technically the correct answer and my least favorite. It forces an opinion on every team, requires everyone to adopt a specific client library, and means waiting for org-wide adoption before the tool works at all.

Having worked on some large-scale Kafka deployments in the past, and with some experience in encoding formats , I rolled up my sleeves and, with a touch of madness, wrote my own. I called it kroxy. It started as a weekend project, and is internal for now.

kroxy does one thing. It sits between the service and Kafka, figures out which tenant a connection belongs to, rewrites topic names with the tenant prefix on the way in, strips the prefix on the way out, and proxies everything else untouched. Transactional IDs get the same treatment, because Kafka transactions carry identifiers of their own. No ACLs, no permissions, no event hooks, none of the enterprise features the market proxies offer. Transparent tenancy rewriting at the protocol level, and nothing else.

When you start a world, a provisioner registers the tenant with kroxy and creates the prefixed topics on the real broker. Connection settings are injected as part of the manifest transformation pipeline, in the shape each service already expects. A service asks kroxy for some-topic. kroxy translates that to world-a.some-topic, talks to the shared broker, and translates back on the response. The service believes it is talking to a normal Kafka with its normal topics.

This is also code that cannot be quietly wrong. kroxy sits on the wire protocol, exercised by every client on every request. Implementing it carefully went a long way toward making bugs easy to track down and fix. Of which there were a few.

It worked.

Running it

I started on my laptop, and for a handful of services it was fine. But a full world in our setup is over a hundred services, and by the end it was clear the complete stack was too heavy for the machine I type on. You can run it. You will not enjoy it.

So I ported the implementation to a VM in Google Cloud with Packer and Terraform , and designed the split deliberately: your laptop remains the authoring surface, the VM becomes the backend. You write, the VM runs. The binary syncs over, and the whole stack lives there under systemd: kind for service container orchestration, plus containerized Kafka, Postgres, the secrets emulator, and kroxy, managed with nerdctl as a replacement for docker compose.

The end state is four commands. Two to build and start your VM, one to initialize the stack, one to spin up a world. After the first run, spinning up a new world is a single command.

This is v1. I foresee it evolving to a unified setup where a big cluster hosts all worlds and dedicated providers back all shared infrastructure. But for now, this works.

What I did not build

Scope discipline was half the project.

The obvious end game is PR preview environments. Open a pull request, get a world, see the URL in the PR. Everything above paves the road there, but that feature drags in CI, compute capacity, and provisioning questions that have not been answered yet. The aim was to test the idea by figuring out the building blocks first.

Data seeding is the other gap. An empty environment is not that useful. You need baseline data before you can exercise anything real. It is a real problem, it is on the roadmap, and I have intentionally not solved it yet.

And worlds are throwaway by design. So updating one is as simple as throwing what you currently have away and conjuring a new one, which always reflects the current state of the system. Throwaways also keep the blast radius honest: the worst any bug can do is break a world you were going to delete anyway. Migrating a lingering world forward is a convenience feature for later, if ever.

The takeaway

If you take away nothing else from this, remember: multi-tenancy beats duplication, and transparency is the design constraint that decides adoption. Every solution above, the database prefixes, the fake Secret Manager, kroxy, exists so that services stay completely unaware they are living in a partitioned world. And AI is making experiments like this cheaper to run.

The next time someone tells you an environment cannot be isolated without cloning everything, ask which layer is actually not multi-tenant. The answer is usually shorter than the excuse.

We are always looking for interesting people to solve interesting problems with. If that is you, Upvest is hiring.

Unfortunately, Redis

Bubunyo Nyavor — Tue, 21 Jan 2025 10:28:49 +0000

[This areticle was original posted on my blog, https://www.5error.com/redis]

Redis is designed to be a blazingly fast key-value in-memory database that trades some durability for speed. And for most parts, it lives up to the billing. Setting up and using Redis has always been a breeze. The success and popularity of Redis has made it a very powerful tool in the software engineers' arsenal; with its support of almost all data types and associated operations out of the box, it is a perfect solution for almost all persistence problems. As with everything, people have used it in ways without considering its fitness for the problem at hand, often to great success.

Redis is famed for being able to perform operations that is measured in sub milliseconds. And the key to this is how it is designed.
It is designed to store data in memory. If your first-year computer science class is anything to go by, you will know this provides much faster read and write throughput compared to disk by miles. When we create data structures using variables, we often use arrays, hashes and maps. Redis supports these including esoteric ones like hyperlog, bloom filters and more. It is also efficient at manipulating them because it takes advantage of the underlying data structures that are optimized for in-memory storage without worrying about how to persist it to durable storage. It is also single-threaded, Though a performant single-thread system might sound counterintuitive, there are some peculiar performant advantages to it. And Redis takes advantage of this in a brilliant way to ensure consistency without any cost to performance. Redis's single-thread will scale indefinitely in terms of I/O concurrency. It does this by using an I/O demultiplexing mechanism and a concise event loop designed by the author. Thus there is no synchronization to be done since all commands are serialized. It might look like the CPU might become a bottleneck with this design, but it turns out you will often hit a network bottleneck well before the CPU cannot keep up. The positive side effect of this design is that the atomicity of all operations comes at no extra cost. Redis also uses a proprietary protocol that is much more terse. Couple the isolated event loop with a proprietary communication protocol, and you have a blazing fast in-memory data store that scales indefinitely in theory, and you have a highly performant database.

These facts only hold when the size of your payload and the number of connections remain relatively small. This easily jumps out the window with ever-increasing load parameters. The threshold is, unfortunately, rather low at a high number of connections and increased payload sizes. Modern large-scale micro-services will easily have over 100 running instances at medium scale. And since most instances employ some pooling mechanism so as not to pay a connection cost for each request, a single Redis instance is going to do a bit of work in maintaining those connections, not to talk of serving requests as they come through. To improve performance at medium to high loads, some projects such as KeyDb, Snapchat's drop-in replacement as an alternative to Redis, employ a multithreaded approach and a bit of magic to sustain some high workloads. This has been touted to provide 5x performance over Redis. Another solution which I have seen used is to employ a proxy that multiplexes over multiple redises. One such proxy is the twoemproxy developed at Twitter. Twoemproxy, or nutcracker as it is informally known, is in itself single-threaded and employs key hashing to store keys in shards of multiple redis instances, giving you proper multiplexing. While it may look like this is susceptible to the original problems of a single-threaded application, it is not necessarily the reality because twoemproxy employs a single thread for each Redis instance, turning the whole system into a multi-threaded system. Of course, this is still susceptible to hotkeys. These might look like ideal solutions, but setting up new infrastructure as an intermediary service introduces a new failure point, which is neither trivial nor ideal. But when done right, there are a lot of net positives.

Another major concern with Redis is durability. Redis out of the box does not persist data on disk, only in memory. This means when a server goes down, so does all your data. Durability is serious business and when it becomes a priority, this is where Redis starts to go backwards. Redis was never planned to provide durability beyond RAM. This is evident in the fact that disk persistence was never part of Redis until v0.04. Redis supports two types of persistence modes.

The first one is called snapshotting or RDB. When snapshotting is enabled, Redis will periodically write all your dataset in memory to disk. This is good for point-in-time recovery. But this also means you lose all the data between when the last snapshot was created versus when the failure occurred. For a moderately busy server, there are bound to be significant changes that happen between when snapshots are set, and losing it might not be a good thing. To combat this, some teams set the snapshot times as minimal as possible to combat the amount of data it is possible to lose. This can be a bad idea when your data set is considerably large. Writing a 1gig file to disk every 60 seconds is a recipe for disaster. What happens in the background is Redis forks a child process for background processing, serializing the dataset in memory, making it disk compatible, writing it to a temporary file in the background and rename the file atomically upon finish. Even though the overhead of creating a fork is zero in theory when the OS supports copy-on-write, you still need to turn on the overcommit_memory. This is because if the dataset between the parent process and the child process deviates, Redis will not keep track of the changes and will have to allocate just as much memory your data set has to the child process in order for the snapshot to be successful. In snapshotting mode, you must do everything to ensure that your datastore does not exceed half the RAM allocated to Redis, otherwise, your Redis server will implode with an OOM. This is the default persistence mode because it is simple and safe for small data sets. When your data set starts to increase considerably in size, think twice about snapshots.

The second more durable persistence mode is Append Only File, introduced in Redis 1.1 to solve the drawbacks of snapshotting. In this mode, every Redis command is appended to a file as a log. Very much like WAL logs for conventional RDBMSs. This way, you can build the entire database by replaying the entire file. It can be argued that sequential writes to files are significantly faster than random access but is is still significantly slower than writing in memory, and this goes against the essence of what Redis is, which is an in-memory data store. If you are going to be writing each command to a file, why not use a datastore that is designed for that in the first place? If you use AOF, it means Redis is going to call fsync at a point in time which can be configured in 1 of 3 ways. appendfsync always will call fsync on each command. This is very very safe. But with this option, you might as well throw Redis out the window because the performance becomes inferior to every database designed to fsync on write. If you use this mode without a Battery Backed Write Cache (BBWC) RAID controller, you will get fucked. Have fun figuring out what went wrong. appendfsync everysec will call fsync every second, which means, at most you lose a second of data. This might sound reasonable, but it is not without its drawbacks. If you have an update-intensive application like a counter that updates many times in a second, you end up with a needlessly huge AOF file when in essence, the data payload might only have a small footprint. That said, it is an easier choice and has been the new default since Redis 2.4. appendfsync no delegates the calling of fsync to the operating system. This is the fastest and the least safe method amongst all the appendfsync options. Normally Linux will flush data every 30 seconds with this configuration, but it's up to the kernel's exact tuning.

When I talk to other developers about systems design problems, most people are quick to suggest Redis as a solution to buffer data when producers can not keep up with consumers, leaning heavily on the high write-throughput of Redis. The solution often goes like this; use Redis as a cache layer between the two entities. While this design generally works, I often wonder if people truly understand the tradeoffs and risks involved here. The lack of proper durability in Redis makes the bandwidth of solutions it is a fit for a very narrow one; nonetheless, for that bandwidth, it offers tremendous advantages. If you will drop Redis into your stack, be sure it is absolutely what you need, or stay clear of it. Otherwise, you and your data are going to get burnt.