DEV Community: Othmane

I made a Redis cache fast enough to hide a bug

Othmane — Thu, 24 Sep 2026 19:27:30 +0000

I made a Redis cache fast enough to hide a bug.

The application was a small bookstore. Every uncached request deliberately spent 1.5 seconds in PostgreSQL before returning the catalog.

Then I added cache-aside with a 180-second TTL.

The next request was fast. I inserted a new book into PostgreSQL, refreshed the page, and the book was still missing. The database was correct. Every container was healthy. The customer was reading the old catalog.

I built that failure into a Torollo roadmap on purpose.

Torollo cache-aside walkthrough: the HTTP check fails because Redis serves a stale catalog, DEL cache:books invalidates the entry, and the repeated check finds The Phoenix Project." width="720" height="405">

The cache-aside path

The application checks Redis before touching PostgreSQL:

CACHE_KEY = "cache:books"
CACHE_TTL = 180

body = cache.get(CACHE_KEY)

if body is None:
    cur.execute("SELECT pg_sleep(1.5);")
    cur.execute("SELECT title, author FROM books ORDER BY id;")
    rows = cur.fetchall()

    body = render_catalog(rows)
    cache.setex(CACHE_KEY, CACHE_TTL, body)

The first request misses the cache, queries PostgreSQL and stores the rendered catalog.

The next request finds cache:books and returns it without running the database query again.

That is the part every cache-aside diagram explains. The more interesting part begins when the database changes.

The infrastructure was healthy

I added a book directly to PostgreSQL:

INSERT INTO books (title, author)
VALUES ('The Phoenix Project', 'Gene Kim');

PostgreSQL now contained the new row. Redis still contained the HTML generated before the insert.

Refreshing the store returned the cached HTML, so the new book remained invisible until one of two things happened:

The 180-second TTL expired

or:

DEL cache:books

Redis was not malfunctioning. It returned exactly the value the application had stored.

The failure was in the invalidation strategy.

Checking the configuration would miss the bug

Torollo runs the web server, PostgreSQL and Redis as real Docker containers.

A weak validator could check that:

the Redis container is running;
cache:books exists;
the key has a TTL.

Every one of those checks would pass while the application displayed stale data.

The roadmap instead sends an HTTP request to the running web application and checks the response:

{
  "type": "http_get_contains",
  "params": {
    "node": "web",
    "port": 80,
    "path": "/",
    "expectedText": "Phoenix"
  }
}

The validation fails because the user-visible result is wrong, even though the infrastructure looks healthy.

After deleting cache:books, the next request goes back to PostgreSQL, rebuilds the cached value and includes the new book. A second request confirms that Redis is serving the corrected catalog.

Availability and freshness are different failures

The application also treats Redis as an optimization.

Redis calls have short timeouts. If the cache becomes unavailable, the application falls back to PostgreSQL. The page becomes slower, but it can still respond.

That protects availability. It does not protect freshness.

A healthy cache can return stale data much faster than a database can return the correct data. Monitoring container health or request latency alone will not expose that mistake.

The behavior has to be tested.

Reproduce the failure locally

Torollo is an open-source system-design lab backed by real local containers.

Run it with:

npx torollo start

Open Learning, choose Cache-aside with Redis, and follow the exercise until the stale catalog step.

The source is available on GitHub:

https://github.com/Derssa/Torollo

When a database write changes several cached views, where do you perform invalidation in your systems: inside the write path, through an event consumer, or by accepting a bounded TTL window?

I wanted a system-design lab that could prove me wrong

Othmane — Sat, 12 Sep 2026 12:57:23 +0000

When I practice system design, the part that has always felt a little unsatisfying is that the diagram usually gets the final say.

You can draw a load balancer, put PostgreSQL behind your application servers, add a private subnet, and end up with something that looks perfectly reasonable.

But the diagram itself never proves that PostgreSQL is actually private.

That is the idea I kept coming back to while building Torollo:

What if the diagram was only the intended architecture, and the running system was the thing that decided whether you were right?

That became the core of the project.

The architecture actually runs

Torollo is a local system-design lab.

You build an architecture visually, but the nodes on the canvas are backed by real Docker resources on your machine.

If you add a server, there is a container behind it.

If an exercise uses PostgreSQL, PostgreSQL is actually running.

If the architecture introduces network boundaries, Torollo has real Docker networking underneath them.

The roadmap then validates the environment that exists instead of checking whether your diagram looks correct.

One of the free roadmaps, for example, has you build a resilient three-tier application.

You start with a web server and a database, then progressively introduce:

a load balancer
multiple web servers
network restrictions
failure scenarios

At one point, PostgreSQL is supposed to be isolated from the load balancer.

Torollo actually checks that.

If the load balancer can still reach PostgreSQL on port 5432, the step fails.

You change the configuration, run the validator again, and the step only turns green when that connection is really blocked.

That loop ended up becoming the part of Torollo I care about the most:

Build something → make an assumption → let the runtime prove whether the assumption is true.

The canvas is only one layer

The frontend is built with:

React
TypeScript
React Flow

The backend uses:

Node.js
Express
TypeScript
Dockerode

But I try to keep a strict separation between what the canvas says exists and what Torollo can actually observe.

If the frontend says a server exists, that is not enough for a validator.

The validator can inspect whether the container is actually running.

If an architecture says an application can reach its database, Torollo can test that connection.

If a roadmap asks you to create a table, the validator can inspect PostgreSQL.

If the exercise uses Redis, it can inspect actual Redis state.

That was an important design decision for me, because otherwise Torollo would just become another graph editor with a green checkmark attached to it.

The runtime has to be able to disagree with the diagram.

Docker networking made this much harder

The container lifecycle was relatively straightforward.

Networking was not.

Some Torollo labs use Docker bridge networks, routing, NAT and iptables rules to make the architecture behave differently depending on how you configure it.

That is where I started running into the fact that "runs on Docker" does not mean every Docker environment behaves identically.

Rootless Docker is a good example.

It can run the containers, but some of the advanced networking behavior Torollo uses requires capabilities that a rootless daemon cannot provide in the same way.

Initially, I wanted to hide details like that from the learner.

I eventually decided that was the wrong abstraction.

Torollo is specifically trying to make infrastructure behavior tangible. Pretending that the underlying environment does not matter would work against that.

So the CLI now includes:

npx torollo doctor

It checks the Docker environment, ports and some runtime requirements and tries to explain what is wrong before you spend time debugging an exercise that your machine cannot execute properly.

That feature came directly from repeatedly debugging what looked like Torollo failures and discovering that the actual problem was the environment.

I’m not trying to recreate AWS locally

This is probably the easiest trap for a project like Torollo to fall into.

Once you have containers, databases, load balancing, queues, caching and networking, there is always another infrastructure primitive you could implement.

But I don’t want Torollo to become a fake cloud provider.

The goal is much narrower:

Take an architecture concept and make enough of it real that the learner can be wrong about it.

If you say a database is isolated, there should be a way to test that claim.

If you introduce a cache, it should contain real state.

If a worker is supposed to process a queue, there should be an actual worker and queue involved.

If a failure is part of the exercise, something should really fail.

The local environment is there to make the consequences of the architecture visible, not to pretend that Docker on a laptop is equivalent to production infrastructure.

What Torollo includes right now

The current public version has three free roadmaps:

a resilient three-tier application
Redis cache-aside
Redis queue workers

The roadmap format is JSON, so the exercises are not hardcoded into the application.

The core is MIT licensed.

It runs locally.

There is no account requirement.

You currently need:

Node 18+
Docker

Repository:

https://github.com/Derssa/Torollo

Three-tier roadmap:

https://torollo.app/r/devto

The question I’m trying to answer next

I think making the runtime real improves some parts of learning system design.

"PostgreSQL is private" stops being something written next to a box and becomes a statement that can actually be false.

But there is obviously a limit.

A local Docker environment does not reproduce IAM, managed services, cloud control planes, real latency, production observability, regional failures and all the other things that shape real systems.

So the question I’m trying to understand now is:

Where does making the runtime real improve the way you learn system design, and where does it start creating the wrong mental model?

If you work with backend systems, Docker, DevOps, SRE or infrastructure, that is the feedback I would be most interested in.