Integration tests that actually touch Postgres and Redis (no testcontainers)

My unit tests were all green, and the bug I cared about most would have shipped anyway.

The gateway puts every tenant in its own Postgres schema, and it shares one Redis rate-limit budget across replicas. Both of those are things the database and Redis enforce, not my Go code. A fake store that's a map[string][]Note keeps two tenants apart because I wrote it to. It tells me nothing about whether SET LOCAL search_path really isolates them, because a map has no search_path to leak.

So this post is about the tests that put the fakes down and talk to a real Postgres and a real Redis. And since the title probably set an expectation: I didn't reach for the testcontainers library. I'll get to why at the end.

What a fake can never tell you

Four behaviors in this gateway live entirely in the real backend, and every one of them is a place a green fake would lie to me:

• Whether a per-schema advisory lock actually serializes eight replicas booting at once.
• Whether a failing migration rolls its whole schema back the way transactional DDL promises.
• Whether search_path really keeps plain unqualified SQL pointed at the right tenant.
• Whether Redis EXPIRE NX starts the window clock on the first request and leaves it alone after.

None of these is my logic. They're driver and server semantics. The only way to test them is to run them against the thing that implements them.

The gate is an env var, not a build tag

Every integration test starts the same way. If the backend URL isn't set, it skips:

func testPool(t *testing.T) *pgxpool.Pool {
    t.Helper()
    url := os.Getenv("TEST_DATABASE_URL")
    if url == "" {
        t.Skip("TEST_DATABASE_URL not set")
    }
    pool, err := db.Connect(context.Background(), url, 4)
    if err != nil {
        t.Fatalf("connect: %v", err)
    }
    t.Cleanup(pool.Close)
    return pool
}

The Redis test does the same on TEST_REDIS_URL. That's the whole split. Run go test ./... with no env vars and you get the fast unit suite, integration tests skipped. Export the URLs and the same command runs everything. No //go:build integration tag to remember, no second test binary. One knob.

Tenant IDs get a nanosecond suffix (fmt.Sprintf("conc_%d", time.Now().UnixNano())) so reruns against a long-lived database never trip over a stale schema or ledger, and t.Cleanup drops the registry row first, then the schema, so a concurrent MigrateAll can't pick the tenant up mid-drop.

Concurrency: does the advisory lock hold

The migration runner takes a transaction-scoped advisory lock keyed on the schema name (pg_advisory_xact_lock(hashtext($1))) before it reads the ledger and runs DDL. The claim is that without it, two runners both read an empty ledger, both run the create, and the loser dies on the ledger's primary key. A single-threaded fake can't disagree with that claim because it never races. So the test races on purpose:

    const runners = 8
    errs := make(chan error, runners)
    for i := 0; i < runners; i++ {
        go func() {
            errs <- EnsureTenant(ctx, pool, id, schema)
        }()
    }
    for i := 0; i < runners; i++ {
        if err := <-errs; err != nil {
            t.Fatalf("concurrent EnsureTenant: %v", err)
        }
    }

Eight goroutines register and migrate the same tenant at once, mimicking replicas booting during a rolling deploy. All eight must return nil, the notes table must exist, and the ledger row count has to equal the number of migrations, not eight times that. If the lock weren't there, this test flakes on timing luck, which is exactly the failure I want it to catch in CI instead of at 3am.

There's a companion test for the rollback promise: it applies a good migration set to schema A, a set with a deliberately broken SQL statement to schema B, then asserts A still has its table and B rolled back to version zero while keeping its provisioned-but-empty ledger. Postgres transactional DDL is doing that work. A fake would just do whatever my fake code said.

Isolation: the search_path is the only thing keeping tenants apart

The notes store writes plain, unqualified SQL. No schema prefix anywhere. The pinned search_path inside db.WithTenantTx is the entire isolation mechanism. So the test provisions two real tenants and checks that the same SELECT returns only the right rows:

    listA, err := store.List(ctx, schemaA)
    if err != nil {
        t.Fatalf("list %s: %v", schemaA, err)
    }
    if len(listA) != 1 || listA[0].Text != "from-a" || listA[0].UserID != "user-a" {
        t.Fatalf("tenant A notes = %+v, want exactly its own row", listA)
    }

Tenant A wrote from-a, tenant B wrote from-b, and each list has to come back with exactly one row that belongs to it. This is the test I trust least to a fake, because the fake's whole job is to be correct by construction. Here the correctness has to come out of Postgres resolving an unqualified table name against a search_path I set with a string. If I ever broke the pinning, a map-backed store would stay green and this one goes red.

Redis: a shared budget and a real TTL

Rate limiting is a fixed window backed by INCR plus EXPIRE NX, pipelined into one round trip. EXPIRE NX only sets the TTL when the key has none, so the first request in a window starts the clock and the rest leave it alone. That behavior needs Redis 7, which is what compose and CI pin. A fake counter would let me hardcode any TTL story I liked.

The test builds two independent fiber apps over one RedisCounter, standing in for two gateway replicas, spends the budget across both, then waits out the window and checks it comes back:

    // Wait out the window (TTL started at the first request) and the budget
    // must be fresh again.
    time.Sleep(window + 500*time.Millisecond)
    if got := hitReplica(replicaA); got != http.StatusOK {
        t.Fatalf("request after window reset = %d, want %d", got, http.StatusOK)
    }

Two requests on replica A and one on replica B spend a budget of three, even though no single replica saw more than two, and the fourth request is a 429 on either one. That "shared across replicas" property is the reason to use Redis at all, and the only way to see it is with a real Redis holding the key. One detail that bit me: app.Test defaults to a 1s timeout, so a slow Redis round trip flakes the test. Passing -1 disables it.

How CI runs it

CI is one job with two service containers and one test command:

      - run: go test -race ./...
        env:
          TEST_DATABASE_URL: postgres://postgres:postgres@localhost:5432/gateway?sslmode=disable
          TEST_REDIS_URL: redis://localhost:6379/0

GitHub Actions brings up postgres:16 and redis:7 as services with health checks, sets the two URLs, and runs the whole suite with the race detector on. Locally it's docker compose up -d against the same two images, export the same URLs, same go test. The unit tests that don't need a backend still run either way, unchanged.

So why not testcontainers

Because I already had the containers. The compose file exists for local development, CI has first-class service containers, and both the unit-vs-integration split and the cleanup were things I could do with os.Getenv, t.Skip, and t.Cleanup. Pulling in testcontainers would have added a dependency and a container-lifecycle-per-test to buy me something my env var already bought. If I were shipping a library other people run tests against, where I can't assume a compose file or a CI service block, I'd feel differently. For a gateway where I own the CI and the dev setup, the env-var gate is less code doing the same job, and less code is the one thing I never regret in a test suite.