Playwright + Docker: How I Set Up E2E Gating for Devset CE (and What Bit Me)

Before every merge to main I had this moment: "well, probably nothing broke." Unit tests green, integration tests green, I click around the UI manually — works. And still, that little voice in the back of my head: "did you check message dispatch after the last change in collectionContext? schema save from the Ace editor? SPA routing after the frontend gets bundled into the jar?"

Eventually I got tired of it. I opened a branch called ci-add-playwright-e2e-gating-main-with-bundled-FE+BE (yes, my branch names are a whole separate topic) and wrote proper E2E tests. Playwright, Docker, a gate on PRs. Here's how it looks — and where I faceplanted.

Quick context: what is Devset CE?

Devset CE is my source-available side project: a tool for working with Kafka and RabbitMQ — schema repository (JSON / protobuf), message workflows, dispatching, connection management. Architecturally it's a Spring Boot backend that serves its SPA frontend from inside a single jar, also shipped as a Docker image (ghcr.io/devset-io/devset-ce). That one-jar detail matters for this whole story: the E2E suite has to test that exact artifact, not a dev server.

What went in

• Playwright in devset-ce-fe/e2e/ — two spec files for now: smoke.spec.ts (the SPA mounts, /api/workflows returns 200) and schema-repo.spec.ts (create a JSON schema, create a protobuf one, edit the body, reload, verify it persisted).
• docker-compose.e2e.yml — Kafka, RabbitMQ, the backend with the frontend baked in, and a Playwright container. Healthchecks, pinned SHAs, depends_on with service_healthy. The whole package.
• scripts/e2e-build.sh — builds the frontend, drops it into devset-ce-be/src/main/resources/static, runs ./gradlew bootJar. Out comes a single artifact, exactly like the one behind ghcr.io/devset-io/devset-ce:latest.
• .github/workflows/e2e.yml — runs on PRs and on push to main. Cancel-in-progress for PRs (if someone pushes 10 commits in 5 minutes, I don't want 10 runs). Trace upload on failure.
• Four npm scripts: e2e:build, e2e:up, e2e:down, e2e:full. Locally I run npm run e2e:full and get exactly what CI gets. No more "works on my machine."

While I was at it, I deleted the old docker-compose.yml from the repo root. It was a zombie from when the project had a different structure — it pointed at a folder that no longer existed, and the README described a workaround for the workaround. Better to have nothing than something that lies.

Why Docker instead of Playwright's webServer

My first thought was the simplest one: Playwright has webServer in its config — spin up the backend with Gradle, Kafka via Testcontainers, done. I talked myself out of it within about 15 minutes:

The backend doesn't start "instantly." Spring Boot has to boot and get on speaking terms with Kafka and RabbitMQ — only then is /api/* actually alive. sleep 10 in the config? No, thanks. In docker-compose I have healthchecks and condition: service_healthy, and it just… happens on its own.

backend:
  depends_on:
    kafka:
      condition: service_healthy
    rabbitmq:
      condition: service_healthy

I want local and CI to be identical. Not 95% identical, not "well, almost." One compose file, one script, two places to run it. When something breaks in CI, I run npm run e2e:full on my machine and reproduce the failure in a minute. That's a currency you pay once — and it pays you back daily.

The browser version ships with Playwright. The mcr.microsoft.com/playwright:v1.59.0-noble image comes with a Chromium already matched to @playwright/test@1.59.0. Nobody forgets npx playwright install, nobody debugs "why does it work on Janek's machine."

And I'm testing the real artifact. Not a dev Vite server with a proxy to the backend, but the jar that serves the frontend from static/. That catches regressions in CORS, in SPA routing, in resource mapping. A mock server would never see them, because it simply never exercises those paths.

The cost? The first run is slower, but with npm/Gradle caches in GitHub Actions the whole workflow fits in about 6–8 minutes. Acceptable.

What bit me (and what surprised me)

A few things ate a meaningful chunk of my time. Writing them down so I have them handy next time.

The Ace editor. JSON schemas are edited in Ace. Ace renders its own DOM, keeps a hidden textarea, page.fill() is a coin flip, keyboard.type() is slow and keymap-sensitive. I started with three "normal" approaches — none of them was stable. Only after opening devtools and poking around Ace's internals did I notice it keeps its Editor instance at containerElement.env.editor (Ace uses it itself, as a re-entry guard in ace.edit()). So I go in through page.evaluate() and call editor.setValue(value, -1):

await page.evaluate((value) => {
  const container = document.querySelector('.ace_editor');
  (container as any).env.editor.setValue(value, -1);
}, schemaBody);

Ugly? Ugly. But it works every single time. I left a // SAFETY: comment so that future me, staring at the as unknown as, doesn't rip it out with a "what even is this."

SQLite chased me off. Devset keeps its state in SQLite. Playwright runs tests within a single file in parallel by default. SQLite serializes all writes — so when three tests fire a DELETE at once, two of them get SQLITE_BUSY and everything goes red. I tried retries, more aggressive cleanup, combinations of both — and finally let go:

test.describe.configure({ mode: 'serial' });

on the schema spec file. Slower, but 100% green. Retry with backoff masks the problem; it doesn't solve it.

getByRole does substring matching. So getByRole('button', { name: 'Edit' }) blew up with a strict-mode violation: "found 2 elements." Because the sidebar had a node whose ID contained the word "edit" — my edit test had named its schema e2e_json_edit_… (brilliant, past me). Lesson: exact: true everywhere I know the full text. It should be the default; it isn't; oh well.

Cleanup after failures. E2E tests against a shared backend have this property: when something dies halfway through, garbage stays behind. So every test appends the IDs it created to a list, and in afterEach I fire a Promise.all of best-effort DELETEs (.catch(() => undefined)). The next run always starts clean.

What I got out of it

• Merges to main are gated. Red E2E = it doesn't go in. No more "I trust nothing broke."
• Playwright traces on failure. I download the zip from the workflow artifacts, open it, and I've got a screenshot + recording + network log. Debugging a remote failure is five clicks now, not an hour-long investigation.
• I'm testing the same jar that ships to production. Frontend embedded in the backend, exactly like the GHCR image. If something works locally and breaks after a release, it's no longer the E2E suite's fault.
• Locally it's one command: npm run e2e:full. A new contributor: git clone, npm run e2e:full, goes to make a coffee, comes back to a result. The barrier to entry just dropped.

What's next

Smoke + schema repo is just the start. On the list: workflows (create / edit / run), message dispatch with collectionContext (a fresh feature — the perfect moment to cement it), connection management for Kafka and RabbitMQ. Every new screen gets its own spec — that's now part of "feature done," not an optional add-on for later.

If any of these tests turns flaky, that'll be its own post. Because a flaky test nobody fixes is worse than no test at all — it teaches the team to ignore a red status, and that's a habit that's very hard to unlearn.

---

Devset CE is source-available — you can poke around the code, run it from a single jar or the Docker image (ghcr.io/devset-io/devset-ce), and tell me everything I did wrong. Start at devset.pl or the devset-io org on GitHub. Issues, feedback and PRs very welcome.