Debugging a Legacy CRA + Django Deployment Pipeline: A DevOps Postmortem

A few weeks ago I was handed two deployment tasks that looked routine on paper: containerize and ship a React frontend, then do the same for its Django backend. Both apps were already running somewhere one on manage.py runserver, the other via a dev Dockerfile nobody had touched in years. "Just make it production-ready" is a deceptively small sentence. Here's what actually happened.

Part 1: The Frontend That Wasn't Next.js

The first Dockerfile I inherited looked like this at the runner stage:

FROM node:22-alpine AS runner
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
CMD ["node", "server.js"]

Clean multi-stage build, sensible layer caching, non-root-adjacent structure. One problem: the project was create-react-app via craco, not Next.js. .next/standalone doesn't exist in a CRA build there's no server to run. This was almost certainly a copy-paste from a Next.js project's Dockerfile that nobody adapted. The build would fail outright the moment it reached that COPY step.

Lesson one: before touching a Dockerfile, check what the build script actually produces. "build": "craco build" outputs a static build/ directory. No amount of Dockerfile cleverness fixes a mismatched deployment model you have to match the artifact, which meant swapping the runner stage entirely to an nginx static file server instead of a Node process.

The Node version trap

With the runner fixed, the next failure was a native module rebuild:

error Command "rebuild" not found.

The Dockerfile had yarn install --frozen-lockfile --ignore-scripts followed by yarn rebuild esbuild sharp except rebuild isn't a Yarn Classic command, it's an npm one. Someone had disabled install scripts (probably for build speed or a supply-chain concern) and then tried to manually force native binaries to compile, using syntax from the wrong package manager entirely.

The fix was almost too simple: drop --ignore-scripts, let yarn install run postinstall naturally, and delete the broken rebuild line. But underneath that surface bug was a nastier one the deps stage was building on node:24-alpine while builder/runner ran node:22-alpine. Native modules like sharp and esbuild compile against a specific Node ABI. Compile on 24, run on 22, and you risk a runtime crash that CI won't catch it only shows up when the container actually starts. Two completely different Node majors across stages had been quietly coexisting because --ignore-scripts was skipping the native compile step that would have caught the mismatch immediately.

Lesson two: every FROM node:X-alpine in a multi-stage build should agree on X, unless you have a very specific reason otherwise. Silent ABI mismatches are the kind of bug that passes CI and fails in production.

OpenSSL, then postcss, then Node itself

Once the build actually reached yarn build, it hit:

Error: error:0308010C:digital envelope routines::unsupported

react-scripts@4.0.3 bundles Webpack 4, which uses Node's legacy MD4 hashing internals removed by default once Node moved to OpenSSL 3 (Node 17+). The standard fix, NODE_OPTIONS=--openssl-legacy-provider, cleared that one.

Then a second, unrelated error surfaced:

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './lib/tokenize' is not defined by "exports"

This one traced to postcss-safe-parser, a transitive dependency of react-scripts@4's CSS minification chain, which ships its own nested, ancient copy of postcss. That old postcss's package.json never declared a wildcard exports field. Node 17+ enforces exports strictly and hard-errors on any path not explicitly declared. Node 16 only warned. Node 24 refused outright.

I initially reached for the obvious fix downgrade the build stage to Node 16, the last version before this became a hard error. It worked, but it was the wrong call to standardize on, and I was right to get pushback on it. Node 16 is EOL; picking it just because it dodges a rule isn't a fix, it's postponing the problem. The better answer, once I thought about who actually owns what: this is application dependency debt, not infrastructure. The real fix is a yarn.lock change a developer should make pinning the nested postcss via resolutions. But as the DevOps engineer, pulling the repo and editing package.json myself wasn't really my lane either.

The answer that stuck: patch the broken nested package.json from inside the Dockerfile itself, after yarn install, using a small inline Node script:

RUN node -e " \
  const fs = require('fs'); \
  const path = 'node_modules/postcss-safe-parser/node_modules/postcss/package.json'; \
  if (fs.existsSync(path)) { \
    const pkg = JSON.parse(fs.readFileSync(path)); \
    pkg.exports = pkg.exports || {}; \
    pkg.exports['./lib/*'] = './lib/*.js'; \
    fs.writeFileSync(path, JSON.stringify(pkg, null, 2)); \
  }"

No lockfile change, no dev involvement, Node stays on 24 across the fleet. It's explicitly a stopgap if the dependency tree shifts and postcss-safe-parser nests its postcss copy somewhere else, this silently stops applying and the original error resurfaces. That's a feature, not a bug: it fails loud and traceable rather than papering over a moving target indefinitely. I flagged the underlying issue to the dev team as a proper fix for later.

Lesson three: know which layer owns which fix. Downgrading infrastructure to dodge an application-level bug is a trap that's easy to fall into when you can fix it from the Dockerfile but "can" and "should" aren't the same question. Isolate the patch, keep it visible, and hand the real fix to whoever owns that code.

The scan that failed for a reason that had nothing to do with any of this

With the build green, Trivy failed the pipeline:

Total: 4 (HIGH: 4, CRITICAL: 0)
c-ares    CVE-2026-33630
libexpat  CVE-2026-56131 / 56407 / 56408

None of these were application dependencies they were OS packages baked into the nginx:alpine base image, stale relative to Alpine's own security advisories because the base image tag hadn't been rebuilt recently. The fix was a single line at the top of the runner stage:

RUN apk update && apk upgrade --no-cache

Lesson four: a clean base image today doesn't stay clean. Container security scanning isn't just about the app it's about everything shipped inside the image, and base image staleness is one of the most common, least glamorous sources of CVE noise in a pipeline.

Part 2: The Backend Nobody Had Documented

The Django API had never had a real production Dockerfile just a dev one, running manage.py runserver directly against production traffic, with a docker/prod/ directory that looked complete but had never actually been wired up.

Before writing anything, I had to reconstruct the actual setup by hand:

docker exec -it starsight-api-api-1 python --version   # 3.9.25
cat docker/dev/Dockerfile                                # base image, deps
cat docker/dev/entrypoint.sh                              # startup logic
cat app/requirements/*.txt | grep -i gunicorn              # already installed, unused
grep -rn "STATIC_ROOT" app/core/settings*.py               # static file config

This is worth calling out on its own: the existing docker/prod/entrypoint.sh had a bug that would have silently broken production had it ever shipped.

python manage.py migrate --no-input
python manage.py runserver 0.0.0.0:8000
python manage.py loaddata */fixtures/*.json
rm celerybeat.pid
exec "$@"

runserver is a blocking, foreground process it never returns. Every line after it, including exec "$@" (which should hand off to the actual container command, i.e. Gunicorn), would never execute. gunicorn==20.0.4 was sitting in prod.txt, fully installed, completely unused, because the entrypoint script never got past the dev server it was supposed to replace.

There was also a syntax bug in the other environment's entrypoint:

if ["$DATABASE" = "postgres"]

Missing a space after if POSIX shell needs [ "$DATABASE" = "postgres" ], since [ is itself a command. As written, this throws not found and silently skips the entire wait-for-database loop, meaning the app could start racing against a Postgres container that wasn't ready yet, with no visible error.

Neither of these bugs had ever caused a visible incident, because neither entrypoint had ever actually been exercised in a real production deploy. That's the uncomfortable part: untested infrastructure code doesn't fail loudly, it just sits there until the day it does.

Building it properly

The corrected entrypoint dropped the dev-only fixture loading and the blocking runserver call entirely, fixed the shell syntax, and left exactly one job wait for the database, apply already-committed migrations, then hand off to whatever the container's real command is:

#!/bin/sh
set -e

if [ "$DATABASE" = "postgres" ]; then
  while ! nc -z "$SQL_HOST" "$SQL_PORT"; do
    sleep 0.1
  done
fi

python manage.py migrate --no-input
exec "$@"

Notice what's not there: makemigrations. Auto-generating migrations at deploy time is a trap migrations should be written and reviewed in development, committed to the repo, and production should only ever apply what's already there. An entrypoint that can generate schema changes on the fly is an entrypoint that can silently diverge from what a reviewer actually approved.

The Dockerfile itself went through the same evolution as the frontend's first two stages (builder compiles wheels for native extensions like psycopg2 and Pillow, runtime installs only the shared libs those wheels actually need at runtime), then a third base stage once it became clear the ENV/WORKDIR lines were duplicated across both:

FROM python:3.9-slim-bookworm AS base
WORKDIR /app
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1

FROM base AS builder
# compiles wheels, discarded after build

FROM base AS runtime
# copies wheels, runs as non-root, Gunicorn as the actual process

This is worth being precise about: it's not a "GitOps best practice" GitOps is about Git as the source of truth for declarative infrastructure state, completely orthogonal to Dockerfile stage count. It's just good multi-stage design: a shared base stage means PYTHONUNBUFFERED or the workdir path only needs to change in one place, and both builder and runtime inherit it automatically instead of two copies that can silently drift out of sync.

The part that almost got skipped: the reverse proxy nobody remembered

docker/prod/ also contained a full jwilder/nginx-proxy setup its own Dockerfile, an nginx.conf, a vhost.d/default referencing static/media paths under /home/app/web/. None of those paths matched this project's actual STATIC_ROOT convention. That mismatch was the tell: this wasn't a working, tested config, it was a leftover from a boilerplate template, predating the team's move to Nginx Proxy Manager for reverse proxying everything externally. I excluded it entirely rather than trying to make broken paths correct the app just needed to expose a port for NPM to point at, same pattern as every other service on that droplet.

Protecting the thing that actually mattered

The Postgres container behind this API had 46 hours of live data by the time I got to it. The single most important constraint on the entire compose rewrite wasn't performance or elegance, it was: don't touch that volume.

db:
  image: postgres:12.1-alpine
  volumes:
    - postgres_data:/var/lib/postgresql/data

Docker Compose derives a named volume's actual identity from the project directory, not the compose filename so switching from docker-compose.yml to docker-compose.prod.yml in the same directory resolves to the same underlying volume automatically. I verified this explicitly rather than assuming:

docker volume ls | grep postgres_data
# starsight-api_postgres_data

The deploy step in CI also deliberately avoids a full docker compose down:

docker compose -f docker-compose.prod.yml up -d --no-deps api celery celery-beat

--no-deps means only the application containers get recreated on every push. db and redis are never touched by the automated pipeline at all. An automated deploy that can accidentally tear down stateful infrastructure on every merge is a liability waiting for a bad day, better to make that structurally impossible than to rely on remembering not to add a flag.

What Actually Mattered Here

None of these bugs were individually hard. A missing space in a shell conditional. A blocking dev server call. A Node ABI mismatch nobody would notice until runtime. A copy-pasted Dockerfile section from the wrong framework. What made this genuinely tricky was that almost none of it was visible until you went looking the existing docker/prod directory looked complete from a file listing. It had a Dockerfile, an entrypoint, an nginx setup. It just didn't work, and nothing had ever forced it to run.

The actual work of this pipeline rebuild wasn't writing Dockerfiles, it was reconstructing what was true, one cat and docker exec at a time, before writing a single line of infrastructure code. Every fix downstream of that was straightforward. Getting to a Dockerfile you can actually trust starts with refusing to guess at what's already there.