How I standardized PySpark deployments with multi-stage Docker builds

Why I chose this topic: I’ve spent too many 2:00 AM outages debugging "it works on my machine" failures caused by mismatched Python versions or missing shared libraries in PySpark executors. This pattern is the only way I’ve found to force consistency across the entire SDLC.

You ship the job. It passes CI. Then it hits the cluster, and the executor pods crash-loop with an ImportError because your local pandas version is 2.2.0, but the base image on your EMR cluster is pinning an ancient build of 1.3.5. You spend three hours SSHing into nodes or digging through CloudWatch logs, only to realize the environment variables are different, the LD_LIBRARY_PATH is missing, or someone updated a private package in Artifactory without telling you.

It’s a miserable loop. You’re managing infrastructure drift rather than writing data pipelines. We treat our application code like a first-class citizen, but we treat our execution environment like a neglected basement.

The real problem

The problem isn't your code; it’s the disconnect between the build-time environment and the runtime environment. Most PySpark deployments rely on "bootstrap scripts" or "init actions" to install dependencies on the fly. This is a recipe for disaster. Every node in your cluster tries to pip install simultaneously, resulting in network throttling, race conditions, or partial installs that fail midway through a 4-hour job.

If you aren't shipping a single, immutable container image that contains your OS, your Python runtime, your dependencies, and your job code, you aren't doing reproducible data engineering. You’re just gambling.

Photo by Bernd 📷 Dittrich on Unsplash

Step 1: Defining the build stage

We need to keep the final image slim. We don’t need compilers, C++ headers, or git credentials in the production image. We use a multi-stage build to isolate the "messy" build tools from the "clean" runtime.

I start with an official Python slim image. Why slim? Because alpine with PySpark is a nightmare of musl vs glibc incompatibilities that will eventually break your C-extensions like pyarrow.

# Stage 1: Build dependencies
FROM python:3.10-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libpq-dev \
    gcc \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt .

# Install to a local folder so we can copy it easily
RUN pip install --no-cache-dir --user -r requirements.txt

Step 2: Constructing the runtime stage

Now we move to the runtime stage. We copy only the installed packages from the builder stage. I prefer to keep the Python site-packages in a predictable location and ensure the PYTHONPATH is set correctly. Note that I am not including the job code here yet; I like to keep the environment static and the code dynamic (or baked in, if your CD pipeline allows).

# Stage 2: Final runtime
FROM python:3.10-slim

# Install runtime-only system dependencies (e.g., libpq for psycopg2)
RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq5 \
    && rm -rf /var/lib/apt/lists/*

# Copy installed packages from builder
COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local/bin:$PATH
ENV PYTHONPATH=/root/.local/lib/python3.10/site-packages:$PYTHONPATH

# Set up non-root user for security
RUN useradd -m sparkuser
USER sparkuser
WORKDIR /home/sparkuser/app

# Copy your source code
COPY --chown=sparkuser:sparkuser ./src ./src

Step 3: Integrating with Spark configuration

Once you have your image pushed to your container registry (ECR, GCR, or ACR), you need to tell Spark to actually use it. The configuration keys are non-negotiable. If you aren't running on K8s, the logic is similar for YARN, but Kubernetes is where this pattern truly shines.

When submitting your job via spark-submit or a K8s manifest, point directly to your image.

# Example spark-submit command
spark-submit \
  --master k8s://https://<k8s-api-server>:6443 \
  --deploy-mode cluster \
  --conf spark.kubernetes.container.image=your-registry/repo/pyspark-job:latest \
  --conf spark.kubernetes.container.image.pullPolicy=Always \
  --conf spark.executorEnv.PYTHONPATH=/home/sparkuser/app/src \
  local:///home/sparkuser/app/src/main.py

One subtle trap: spark.kubernetes.container.image.pullPolicy. If you tag images as latest (which you shouldn't do in production, but we all do in dev), set this to Always. If you use immutable tags like v1.2.3, set it to IfNotPresent.

Photo by BoliviaInteligente on Unsplash

Lessons learned from production

• The C-Extension Trap: If your job relies on pyarrow or pandas, verify that your Docker build is using the same glibc version as the base Spark image. If you mix and match, you will get obscure segmentation fault errors that appear only on 1 out of every 50 executors.
• Size Matters: If your image is over 2GB, your pod startup time will skyrocket. If you’re pulling 2GB per executor on a 100-node cluster, you are effectively performing a self-inflicted DDoS attack on your container registry. Keep it lean, or use a local pull-through cache.
• The User Problem: Spark containers often default to root. If your corporate security policy requires non-root users, you must explicitly set USER sparkuser in the Dockerfile and ensure your K8s SecurityContext doesn't conflict with that user ID.
• Dependency Locking: Always use pip-compile or poetry to generate a requirements.txt with hashes. A simple pip install without version pinning is a ticking time bomb. I’ve seen production jobs break because a sub-dependency released a "patch" that broke the Spark context initialization.

Conclusion

Standardizing on multi-stage builds isn't just about "best practices." It's about reclaiming your time. By defining the environment in Docker, you make the environment part of the code review process. If a developer needs a new library, they modify the Dockerfile or requirements.txt, which triggers a CI build that tests that change before it ever touches production. No more surprises, no more bootstrap scripts, and no more guessing why the job failed at 2:00 AM.

Try it: Take your most unstable PySpark job, write a multi-stage Dockerfile for it, and deploy it to a staging environment. Compare the startup time and the frequency of "environment-related" failures against your current deployment method. You won’t go back.

---

Tags: #docker #pyspark #dataengineering #devops

Cover photo by Jonas Smith on Unsplash.