Hardening Data Jobs: Non-Root Containers and Read-Only Filesystems

Why this matters: Data jobs often run with excessive privileges, increasing the blast radius if an application dependency is compromised. By restricting your containers to non-root users with read-only filesystems, you transform your runtime environment into a significantly harder target for unauthorized code execution.

You ship the job. It passes CI. Then, a library deep in your dependency tree triggers a remote code execution vulnerability, and because your process is running as root with a writable filesystem, the attacker gains persistence and the ability to modify local binaries. This is not a hypothetical risk; it is the default state of most unhardened containerized data pipelines.

Hardening a container for data engineering tasks—like running Spark executors or Python ETL scripts—requires a shift in how we handle temporary storage and process identity. You cannot simply flip a switch; you must reconcile your code’s need to write temporary files with the security posture of an immutable container.

Prerequisites

To follow this tutorial, you need the following:

• Docker Engine 20.10+ (for advanced security options like --read-only).
• A Linux-based base image (e.g., python:3.11-slim).
• Kubernetes or Docker Compose to manage runtime security constraints.
• An existing ETL job written in Python or PySpark that you want to harden.

Photo by Claudio Sanabria on Unsplash

Step 1: Defining the Non-Root User

The default behavior of a Dockerfile is to run processes as root (UID 0). You must explicitly create a system user and switch to it before the ENTRYPOINT.

# Use a specific, pinned version
FROM python:3.11.9-slim

# Create a group and user
RUN groupadd -r datauser && useradd -r -g datauser datauser

# Set working directory and change ownership
WORKDIR /app
RUN chown -R datauser:datauser /app

# Switch to the non-root user
USER datauser

# Copy application code
COPY --chown=datauser:datauser . .

ENTRYPOINT ["python", "main.py"]

By creating a system user with useradd -r, you ensure the UID is typically below 1000, which is standard for service accounts. The --chown flag in the COPY instruction is crucial; it ensures your application code is owned by the non-root user, preventing "permission denied" errors when the job starts.

Step 2: Preparing for a Read-Only Filesystem

Many data jobs fail when you enable a read-only filesystem because they attempt to write logs, temporary data, or cache files to default system paths (like /tmp or /var/cache).

You must identify where your application writes and redirect those paths to tmpfs mounts. For a standard Python job, these are usually /tmp and the application's internal cache directories.

Modify your Dockerfile to create these mount points owned by your user:

# Create directories for temporary data
RUN mkdir -p /tmp/data_job && chown -R datauser:datauser /tmp/data_job

# Set environment variables so libraries look here instead of system defaults
ENV TMPDIR=/tmp/data_job

When you deploy this container, you will map these directories to memory-backed volumes at runtime.

Step 3: Enforcing Security via Docker Compose

Once your image is built to run as a non-root user, you must enforce the read-only constraint at the orchestration level. In docker-compose.yaml, you use the read_only flag and explicitly mount tmpfs for the paths identified in Step 2.

services:
  data-pipeline:
    image: my-hardened-job:latest
    read_only: true
    user: "datauser"
    tmpfs:
      - /tmp
      - /tmp/data_job
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETUID
      - SETGID
    # Additional security: prevent privilege escalation
    security_opt:
      - no-new-privileges:true

The tmpfs directive tells the container runtime to mount a temporary, memory-backed filesystem at these paths. Even though the root filesystem is read-only, your job can write to these specific directories, and that data will vanish when the container stops—exactly what you want for transient data jobs.

Step 4: Testing for Compliance

Before deploying, verify that your container adheres to these constraints. You can run the container manually to inspect its capabilities and filesystem permissions:

# Run the container and drop into a shell to test
docker run --rm -it --read-only --tmpfs /tmp --user datauser my-hardened-job:latest /bin/bash

# Test that writing to root fails
touch /test.txt
# Expected: touch: cannot touch '/test.txt': Read-only file system

# Test that writing to tmp succeeds
touch /tmp/test.txt && ls /tmp/test.txt
# Expected: /tmp/test.txt

If your job requires specific Linux capabilities, avoid cap_drop: ALL. Instead, identify the specific capability needed (e.g., NET_BIND_SERVICE) and add only that. Use docker inspect on your container to verify that the ReadonlyRootfs field is set to true.

Photo by Tyler on Unsplash

Common pitfalls

• Hardcoded paths: Many third-party data libraries (like older versions of Pandas or certain JVM-based Spark wrappers) default to writing to /tmp or /usr/local/lib. If you cannot change the library's behavior, use environment variables (e.g., PYTHONPYCACHEPREFIX) to redirect these to your tmpfs mount points.
• Multi-stage build context: If you use a multi-stage build, ensure the final stage contains the non-root user configuration. It is common to forget the useradd command in the final stage, causing the container to default back to root.
• Log rotation: If your data job logs heavily to a file within the container, it will crash once the tmpfs is full. Ensure logs are redirected to stdout or stderr, which is standard practice for containerized applications, allowing the logging driver to handle persistence.
• Package Managers: If you need to install system dependencies during the build, do so in a stage prior to switching users. Once you switch to the non-root user, you lose the ability to use apt-get or yum effectively, which is actually a security feature—it prevents runtime package installation by an attacker.

Conclusion

Running data jobs as non-root with a read-only filesystem is an exercise in constraint management. By shifting the responsibility of file management to tmpfs and stripping the container of unnecessary privileges, you minimize the risk of your data pipeline becoming a pivot point for an attacker.

Try it: Take one of your existing Python-based Spark jobs, add a dedicated system user to the Dockerfile, and attempt to run it with the --read-only flag. Observe which directory paths throw "Permission Denied" errors and systematically move those paths to tmpfs mounts.

Cover photo by Rob Wingate on Unsplash.

---

This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.