Muhammad Ikramullah Khan

Posted on Jan 9

Scrapy with Docker: Deploy Anywhere in Minutes

#webdev #programming #docker #beginners

I spent 3 hours setting up Scrapy on a new server once. Installing Python, dependencies, libraries, system packages. Then it worked on my machine but broke on the server because of version differences.

The next time, I used Docker. Setup took 5 minutes. It worked identically on my laptop, the server, and my teammate's Windows machine.

Docker eliminates "works on my machine" problems forever. Let me show you how to Dockerize your Scrapy project properly.

What Is Docker and Why Use It?

Docker packages your entire application (code + dependencies + environment) into a container.

Benefits for Scrapy:

Same environment everywhere (laptop, server, cloud)
No dependency hell
Easy deployment
Isolation from host system
Version control for entire environment

Analogy: Shipping containers for code. Works the same wherever you ship it.

Installing Docker

Linux

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

Mac/Windows

Download Docker Desktop from docker.com

Verify installation:

docker --version

Creating a Dockerfile

This defines your container.

Basic Dockerfile

Create Dockerfile in your project root:

# Use official Python image
FROM python:3.11-slim

# Set working directory
WORKDIR /app

# Copy requirements first (for caching)
COPY requirements.txt .

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy project files
COPY . .

# Run spider
CMD ["scrapy", "crawl", "myspider"]

Build Image

docker build -t myspider .

Run Container

docker run myspider

That's it! Your spider runs in a container.

Better Dockerfile (Production Ready)

FROM python:3.11-slim

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    libxml2-dev \
    libxslt1-dev \
    && rm -rf /var/lib/apt/lists/*

# Create non-root user
RUN useradd -m -u 1000 scrapy

# Set working directory
WORKDIR /app

# Copy and install requirements as root
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy project files
COPY . .

# Change ownership
RUN chown -R scrapy:scrapy /app

# Switch to non-root user
USER scrapy

# Set environment variables
ENV PYTHONUNBUFFERED=1

# Run spider
CMD ["scrapy", "crawl", "myspider"]

What this adds:

System dependencies (for lxml, etc.)
Non-root user (security)
Unbuffered output (see logs immediately)
Proper cleanup (smaller image)

Docker Compose for Complete Setup

Docker Compose manages multiple containers.

docker-compose.yml

version: '3.8'

services:
  spider:
    build: .
    container_name: myspider
    environment:
      - DATABASE_URL=postgresql://postgres:password@db:5432/scrapy
      - LOG_LEVEL=INFO
    volumes:
      - ./data:/app/data
      - ./logs:/app/logs
    depends_on:
      - db
    restart: unless-stopped

  db:
    image: postgres:15
    container_name: scrapy_db
    environment:
      - POSTGRES_DB=scrapy
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  postgres_data:

Run with Compose

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f spider

# Stop
docker-compose down

# Rebuild and restart
docker-compose up -d --build

Passing Arguments to Spider

Method 1: Environment Variables

# Dockerfile
CMD ["scrapy", "crawl", "myspider"]

# docker-compose.yml
services:
  spider:
    environment:
      - START_PAGE=1
      - END_PAGE=100

# spider.py
import os

class MySpider(scrapy.Spider):
    name = 'myspider'

    def __init__(self):
        self.start_page = int(os.getenv('START_PAGE', 1))
        self.end_page = int(os.getenv('END_PAGE', 10))

Method 2: Command Override

docker run myspider scrapy crawl myspider -a start=1 -a end=100

Or with compose:

services:
  spider:
    command: scrapy crawl myspider -a start=1 -a end=100

Persistent Data with Volumes

Data inside containers disappears when container stops. Use volumes!

Named Volumes

services:
  spider:
    volumes:
      - scraped_data:/app/output
      - spider_logs:/app/logs

volumes:
  scraped_data:
  spider_logs:

Data persists even after container removal.

Bind Mounts (Local Folders)

services:
  spider:
    volumes:
      - ./output:/app/output  # Maps local ./output to container /app/output
      - ./logs:/app/logs

Now files appear in your local folders!

Multi-Stage Builds (Smaller Images)

Reduce image size by splitting build and runtime:

# Build stage
FROM python:3.11-slim as builder

WORKDIR /app

# Install build dependencies
RUN apt-get update && apt-get install -y gcc g++ libxml2-dev libxslt1-dev

# Install Python packages
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Runtime stage
FROM python:3.11-slim

# Install only runtime dependencies
RUN apt-get update && apt-get install -y libxml2 libxslt1.1 && rm -rf /var/lib/apt/lists/*

# Copy installed packages from builder
COPY --from=builder /root/.local /root/.local

# Add to PATH
ENV PATH=/root/.local/bin:$PATH

WORKDIR /app
COPY . .

CMD ["scrapy", "crawl", "myspider"]

Result: Much smaller final image!

Scheduling with Docker

Method 1: Cron in Container

FROM python:3.11-slim

# Install cron
RUN apt-get update && apt-get install -y cron

# Copy crontab
COPY crontab /etc/cron.d/spider-cron
RUN chmod 0644 /etc/cron.d/spider-cron
RUN crontab /etc/cron.d/spider-cron

WORKDIR /app
COPY . .

# Start cron
CMD ["cron", "-f"]

# crontab file
0 2 * * * cd /app && scrapy crawl myspider >> /var/log/cron.log 2>&1

Method 2: External Cron

# Host machine crontab
0 2 * * * docker run myspider

Method 3: Docker Compose with Restart Policy

services:
  spider:
    build: .
    restart: "no"  # Run once and stop

Then use host cron:

0 2 * * * cd /path/to/project && docker-compose up

Logging in Docker

View Container Logs

# Follow logs
docker logs -f myspider

# Last 100 lines
docker logs --tail 100 myspider

# With timestamps
docker logs -t myspider

Persist Logs to Host

services:
  spider:
    volumes:
      - ./logs:/app/logs

Now logs save to local ./logs/ folder.

Centralized Logging

services:
  spider:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Limits log size to prevent filling disk.

Environment-Specific Builds

Development Dockerfile

# Dockerfile.dev
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

# Install dev tools
RUN pip install ipdb pytest

COPY . .

# Enable debug logging
ENV LOG_LEVEL=DEBUG

CMD ["scrapy", "crawl", "myspider"]

Production Dockerfile

# Dockerfile.prod
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Production logging
ENV LOG_LEVEL=INFO

# Run as non-root
USER nobody

CMD ["scrapy", "crawl", "myspider"]

Build Specific Version

# Development
docker build -f Dockerfile.dev -t myspider:dev .

# Production
docker build -f Dockerfile.prod -t myspider:prod .

Scrapy + Selenium + Docker

For JavaScript-heavy sites:

FROM python:3.11-slim

# Install Chrome and ChromeDriver
RUN apt-get update && apt-get install -y \
    wget \
    gnupg \
    unzip \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list \
    && apt-get update \
    && apt-get install -y google-chrome-stable \
    && rm -rf /var/lib/apt/lists/*

# Install ChromeDriver
RUN wget -q "https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip" \
    && unzip chromedriver_linux64.zip \
    && mv chromedriver /usr/local/bin/ \
    && chmod +x /usr/local/bin/chromedriver

WORKDIR /app

COPY requirements.txt .
RUN pip install scrapy scrapy-selenium

COPY . .

CMD ["scrapy", "crawl", "myspider"]

Scrapy + Playwright + Docker

Modern JavaScript rendering:

FROM python:3.11-slim

# Install dependencies for Playwright
RUN apt-get update && apt-get install -y \
    libnss3 \
    libatk-bridge2.0-0 \
    libdrm2 \
    libxkbcommon0 \
    libgbm1 \
    libasound2 \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt .
RUN pip install scrapy scrapy-playwright
RUN playwright install chromium
RUN playwright install-deps chromium

COPY . .

CMD ["scrapy", "crawl", "myspider"]

Docker Networking

Connect Multiple Spiders

services:
  spider1:
    build: .
    command: scrapy crawl spider1
    networks:
      - scrapy-network

  spider2:
    build: .
    command: scrapy crawl spider2
    networks:
      - scrapy-network

  db:
    image: postgres:15
    networks:
      - scrapy-network

networks:
  scrapy-network:
    driver: bridge

All services can communicate on scrapy-network.

Health Checks

Monitor container health:

FROM python:3.11-slim

WORKDIR /app
COPY . .

# Add health check script
COPY healthcheck.py .

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD python healthcheck.py || exit 1

CMD ["scrapy", "crawl", "myspider"]

# healthcheck.py
import sys
import os

# Check if critical files exist
if not os.path.exists('/app/data/output.json'):
    sys.exit(1)

# Check file age
import time
age = time.time() - os.path.getmtime('/app/data/output.json')
if age > 3600:  # More than 1 hour old
    sys.exit(1)

sys.exit(0)  # Healthy

Optimizing Docker Images

Use .dockerignore

Create .dockerignore:

__pycache__/
*.pyc
*.pyo
*.pyd
.git/
.gitignore
.venv/
venv/
*.log
.env
.scrapy/

Excludes unnecessary files from image.

Layer Caching

# Good (requirements cached separately)
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

# Bad (re-installs on any code change)
COPY . .
RUN pip install -r requirements.txt

Use Slim Images

# Large (1 GB)
FROM python:3.11

# Smaller (150 MB)
FROM python:3.11-slim

# Smallest (50 MB, but missing some tools)
FROM python:3.11-alpine

Complete Production Example

Here's everything together:

Dockerfile:

FROM python:3.11-slim

RUN apt-get update && apt-get install -y \
    gcc g++ libxml2-dev libxslt1-dev \
    && rm -rf /var/lib/apt/lists/*

RUN useradd -m -u 1000 scrapy

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .
RUN chown -R scrapy:scrapy /app

USER scrapy

ENV PYTHONUNBUFFERED=1

HEALTHCHECK --interval=60s --timeout=5s \
  CMD test -f /app/logs/spider.log || exit 1

CMD ["scrapy", "crawl", "myspider"]

docker-compose.yml:

version: '3.8'

services:
  spider:
    build: .
    container_name: production_spider
    environment:
      - DATABASE_URL=postgresql://postgres:${DB_PASSWORD}@db:5432/scrapy
      - LOG_LEVEL=INFO
      - USER_AGENT=MyBot/1.0
    volumes:
      - ./data:/app/data
      - ./logs:/app/logs
    depends_on:
      - db
    restart: unless-stopped
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    networks:
      - scrapy_network

  db:
    image: postgres:15
    container_name: scrapy_postgres
    environment:
      - POSTGRES_DB=scrapy
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - scrapy_network
    restart: unless-stopped

volumes:
  postgres_data:

networks:
  scrapy_network:
    driver: bridge

.env:

DB_PASSWORD=your_secure_password

Deploy:

docker-compose up -d
docker-compose logs -f spider

Troubleshooting Docker Issues

Container Exits Immediately

Check logs:

docker logs myspider

Common causes:

Command syntax error
Missing dependencies
Python import errors

Can't Connect to Database

Check network:

docker network ls
docker inspect scrapy_network

Use service name (not localhost):

# WRONG
DATABASE_URL = 'postgresql://localhost:5432/db'

# RIGHT (in Docker)
DATABASE_URL = 'postgresql://db:5432/db'

Permission Denied

Run as correct user:

USER scrapy

Or fix ownership:

RUN chown -R scrapy:scrapy /app

Summary

Why Docker:

Consistent environment everywhere
Easy deployment
Isolated from host
Version controlled environment

Basic workflow:

Create Dockerfile
Build image: docker build -t myspider .
Run container: docker run myspider

Production setup:

Multi-stage builds (smaller images)
Non-root user (security)
Volume mounts (persist data)
Docker Compose (manage services)
Health checks (monitor status)

Best practices:

Use .dockerignore
Layer caching (requirements first)
Slim base images
Environment variables for config
Volume mounts for data/logs

Remember:

Service names for networking (not localhost)
Volumes for persistent data
Logs to stdout or volume
Health checks for monitoring

Start with basic Dockerfile, then add Docker Compose as you need databases or multiple services!

Happy scraping! 🕷️