DEV Community

suyaa
suyaa

Posted on

Django x Postgres x Selenium x Chromedriver with Docker

disclaimer: I am new to Docker so please take this with a grain of salt. at the time of writing, this setup worked on my computer.

also please let me know in the comments if I got something wrong

the files

These files assumes that your Dockerfile lives in the root of your Django project (i.e. in the same level as manage.py).

Dockerfile 

FROM python:3.9-slim-buster

# set environment variables
ENV PIP_DISABLE_PIP_VERSION_CHECK 1
# don't write .pyc files
ENV PYTHONDONTWRITEBYTECODE 1 
# prevent Docker from buffering stdout
ENV PYTHONUNBUFFERED 1 

# set working directory
WORKDIR /code

RUN apt-get update && apt-get install -y \
  curl \
  gnupg2 \
  unzip \
  wget

# chrome
RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' \
  && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
  && apt-get update \
  && apt-get install google-chrome-stable -y

# chromedriver
RUN CHROME_VERSION="$(google-chrome --version)" \
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \
  && CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  && cd /tmp \
  && unzip chromedriver_linux64.zip \
  && rm -rf chromedriver_linux64.zip \
  && mv chromedriver /usr/local/bin/chromedriver \
  && chmod +x /usr/local/bin/chromedriver \
  && cd /code

COPY ./requirements.txt .

# install dependencies
RUN apt-get update &&  apt-get install -y \
    gcc \ 
  less \
  libmagickwand-dev \
  libpq-dev \
  vim \
    && rm -rf /var/lib/apt/lists/* \
    &&  pip install -r requirements.txt

# copy project
COPY . .
Enter fullscreen mode Exit fullscreen mode

docker-compose.yml 

version: "3.9"
services:
  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    environment:
      - CHROME_DRIVER_PATH=/usr/local/bin/chromedriver
    volumes:
      - .:/code  
    ports:
      - "8000:8000"
    depends_on:
      - db

  db:
    image: postgres:13
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"

  redis:
    image: redis:alpine

  celery:
    build:
      context: .
    command: celery -A rises_ops worker -l INFO
    environment:
      - CHROME_DRIVER_PATH=/usr/local/bin/chromedriver
    volumes:
      # same volume as web
      - .:/code
    depends_on:
      - db
      - redis
      - web
volumes:
  postgres_data:
Enter fullscreen mode Exit fullscreen mode

migration script

after running docker compose up --build, run the migration from your terminal

$ docker-compose exec web python manage.py migrate
Enter fullscreen mode Exit fullscreen mode

(there is probably a smarter way to do this, let me know in the comments)

references

blogs that actually helped

https://learndjango.com/tutorials/django-docker-and-postgresql-tutorial
https://learndjango.com/tutorials/django-docker-and-postgresql-tutorial

debugging tips 

# build image with no cache
$ docker build . --no-cache

# output build process to your terminal
$ docker build . --progress=plain

# re-build your services
$ docker compose up --build

# clean cache
$ docker system prune
$ docker image prune
Enter fullscreen mode Exit fullscreen mode

note that the pg_data volume is actually saved in Host, so the DB data will persist even after you delete your conatiners. to clear even the DB,

# identify your volume
# format is project_db-data e.g. myproject_db-data
$ docker volume ls

# remove
$ docker volume rm <VOLUME ID>
Enter fullscreen mode Exit fullscreen mode

ref: https://torajirousan.hatenadiary.jp/entry/2020/07/08/094325

docker-compose.yml

the docker-compose.yml is pretty straightforward. we have 4 services: web db redis celery.

  • web: the Django werver
  • db: PostgreSQL DB
  • redis: Redis used in the Celery workers
  • celery: worker process for Django

the commands written in the command: section of the file should be similar to what you execute in your local terminal.

one thing to note is that, web and celery share the same volume, since both of them need access to the Django code

Dockerfile

the bulk of this file is to install chrome and chromedriver. both of them are not in the default apt-get repository, so you have to

  • add the repository for google-chrome-stable, and
  • download the zip file for chromedriver

in my past experience, Ubuntu has chromium and chromium-driver in the default repository, so this might be a better option if image size is not a problem for you.

chrome

to explain the process for installing Google Chrome,

# add the repository to the list
RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' \
Enter fullscreen mode Exit fullscreen mode 
  # add key for signed packages
  && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
Enter fullscreen mode Exit fullscreen mode 
  # don't forget to update the list after adding the repo
  && apt-get update \
  # finally install the package
  && apt-get install google-chrome-stable -y
Enter fullscreen mode Exit fullscreen mode

chromedriver

getting chromedriver is even more complex.

this is because

  • chrome and chromedriver needs to have compatible versions
  • BUT chrome and chromedriver versions are not always the same

to get the compatible versions, you have to first download the chrome, and get the compatible version from a web page.

let's go through this with an example.

at the time of writing (Sep. 2022), the latest version of google-chrome is 105.0.5195.102.

$ google-chrome --version
# Google Chrome 105.0.5195.102
Enter fullscreen mode Exit fullscreen mode

to get the corresponding chromedriver version, go to
https://chromedriver.storage.googleapis.com/LATEST_RELEASE_105

(if your chrome version was 80.x.x.x, you would instead go to https://chromedriver.storage.googleapis.com/LATEST_RELEASE_80 )

the page would show you that the corresponding chromedriver version is 105.0.5195.52

Dockerfile does this in the script by first getting the 105 part from the chrome version 105.0.5195.102.

# get chrome version
RUN CHROME_VERSION="$(google-chrome --version)" \
  # remove string "Google Chrome " from the output
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  # get the major version
  # `%%` removes the longest match of the following pattern `.*`
  # so it removes `.0.5195.52` and stores `105` in the variable
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \
Enter fullscreen mode Exit fullscreen mode

next, it accesses the LATEST_RELEASE_ page, and stores the response in the variable CHROMEDRIVER_VERSION.

&& CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
# 105.0.5195.52
# coincindentally, chrome & chromedriver versions match
Enter fullscreen mode Exit fullscreen mode

then, download the zip via curl --output, and unzip in the /tmp folder

  # specify output location to /tmp/chromedriver_linux64.zip 
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  # go to `/tmp`
  && cd /tmp \
  # unzip in `/tmp`
  && unzip chromedriver_linux64.zip \
Enter fullscreen mode Exit fullscreen mode

finally, we move chromedriver to /usr/local/bin and make sure chromedriver is executable

  # remove zip file because we don't need it anymore
  && rm -rf chromedriver_linux64.zip \
  # move executable to /usr/local/bin
  && mv chromedriver /usr/local/bin/chromedriver \
  # add permission
  && chmod +x /usr/local/bin/chromedriver \
Enter fullscreen mode Exit fullscreen mode

as a clean up step, we move back to the WORKDIR

  && cd /code
Enter fullscreen mode Exit fullscreen mode

put together, we have

RUN CHROME_VERSION="$(google-chrome --version)" \
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \
  && CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  && cd /tmp \
  && unzip chromedriver_linux64.zip \
  && rm -rf chromedriver_linux64.zip \
  && mv chromedriver /usr/local/bin/chromedriver \
  && chmod +x /usr/local/bin/chromedriver \
  && cd /code
Enter fullscreen mode Exit fullscreen mode

(same result as docker-compose.yml in the first section)

ref: https://qiita.com/from_host/items/a12d75368ece7bf5bcc1

Top comments (1)

Collapse
 
giyosiddin_muxammadiyev_ profile image
G'iyosiddin Muxammadiyev •

Perfect