DEV Community: suyaa

Django x Postgres x Selenium x Chromedriver with Docker

suyaa — Thu, 08 Sep 2022 16:23:55 +0000

disclaimer: I am new to Docker so please take this with a grain of salt. at the time of writing, this setup worked on my computer.

also please let me know in the comments if I got something wrong

the files

These files assumes that your Dockerfile lives in the root of your Django project (i.e. in the same level as manage.py).

Dockerfile

FROM python:3.9-slim-buster

# set environment variables
ENV PIP_DISABLE_PIP_VERSION_CHECK 1
# don't write .pyc files
ENV PYTHONDONTWRITEBYTECODE 1 
# prevent Docker from buffering stdout
ENV PYTHONUNBUFFERED 1 

# set working directory
WORKDIR /code

RUN apt-get update && apt-get install -y \
  curl \
  gnupg2 \
  unzip \
  wget

# chrome
RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' \
  && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
  && apt-get update \
  && apt-get install google-chrome-stable -y

# chromedriver
RUN CHROME_VERSION="$(google-chrome --version)" \
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \
  && CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  && cd /tmp \
  && unzip chromedriver_linux64.zip \
  && rm -rf chromedriver_linux64.zip \
  && mv chromedriver /usr/local/bin/chromedriver \
  && chmod +x /usr/local/bin/chromedriver \
  && cd /code

COPY ./requirements.txt .

# install dependencies
RUN apt-get update &&  apt-get install -y \
    gcc \ 
  less \
  libmagickwand-dev \
  libpq-dev \
  vim \
    && rm -rf /var/lib/apt/lists/* \
    &&  pip install -r requirements.txt

# copy project
COPY . .

docker-compose.yml

version: "3.9"
services:
  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    environment:
      - CHROME_DRIVER_PATH=/usr/local/bin/chromedriver
    volumes:
      - .:/code  
    ports:
      - "8000:8000"
    depends_on:
      - db

  db:
    image: postgres:13
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    environment:
      - "POSTGRES_HOST_AUTH_METHOD=trust"

  redis:
    image: redis:alpine

  celery:
    build:
      context: .
    command: celery -A rises_ops worker -l INFO
    environment:
      - CHROME_DRIVER_PATH=/usr/local/bin/chromedriver
    volumes:
      # same volume as web
      - .:/code
    depends_on:
      - db
      - redis
      - web
volumes:
  postgres_data:

migration script

after running docker compose up --build, run the migration from your terminal

$ docker-compose exec web python manage.py migrate

(there is probably a smarter way to do this, let me know in the comments)

references

blogs that actually helped

https://learndjango.com/tutorials/django-docker-and-postgresql-tutorial
https://learndjango.com/tutorials/django-docker-and-postgresql-tutorial

debugging tips

# build image with no cache
$ docker build . --no-cache

# output build process to your terminal
$ docker build . --progress=plain

# re-build your services
$ docker compose up --build

# clean cache
$ docker system prune
$ docker image prune

note that the pg_data volume is actually saved in Host, so the DB data will persist even after you delete your conatiners. to clear even the DB,

# identify your volume
# format is project_db-data e.g. myproject_db-data
$ docker volume ls

# remove
$ docker volume rm <VOLUME ID>

ref: https://torajirousan.hatenadiary.jp/entry/2020/07/08/094325

`docker-compose.yml`

the docker-compose.yml is pretty straightforward. we have 4 services: web db redis celery.

web: the Django werver
db: PostgreSQL DB
redis: Redis used in the Celery workers
celery: worker process for Django

the commands written in the command: section of the file should be similar to what you execute in your local terminal.

one thing to note is that, web and celery share the same volume, since both of them need access to the Django code

`Dockerfile`

the bulk of this file is to install chrome and chromedriver. both of them are not in the default apt-get repository, so you have to

add the repository for google-chrome-stable, and
download the zip file for chromedriver

in my past experience, Ubuntu has chromium and chromium-driver in the default repository, so this might be a better option if image size is not a problem for you.

chrome

to explain the process for installing Google Chrome,

# add the repository to the list
RUN sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' \

  # add key for signed packages
  && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \

  # don't forget to update the list after adding the repo
  && apt-get update \
  # finally install the package
  && apt-get install google-chrome-stable -y

chromedriver

getting chromedriver is even more complex.

this is because

chrome and chromedriver needs to have compatible versions
BUT chrome and chromedriver versions are not always the same

to get the compatible versions, you have to first download the chrome, and get the compatible version from a web page.

let's go through this with an example.

at the time of writing (Sep. 2022), the latest version of google-chrome is 105.0.5195.102.

$ google-chrome --version
# Google Chrome 105.0.5195.102

to get the corresponding chromedriver version, go to
https://chromedriver.storage.googleapis.com/LATEST_RELEASE_105

(if your chrome version was 80.x.x.x, you would instead go to https://chromedriver.storage.googleapis.com/LATEST_RELEASE_80 )

the page would show you that the corresponding chromedriver version is 105.0.5195.52

Dockerfile does this in the script by first getting the 105 part from the chrome version 105.0.5195.102.

# get chrome version
RUN CHROME_VERSION="$(google-chrome --version)" \
  # remove string "Google Chrome " from the output
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  # get the major version
  # `%%` removes the longest match of the following pattern `.*`
  # so it removes `.0.5195.52` and stores `105` in the variable
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \

next, it accesses the LATEST_RELEASE_ page, and stores the response in the variable CHROMEDRIVER_VERSION.

&& CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
# 105.0.5195.52
# coincindentally, chrome & chromedriver versions match

then, download the zip via curl --output, and unzip in the /tmp folder

  # specify output location to /tmp/chromedriver_linux64.zip 
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  # go to `/tmp`
  && cd /tmp \
  # unzip in `/tmp`
  && unzip chromedriver_linux64.zip \

finally, we move chromedriver to /usr/local/bin and make sure chromedriver is executable

  # remove zip file because we don't need it anymore
  && rm -rf chromedriver_linux64.zip \
  # move executable to /usr/local/bin
  && mv chromedriver /usr/local/bin/chromedriver \
  # add permission
  && chmod +x /usr/local/bin/chromedriver \

as a clean up step, we move back to the WORKDIR

  && cd /code

put together, we have

RUN CHROME_VERSION="$(google-chrome --version)" \
  && export CHROMEDRIVER_RELEASE="$(echo $CHROME_VERSION | sed 's/^Google Chrome //')"  \
  && export CHROMEDRIVER_RELEASE=${CHROMEDRIVER_RELEASE%%.*} \
  && CHROMEDRIVER_VERSION=$(curl http://chromedriver.storage.googleapis.com/LATEST_RELEASE_${CHROMEDRIVER_RELEASE}) \
  && curl --output /tmp/chromedriver_linux64.zip "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" \
  && cd /tmp \
  && unzip chromedriver_linux64.zip \
  && rm -rf chromedriver_linux64.zip \
  && mv chromedriver /usr/local/bin/chromedriver \
  && chmod +x /usr/local/bin/chromedriver \
  && cd /code

(same result as docker-compose.yml in the first section)

ref: https://qiita.com/from_host/items/a12d75368ece7bf5bcc1

Django cheatsheet - Views

suyaa — Mon, 05 Sep 2022 11:39:50 +0000

View are something that accepts a request, and returns some kind of a response.

In the render() function, you can pass a context that are variables you can pass to your template.

how to add a new view

define the URL in your-project/your-app/urls.py
add a view in your-projects/your-app/views/new-view.py
import the view in your-projects/your-app/views/__init__.py

shortcuts: render, redirect

from django.shortcuts import render, redirect

def index(request):
  students = Students.objects.all()
  context = { "students": students }
  return render("index.html", context)

def new(request):
  # save a new student
  return redirect("your_app:new")

Note that you can't pass a context with a redirect; you can only pass a URL and query parameters.

messages

https://docs.djangoproject.com/en/4.1/ref/contrib/messages/#using-messages-in-views-and-templates

If you want to use "messages" (e.g. flash messages, toast messages in other frameworks), you can use the messages framework.

Add this import,

from django.contrib import messages

... and call the messages like this in your views.

messages.debug(request, '%s SQL statements were executed.' % count)
messages.info(request, 'Three credits remain in your account.')
messages.success(request, 'Profile details updated.')
messages.warning(request, 'Your account expires in three days.')
messages.error(request, 'Document deleted.')

The template can show the messages like this:

{% if messages %}
<ul class="messages">
    {% for message in messages %}
    <li>{{ message }}</li>
    {% endfor %}
</ul>
{% endif %}

It' s important to iterate all the messages, otherwise the messages will not be cleared.

from django.contrib import messages

# view to create new student
def new(request):
  if request.method == "POST":
    # do something with the form 
    student.save()
    # send messages
    messages.success(request, "saved student")

formsets

You can use formsets when you want to handle multiple records of the same model (e.g. multiple students).

Formsets come in handy because they will do the validation on all the records for you.

In your view, you can define your formset like this:

# your_project/your_app/views/new.py

from django.forms import modelformset_factory
from django.shortcuts import redirect, render

from your_app.models import Student

def new(request):
  # fields: defines which fields to show in the form
  # extra: defines how many extra rows you want in the form, on top of existing records
  StudentFormset = modelformset_factory(Student, fields=["last_name", "first_name", "photo"], extra=10)

  if request.method == "GET":
    # by default, formset will populate all records in that model
    # specify the set of records to be "no students"
    formset = StudentFormset(query=Student.objects.none())
    context = { "formset": formset }
    return render("your_app/new.html", context)

  if request.method == "POST":
    # populate the formset with user input
    formset = StudentFormset(request.POST, request.FILES)

    if formset.is_valid():
      # save the entire formset
      formset.save()

      # make sure to redirect after a POST request
      return redirect("your_app:new")

If you need to specify a foreign key on a formset, you do have to iterate through each record like so:

# your_project/your_app/views/new.py

if formset.is_valid():
  # get the instances via the save method
  # but don't commit yet
  instances = formset.save(commit=False)
  for student in instances:
    student.school_id = 1
    # populate school_id and save one by one
    student.save()

In this case, an "atomic transaction" might be needed, where if either all transactions succed, or no transaction succeeds.

You can use the decorator or the with: block (please see the documentation for the latter option; I prefer decorators with less indent).

# define a helper function with a decorator
from django.db import transaction

@transaction.atomic
def save_students_with_school(formset, school_id):
  instances = formset.save(commit=False)
  for student in instances:
    student.school_id = school_id
    # populate school_id and save one by one
    student.save()

# call them in your view like this

if formset.is_valid():
  # call the atomic transaction
  save_students_with_school(formset, 1)

Django SQL cheatsheet

suyaa — Mon, 05 Sep 2022 11:11:33 +0000

SELECT

filtering

# filter students by ID
Student.objects.get(pk=1)

# get array of students
Student.objects.filter(enrolled=True)

# filter by pk
Student.objects.filter(school__pk=1) # school_id = 1

# is null
Student.objects.filter(photo__isnull=True)

order by, distinct

# get distinct last_name
Student.objects.order_by("last_name").distinct()

# order by desc/multiple fields
Student.objects.order_by("-pk", "last_name")

get result as an array

>>> Student.objects.values_list("last_name", flat=True)
["Smith", "Johnson", "Williams"]

values return a dict
values_list returns a tuple, but with flat=True option returns an array

count

>>> Student.objects.filter(school__pk=2).count()
120

exists

Student.objects.filter(photo__isnull=True).exists()

CREATE

s = Student(last_name="Smith", first_name="John", school_id=1)
s.save()

UPDATE

s = Student.objects.get(pk=1)
s.last_name="Wilson"
s.save()

DELETE

s = Student.objects.get(pk=1)
s.delete()

all, none

Student.objects.none()
Student.objects.all()