DEV Community: Anuj Sharma

Django development using Docker as host - Part 5: Django development

Anuj Sharma — Sun, 17 Jan 2021 07:27:53 +0000

Till now we covered application setup and docker services running via docker-compose. In this step, we will use the docker container to develop the Django application.

Let's dive-in

1. Start `docker-compose` services

Before we start with the Django development, we need to run the docker containers with the required services.
Execute the following command to run the services defined in the docker-compose.yml file

docker-compose up --build

After the service has been started, execute the below command in a new terminal to see the running containers

docker ps

You will see the list of running containers.

2. Django settings

When the docker-compose runs for the first time, you will see a Django exception

django.core.exceptions.ImproperlyConfigured: You're using the staticfiles app without having set the STATIC_ROOT setting to a filesystem path.

This is because we have not yet configured the STATIC_ROOT settings where the static files are copied on running collectstatic management command.
Add the following settings to the src/my_app/settings.py file.

Static file settings

STATICFILES_DIRS = [
    os.path.join(os.path.dirname(BASE_DIR), 'static_my_project')
]

STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), 'static_cdn', 'static_root')

MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(os.path.dirname(BASE_DIR), 'static_cdn', 'media_root')

The STATICFILES_DIRS setting defines where the static files will be kept. While development, we will keep the static files inside the static_my_project directory.
STATIC_ROOT is the path where the static files are copied on collectstatic management command

Database settings

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'HOST': 'database',
        'PORT': '3306',
        'NAME': 'app_db',
        'USER': 'app_db',
        'PASSWORD': 'app_db'
    }
}

Where database is the service inside the docker-compose.

Template directory

In order to allow the creation of the template files inside the application directory, add the following to the TEMPLATES.DIRS settings

[
   os.path.join(BASE_DIR, 'templates')
]

This allows us to add the template files inside the src/templates directory and also in the app's templates/ directory.

3. Run Django application

After adding the settings, run the following to start the docker services

docker-compose up --build

After the services have been started, let's verify the following:

Django database has been migrated to the MySQL database. Visit http://localhost:8080 and use the database credentials to log in to the database management.
Open http://localhost:8000 to see the Django application running.

4. Live reload Django changes

As we have the .src/ directory mapped to the /app directory, the host .src/ directory will work as the container's /app directory. Any changes in the .src/ directory files will reload the Django application.
Check the docker-compose logs for the logs.

4. Sync python dependencies

We do not have python installed in our system instead we are using python using Dockerfile container. So the python dependencies should be installed in the container.

Open a bash shell in the running container

docker exec -it app_image bash

Install the dependency using pip

pip install requests

This dependency can be used until the container is running. Once stopped, re-running the container, it will be removed.
It is required to sync the requirements.txt file in the host with the latest dependencies in the container to keep both in sync.

This can be achieved in two ways:

Add the requests dependency with the version number in the host's requirements.txt file

requests==<version>

Freeze the pip dependencies from container to the host:

pip freeze > /requirements.txt

Note, since the requirements.txt in the host is mapped to the same file in the root of the container, the path should start with /. Without / it will create a requirements.txt file inside the src/ directory.

You can create the Django applications by running the manage.py management commands inside the container and edit the source code in the ./src directory.

6. Create Django app

Let's create a Django app

Run the following in the container bash

python manage.py startapp home

It will create a home directory in the ./src directory.

Add view

Add the following content to the ./src/home/views.py file

from django.views.generic import TemplateView


class IndexView(TemplateView):
    template_name = "home/index.html"

Add template

Create a template file index.html inside the templates/home directory with the following content

<html>
<head>
    <title>Index page</title>
</head>

<body>
    <h2>Welcome to the Index page</h2>
</body>
</html>

Add routing

Add the following to the home/urls.py file

from django.urls import path
from home.views import IndexView

app_name = 'home'
urlpatterns = [
    path('', IndexView.as_view()),
]

Include the URLs to the application urls.py file

path('home/', include('home.urls', namespace='home'))

Include the app to the `settings.py` file

INSTALLED_APPS = [
   'home'
]

Open in browser

In the browser, open the following URL

http://localhost:8000/home/

You will see the index page content.

Congratulations, we have completed the setup of Django project using the docker as host.
Now you can develop the Django application without depending on the host system.

😊 😍 💻

Check complete source code on Github

anuj9196 / django-docker-host

Docker host for the Django development. http://bit.ly/django-docker-host

Django development using Docker as host - Part 4: docker-compose

Anuj Sharma — Sun, 17 Jan 2021 07:27:41 +0000

In the previous step, we initialized the Django application using the docker image.

In this part, we will be creating the docker-compose.yml file and define the required services to run the application.

Let's start

1. Create directories

Create two directories at the root of the project, src and static_my_project in the host where source code of the Django application will reside in the src directory and the static files in the static_my_project directory.
The directory structure will look like

- /
  | - scripts
  | - src
  | - static_my_project
  | - docker-compose.yml
  | - Dockerfile
  | - requirements.txt

2. Create `dokcer-compose.yml`

Create the docker-compose.yml file in the host with the content

version: '3.7'

3. Define database service

Let's define a database service that will be consumed by the Django application.
Add the following to the docker-compose.yml

services:
  database:
    image: mysql:8.0
    container_name: app__database
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    volumes:
    - app_db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=app_db
      - MYSQL_DATABASE=app_db
      - MYSQL_USER=app_db
      - MYSQL_PASSWORD=app_db
    ports:
    - 3306:3306

volumes:
  app_db:

We are using mysql version 8.0 image for the database.
We have also defined a docker volume to store database files inside app_db.
The database will be running on port 3306.

4. Define adminer server (optional)

Adminer is a web-based database management service. Using this service we can directly access the database in the browser.
Add the following content to the docker-compose.yml file

services:
  adminer:
    image: adminer:latest
    restart: always
    ports:
    - 8080:8080

The adminer services will be running on port 8080.

5. Define app service

Let's create an application service. We will use the Dockerfile to build the image.
Add the following content to the docker-compose.yml

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: app_image
    command: ["/scripts/docker/wait_for_it.sh", "database:3306", "--", "/scripts/docker/docker_start.sh"]
    depends_on:
      - database
    ports:
    - 8000:8000
    volumes:
    - ./requirements.txt:/app/requirements.txt
    - ./static_my_project:/static_my_project
    - ./src:/app

The application will be build using the Dockerfile we created in step 1
When running, the application container will be named app_image
Django by default runs on port 8000, we have mapped the port 8000 of the host to the container so that the server can be accessed from the host browser.
We have mapped the following host volumes to the container files/folder.
- requirements.txt -- Any modification to the requirements.txt file in the container will be updated to the host requirements.txt file
- The static_my_project directory of the host will be mapped to the static_my_project directory in the container.
- The /app directory is mapped to the ./src directory in the container. Any changes to the host ./src or container /app directory will be synced.

The contents of the docker-compose.yml

version: '3.7'

services:

  app:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: app_image
    command: ["/scripts/docker/wait_for_it.sh", "database:3306", "--", "/scripts/docker/docker_start.sh"]
    depends_on:
      - database
    ports:
    - 8000:8000
    volumes:
    - ./requirements.txt:/app/requirements.txt
    - ./static_my_project:/static_my_project
    - ./src:/app

  database:
    image: mysql:8.0
    container_name: app__database
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    volumes:
    - app_db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=app_db
      - MYSQL_DATABASE=app_db
      - MYSQL_USER=app_db
      - MYSQL_PASSWORD=app_db
    ports:
    - 3306:3306

  adminer:
    image: adminer:latest
    restart: always
    ports:
    - 8080:8080

volumes:
  app_db:

6. Run docker-compose

To run the defined services inside the docker-compose.yml file, let's run the services by executing

docker-compose up --build

It will build the application image and run the services.

Let's verify first if database and adminer services are running. Open the following link in the browser

http://localhost:8080

If the adminer login page appears, enter the following details

Server: database
Username: app_db
Password: app_db
Database: app_db

If the login works, the database and the adminer services are successfully running.

Django development using Docker as host - Part 3: Create Django project

Anuj Sharma — Sun, 17 Jan 2021 07:27:31 +0000

Till now we have completed with the Dockerfile and the required script files. In this step, we will create a Django project inside the container.

Let's start

1. Build docker image

First, we need to build the image using the Dockerfile. Execute the following command to build the image and add a tag my_app

docker build -t my_app .

After the image building has completed, you can see the image by running

docker images

2. Run the image

Run the bash terminal using the image

docker run -it --rm -v "$(pwd)"/src:/app/ app_tut bash

-it will run the image in interactive tty mode
--rm will remove the container when the run is exited
-v "$(pwd)/src:/app/ will mount the ./src directory in current working directory to the /app inside the container
my_app is the image name
bash is the command to execute inside the container.

3. Create Django project

First, verify the Django installation inside the container by running the following command inside the container bash session

pip list

It will give a list of installed dependencies inside the container. Verify the Django installation.

Package    Version
---------- -------
Django     3.1.5

As Django is installed, run the below command to create a Django project with name my_app in the current directory.

django-admin startproject my_app .

To confirm the project creation, list the files in the current directory

ls -la

You will see a directory my_app and manage.py file

-rwxr-xr-x 1 root root  662 Jan 16 17:54 manage.py
drwxr-xr-x 7 root root  224 Jan 16 17:54 my_app

Did you noticed something?
You can see the my_app directory and the manage.py in the host src directory as well.
This is because we have mounted the src directory with the /app directory using -v parameter.

This is all for initializing our Django project. In the next step, we will use the docker-compose to run the Django application and mounted the volumes.

Type exit to exit from the docker image shell.

4. Install python dependencies

Since we will be using the MySQL database with the Django project, mysqlclient python dependency needs to be installed.
Add the following to the requirements.txt because its installation will require the build-essential library.

mysqlclient==2.0.3

Django development using Docker as host - Part 2: Create scripts

Anuj Sharma — Sun, 17 Jan 2021 07:27:18 +0000

In Part 1, we created a Dockerfile with the required system dependencies.

In this part, we will be creating the required files and scripts.

Let's dive in

1. Create `requirements.txt`

First, we need to define our basic python dependency for the Django project setup. Although it is not required, to omit any error with the pip install -r requirements.txt inside the container, let's add our first dependency.

Django==3.1.5

2. Create `entrypoint.sh`

Entrypoint script is executed every time the image is run. I recommend not to run the Django server using this.
Create a file scripts/docker/entrypoint.sh with the following content

#!/bin/sh

exec "$@"

3. Create `docker_start.sh`

We will use this script to run the Django server.
Create a file scripts/docker/docker_start.sh with the following content

#!/usr/bin/env bash

echo -e "\e[34m >>> Migrating changes \e[97m"
python manage.py migrate
echo -e "\e[32m >>> migration completed \e[97m"

echo -e "\e[34m >>> Collecting Static files \e[97m"
python manage.py collectstatic --noinput
echo -e "\e[32m >>> Static files collect completed \e[97m"

python manage.py runserver 0.0.0.0:8000

It will run the migration, copies the static files and run the server over port 8000

4. Create `wait_for_it.sh`

This script is needed to wait for the database connection to start before starting the Django server.
Create a file scripts/docker/wait_for_it.sh with the content from git repository vishnubob/wait-for-it

5. Make scripts executable

In order to execute the scripts, it is required to make them executable.
Run the following command in the terminal from the project directory

chmod +x scripts/docker/entrypoint.sh
chmod +x scripts/docker/docker_start.sh
chmod +x scripts/docker/wait_for_it.sh

That's it for this part. Next, we will create the Django project using the docker image.

Django development using Docker as host - Part 1: Dockerfile

Anuj Sharma — Sun, 17 Jan 2021 07:27:01 +0000

Django

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel.

Docker

Docker is a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers.

Recently while developing a Django project, I faced multiple issues installing the system dependencies (mysqlclient, etc) to install the required package for the working of my Django application in the local development environment.

This is always a hassle installing the system dependencies in the development machine, when

The host is formatted
You change your development machine
Change of the operating system (say windows to Linux)
Upgrading of operating system (removes support for the earlier version of libraries, always comes with the new version of libraries which may not have support for the application)

And again you start the race with the system setup, killing lots of time.

Even though we use Dockerfile to create production-ready images to run the production server, why not use the same as host and remove the system dependencies with all phases of the development?

Using the Docker as host, the release servers will always be in sync with the development machine and you can always look over the dependencies required for your application.

Let's jump in

If you have not already installed Docker and docker-compose on your machine, first install both by following the official guide.

Docker: https://docs.docker.com/get-docker/
docker-compose: https://docs.docker.com/compose/install/

1. Create `Dockerfile`

We need to first create a Dockerfile in the project directory, where we will be creating the Django project.
To keep the image size lower, we will be using the latest slim python image python:3.9.1-slim.
Add the following to the first line of the Dockerfile

FROM python:3.9.1-slim

2. Add non-root user

It is always recommended to create a non-root user instead of using the root user in the Dockerfile. So let's create a user by the name of the project (myapp).

ARG APP_USER=myapp
RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -m -g ${APP_USER} ${APP_USER}

The ARG APP_USER=myapp creates an argument variable so that the value myapp can be referenced using the variable throughout the Dockerfile.
The next command adds a group and creates a user with the myapp, creates home directory /home/myapp for the user and assigns the permission to the home directory.

3. Install runtime dependencies

With Django, we are going to use the latest version of MySQL, so let's add the required system dependencies to the Dockerfile. Also, we will clean the Linux apt list as it is no longer required.

# default-libmysqlclient-dev -- Required for mysql database support
RUN set -ex \
    # Runtime dependencies
    && RUN_DEPS=" \
    default-libmysqlclient-dev \
    " \
    && seq 1 8 | xargs -I{} mkdir -p /usr/share/man/man{} \
    && apt-get update && apt-get install -y --no-install-recommends $RUN_DEPS \
    # Remove package list
    && rm -rf /var/lib/apt/lists/* \
    && mkdir /static_my_project

What above command does is

Installs the mysqlclient system dependency
Removes the apt lists

4. Copy files to the container

Now, change the working directory to the /app/ directory

WORKDIR /app/

Let's copy the requirements.txt file for the dependency instalment. (We will create it in further steps)

ADD requirements.txt /requirements.txt

Let's copy the application source code and the scripts to the container

COPY ./src /app/
COPY scripts/ /scripts/

5. Install python dependencies

Now we will install the required dependencies from the requirements.txt file

RUN set -ex \
    # Define build dependencies, they will be removed after build completes and libraries has been installed
    && BUILD_DEPS=" \
    build-essential \
    " \
    && apt-get update && apt-get install -y --no-install-recommends $BUILD_DEPS \
    && pip install -r /requirements.txt  \
    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $BUILD_DEPS \
    && rm -rf /var/lib/apt/lists/*

The above command does the following

Installs the build dependencies
Installs the python dependencies
Removes the build dependencies
Deletes the repository lists

Now we need to expose the port 8000 to access the running server

EXPOSE 8000

6. Add `ENTRYPOINT`

Let's add an ENTRYPOINT which allows running manage.py commands without running the server directly. We will run the server by executing a command on the image. (I personally use this pattern because it gives you the flexibility to directly run Django shell without running the server).

ENTRYPOINT ["/scripts/docker/entrypoint.sh"]

That's it for the Dockerfile. Here is the final content

FROM python:3.9.1-slim

ARG APP_USER=myapp
RUN groupadd -r ${APP_USER} && useradd --no-log-init -r -m -g ${APP_USER} ${APP_USER}

# default-libmysqlclient-dev -- Required for mysql database support
RUN set -ex \
    # Runtime dependencies
    && RUN_DEPS=" \
    default-libmysqlclient-dev \
    " \
    && seq 1 8 | xargs -I{} mkdir -p /usr/share/man/man{} \
    && apt-get update && apt-get install -y --no-install-recommends $RUN_DEPS \
    # Remove package list
    && rm -rf /var/lib/apt/lists/* \
    && mkdir /static_my_project

WORKDIR /app/

ADD requirements.txt /requirements.txt
COPY ./src /app/
COPY scripts/ /scripts/

# build-essential -- Required to build python mysqlclient library. https://packages.debian.org/sid/build-essential
RUN set -ex \
    # Define build dependencies, they will be removed after build completes and libraries has been installed
    && BUILD_DEPS=" \
    build-essential \
    " \
    && apt-get update && apt-get install -y --no-install-recommends $BUILD_DEPS \
    && pip install -r /requirements.txt  \
    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $BUILD_DEPS \
    && rm -rf /var/lib/apt/lists/*

ENTRYPOINT ["/scripts/docker/entrypoint.sh"]

DEV Community: Anuj Sharma

Django development using Docker as host - Part 5: Django development

TOC

Let's dive-in

1. Start docker-compose services

2. Django settings

Static file settings

Database settings

Template directory

3. Run Django application

4. Live reload Django changes

4. Sync python dependencies

6. Create Django app

Let's create a Django app

Add view

Add template

Add routing

Include the app to the settings.py file

Open in browser

anuj9196 / django-docker-host

Docker host for the Django development. http://bit.ly/django-docker-host

Django development using Docker as host - Part 4: docker-compose

TOC

Let's start

1. Create directories

2. Create dokcer-compose.yml

3. Define database service

4. Define adminer server (optional)

5. Define app service

6. Run docker-compose

Django development using Docker as host - Part 3: Create Django project

TOC

Let's start

1. Build docker image

2. Run the image

3. Create Django project

4. Install python dependencies

Django development using Docker as host - Part 2: Create scripts

TOC

Let's dive in

1. Create requirements.txt

2. Create entrypoint.sh

3. Create docker_start.sh

4. Create wait_for_it.sh

5. Make scripts executable

Django development using Docker as host - Part 1: Dockerfile

Django

Docker

TOC

Let's jump in

1. Create Dockerfile

2. Add non-root user

3. Install runtime dependencies

4. Copy files to the container

5. Install python dependencies

6. Add ENTRYPOINT

1. Start `docker-compose` services

Include the app to the `settings.py` file

2. Create `dokcer-compose.yml`

1. Create `requirements.txt`

2. Create `entrypoint.sh`

3. Create `docker_start.sh`

4. Create `wait_for_it.sh`

1. Create `Dockerfile`

6. Add `ENTRYPOINT`