DEV Community: Jakub Dubec

How We Built an Application to Test Student Docker Images for Database Systems University Course

Jakub Dubec — Thu, 23 Feb 2023 22:22:21 +0000

Introduction & Motivation

In today's digital age, software development has become an increasingly important field of study, with universities and colleges offering a range of courses to prepare students for the industry. One such course is the Database System course on the Faculty of Informatics and Information Technologies STU in Bratislava, which teaches students how to design, build and manage databases. As part of this course, students have to create an HTTP server which interacts with a database, to demonstrate their understanding of the concepts covered in the course.

However, running and validating these HTTP servers can be a time-consuming and challenging task for instructors, especially as the number of students increases. To address this problem, an web application was developed which can accept Docker images created by students, run the image, provide environment variables for database connections and perform a set of HTTP requests to validate the implementation. This web-based application is created using the Django web framework, uses Redis for queue management and the Docker Python SDK to communicate with the Docker Daemon.

By automating the testing process, this application not only saves time for instructors but also ensures a consistent and fair evaluation of student work. Moreover, it provides students with an opportunity to learn about the use of Docker in a real-world scenario, thereby preparing them for the industry.

This post is structured as follows: The first chapter Requirements and Design, describes the requirements for such an application, defines its processes, breaks it down into logical components, and proposes a data model. The second chapter Implementation, provides an introduction to key implementation issues, such as implementing asynchronous tasks and LDAP authentication. It also showcases the usage of Docker with Python SDK in the project, including network configuration, and describes the deployment configuration using supervisord. The final chapter summarizes the efforts and provides links to the code repositories.

Requirements and design

Our university courses rely on GitHub for Education. Our idea was to build a web application that enables students to log in with their academic credentials (OpenLDAP) and submit a link to their Docker image hosted by GitHub. The simplified workflow is shown in the sequence diagram below.

When students submit a URL to their Docker image, details are saved in the database and a task is added to the Redis queue. The student is then redirected to the detail page of the Task, which refreshes periodically and shows the current status of the task (including its results if finished).

Tasks are processed by the worker, as visualized in the simplified diagram below. We use the django-rq package to implement an async queue using Redis.

The worker creates a Docker container for the student from the provided image and performs a series of pre-defined URL requests. The results are saved to the database.

The database model of the application is shown in the image below.

The following entities are present in the database model:

Assignment: Assignments for students that they have to complete during the term.
Scenario: The definition of the test request for a specific Assignment (including expected results). Some scenarios are private and are only used by the course supervisors for the final evaluation.
Task: The test request created by the student. It contains information about the Docker image and waits in the queue until processing. The entity has the following states: ok, fail and pending.
TaskRecord: An instance of a Scenario for the Task. Entity has following states: success, fail and mismatch.
AuthSource: LDAP configurations.
AuthUser: User definitions.

The entire project is shown in the component diagram below. The Tester package represents our application. We use PostgreSQL for data storage, Redis for the queue, and Docker for containerization.

The diagrams presented are simplified to improve readability.

Implementation

The implementation of the application was done using the Django web framework. Django was chosen for its user-friendly ORM, customizable authentication, auto-generated administration, and rich ecosystem of libraries and resources. In this chapter, we will take a closer look at some of the key implementation issues and how they were addressed.

The chapter is structured into five main sections, each covering a specific aspect of the implementation. The first section covers the implementation of asynchronous tasks, which were necessary to handle the running of Docker images in a non-blocking and scalable way. The second section discusses the implementation of LDAP authentication, which was used to authenticate users against the university's OpenLDAP server.

The third section delves into the usage of the Python Docker SDK in the project. This SDK was used to communicate with the Docker daemon and perform various operations, such as creating and managing Docker containers. The fourth section is dedicated to configuration of performing the HTTP requests using the long-pooling method (in case if the container applications is not yet fully loaded).
Finally, the fifth section describes the deployment using the supervisord and systemd.

Asynchronous tasks

Performing tests on student images can be a time-consuming process, taking several minutes. Application has to download the image from the registry, create a container, wait for the application to load, and perform the test scenarios. To efficiently utilize the available hardware and handle increased server demand during peak times, a task queue was implemented using django-rq.

Student creates a Task, which contains all the necessary information about the test scenarios that will be executed (by providing a path to the Docker image and choosing a Assignment). The task is then added to the Redis queue, which is consumed by workers that process the tasks. Redis provides atomic access to its structures, ensuring that the same task is not processed multiple times.

The implementation of the task queue can be divided into three main steps: configuring the queue, implementing the workers, and adding tasks to the queue. The first step involves configuring the Redis queue using Django settings. The second step requires implementing the workers to process the tasks in the queue, which is done by creating a Python function that performs the necessary test scenarios. The final step involves adding tasks to the queue, which is done by calling the function with the necessary parameters and pushing it onto the queue.

Configuring the queue

Configuring the task queue involves defining the Redis queue and its connection settings in the Django settings file. The example below defines a task queue called default and connects to the Redis server using REDIS_HOST, REDIS_PORT, RQ_REDIS_DB, and REDIS_PASSWORD environment variables (with fallback to localhost:6379 on database 0 without a password).

# dbs_tester/settings/base.py
RQ_QUEUES = {
    'default': {
        'HOST': os.getenv('REDIS_HOST', 'localhost'),
        'PORT': int(os.getenv('REDIS_PORT', 6379)),
        'DB': int(os.getenv('RQ_REDIS_DB', 0)),
        'PASSWORD': os.getenv('REDIS_PASSWORD', None),
        'DEFAULT_TIMEOUT': 360,
    }
}

RQ_EXCEPTION_HANDLERS = [
    'apps.core.jobs.exception_handler'
]

RQ_SHOW_ADMIN_LINK = True

The RQ_QUEUES setting defines the task queue and its connection settings. The DEFAULT_TIMEOUT setting defines the default timeout for tasks in the queue.

The RQ_EXCEPTION_HANDLERS setting is used to define custom exception handlers for handling task failures. To handle failed tasks, a custom exception handler is implemented in the Django application. When a task fails with an exception, the custom exception handler can change the state of the task. This ensures that failed tests are properly recorded and can be reviewed later.

# apps/core/jobs.py
def exception_handler(job, exc_type, exc_value, traceback):
    try:
        task = Task.objects.get(pk=job.args[0])
    except Task.DoesNotExist:
        return

    task.status = Task.Status.FAILED
    task.message = str(exc_value)
    task.save()

When a task fails with an exception, the exception handler is triggered, and it changes the state of the task to fail. Additionally, the handler saves information about the exception, which can be used to help diagnose and fix any issues with the student's container.

By providing this information to the students, they can quickly identify and fix any issues with their containers, leading to a more efficient testing process. This approach helps to ensure that students are able to learn from their mistakes and improve their skills, while also ensuring that the application is able to provide accurate and reliable feedback.

The RQ_SHOW_ADMIN_LINK setting is used to provide access to a simple web-based interface for monitoring the status of the task queue. This setting displays a link to the Django administration panel on the RQ dashboard, providing a convenient way to monitor and manage the task queue.

With this approach, multiple tasks can be processed in parallel, making the process more scalable and efficient.

Implementing the workers

In this implementation, the worker is defined as a class called BasicJob, which contains four methods: prepare, run, cleanup, and a static method called execute. The workflow for the worker is defined in the Requirements and Design chapter of the blog post.

# apps/core/jobs.py
class BasicJob:
    def __init__(self, task: Task, public_only: bool):
        self._task = task
        self._public_only = public_only
        self._database_name = ''.join(random.choices(string.ascii_letters, k=10)).lower()
        self._database_password = ''.join(random.choices(string.ascii_letters, k=10)).lower()

    def prepare(self):
        with connection.cursor() as cursor:
            cursor.execute(
                f"CREATE DATABASE {self._database_name} TEMPLATE {self._task.assigment.database or 'template0'};"
            )
            cursor.execute(
                f"CREATE USER {self._database_name} WITH ENCRYPTED PASSWORD '{self._database_password}';"
            )
            cursor.execute(f"GRANT ALL PRIVILEGES ON DATABASE {self._database_name} TO {self._database_name};")

    def run(self):
        pass

    def cleanup(self):
        with connection.cursor() as cursor:
            cursor.execute(f"DROP DATABASE {self._database_name};")
            cursor.execute(f"DROP USER {self._database_name};")

    @staticmethod
    def execute(task_id: UUID, public_only: bool) -> Optional[Task]:
        try:
            task = Task.objects.get(pk=task_id)
        except Task.DoesNotExist:
            logging.error("Task %s does not exist!", task_id)
            return None

        if task.status != Task.Status.PENDING:
            logging.warning("Task %s is already done! Skipping.", task.pk)
            return None

        job = BasicJob(task, public_only)
        job.prepare()
        try:
            job.run()
        except Exception as e:
            task.status = Task.Status.FAILED
            task.message = str(e)
            task.save()

        job.cleanup()

        return task

The prepare method creates a temporary database user with a database according to the Assignment specification, while the run method executes the container and performs the tests. The cleanup method removes the temporary user and database. We use the super sophisticated design pattern called Gotta catch them all to catch any exceptions that may be raised from the run method (I was not able to find the author of the image so I can't credit him).

However, workers cannot be specified as a class or static method. Instead, only a simple Python function can be used. To work around this limitation, a simple wrapper was implemented that calls a static method to create and execute BasicJob. The attributes for our job are the task_id, which is the UUID of the student task, and the public_only flag, which is used to determine if only public tests will be performed. Private scenarios were also defined for the final evaluation, so students have to think about edge-cases.

# apps/core/jobs.py
def basic_job(task_id: UUID, public_only: bool) -> Optional[Task]:
    return BasicJob.execute(task_id, public_only)

Adding tasks to the queue

Once we have our task definition prepared, we can create a view that will add it to the queue.

# apps/web/views/tasks.py
class CrateTaskView(LoginRequiredMixin, CreateView):
    model = Task
    form_class = TaskForm
    template_name = 'web/task.html'

    def __init__(self):
        self.object = None

    def form_valid(self, form):
        self.object = form.save(commit=False)
        self.object.user = self.request.user
        self.object.save()
        django_rq.enqueue(basic_job, self.object.pk, not self.request.user.is_staff)
        return HttpResponseRedirect(self.get_success_url())

In the provided code block, the CrateTaskView uses LoginRequiredMixin to ensure that only logged-in users can add tasks to the queue. CreateView is used to simplify the implementation of the form validation. If the TaskForm is valid, the form_valid method is executed.

The form_valid method first creates a new Task object from the form data and saves it to the database. The django_rq.enqueue method is then used to add the apps.core.jobs.basic_job function to the queue, along with the task.id and a flag to determine if only public tests will be performed (only staff users can perform private scenarios). The basic_job function will then be executed by the worker.

LDAP

To ensure that only users from the university can access the application, LDAP authentication was implemented. To make the application as customizable as possible, we created a database entity called auth_source that stores the LDAP configurations. A custom Django authentication backend called LdapBackend was developed, which uses the auth_source to create user accounts on login.

# apps/core/auth/LdapBackend.py
class LdapBackend(ModelBackend):
    class Config(TypedDict):
        URI: str
        ROOT_DN: str
        BIND: str
        USER_ATTR_MAP: Dict[str, str]
        GROUP_MAP: Dict[str, str]
        FILTER: str

    def _ldap(self, username: str, password: str, auth_source: AuthSource) -> Optional[User]:
        config: LdapBackend.Config = auth_source.content
        connection = ldap.initialize(uri=config['URI'])
        connection.set_option(ldap.OPT_REFERRALS, 0)

        try:
            connection.simple_bind_s(config['BIND'].format(username=username), password)
        except ldap.LDAPError as e:
            logging.warning(
                f"Unable to bind with external service (id={auth_source.pk}, name={auth_source.name}): {e}"
            )
            return None

        try:
            user = User.objects.get(username=username)
        except User.DoesNotExist:
            user = User(
                username=username,
            )
            user.set_unusable_password()

        result = connection.search(
            f"{config['ROOT_DN']}", ldap.SCOPE_SUBTREE, config['FILTER'].format(username=username), ['*']
        )

        user_type, profiles = connection.result(result, 60)

        if profiles:
            name, attrs = profiles[0]

            # LDAP properties
            for model_property, ldap_property in config['USER_ATTR_MAP'].items():
                setattr(user, model_property, attrs[ldap_property][0].decode())

            user.last_login = timezone.now()
            user.save()

            # LDAP groups
            user.groups.clear()
            for ldap_group in attrs.get('memberOf', []):
                if ldap_group.decode() in config['GROUP_MAP']:
                    try:
                        group = Group.objects.get(name=config['GROUP_MAP'][ldap_group.decode()])
                    except Group.DoesNotExist:
                        continue
                    user.groups.add(group)
        else:
            logging.warning(
                f"Could not find user profile for {username} in auth source {auth_source.name}"
                f" (id={auth_source.pk}, name={auth_source.name})"
            )
            return None

        connection.unbind()

        user.save()

        return user

    def authenticate(self, request, username=None, password=None, **kwargs):
        user = None

        for auth_source in AuthSource.objects.filter(is_active=True):
            logging.debug(f'Checking {auth_source.name}')
            if auth_source.driver == AuthSource.Driver.LDAP:
                user = self._ldap(username, password, auth_source)

            if user:
                break

        return user

The _ldap method establish a connection with the LDAP server. If the user credentials are valid, it retrieves the user profile and creates a new user account in th database, if one does not already exist. It also updates the user's information, including their groups.

To use the LdapBackend authentication backend, it must be added to the AUTHENTICATION_BACKENDS list in the Django settings.

# dbs_tester/settings/base.py
AUTHENTICATION_BACKENDS = [
    'django.contrib.auth.backends.ModelBackend',
    'apps.core.auth.LdapBackend'
]

With this implementation, only users from the university who have valid LDAP credentials can access the application.

Docker

As previously mentioned, the students' assignments are submitted as Docker images hosted on the GitHub Container Registry, and to run these images, the machine running the application needs to have access to this registry. This is a one-time configuration that needs to be done on the production server using the docker login command.

We use the official Docker Python SDK to manipulate images and containers. All students' containers run in a separate NAT Docker network.

To illustrate the creation of a Docker container, we can refer to the example code below:

# apps/web/views/tasks.py
client = docker.from_env()

params = {
    'image': self._task.image,
    'detach': True,
    'environment': {
        'DATABASE_HOST': settings.DATABASES['default']['HOST'],
        'DATABASE_PORT': settings.DATABASES['default']['PORT'],
        'DATABASE_NAME': self._database_name,
        'DATABASE_USER': self._database_name,
        'DATABASE_PASSWORD': self._database_password,
    },
    'name': self._task.id,
    'privileged': False,
    'network': settings.DBS_DOCKER_NETWORK,
    'extra_hosts': {
        'host.docker.internal': 'host-gateway',
        'docker.for.mac.localhost': 'host-gateway'
    },
    'ports': {
        '8000/tcp': '9050'
    }
}

container: Container = client.containers.run(**params)
sleep(5)
container.reload()

The docker.from_env() method creates a Docker client connected to the Docker daemon running on the local machine, which allows us to manage the Docker containers and images. Then, we define the parameters for the container and run the container using the client.containers.run() method.

When creating a new container, we pass a dictionary object to the run() method which contains the required parameters, such as:

image: the name of the image to create a container from,
detach: to run the container in the background and return immediately,
environment: a dictionary of environment variables to set inside the container,
network: the name of the NAT Docker network that the container is connected to.

We use the environment variables to pass information about the database connection with the temporary credentials (which will be deleted after the test).

We also pass extra hosts which allow us to access the host's Docker daemon from within the container. This is required when using Docker for Mac or Docker for Windows. Finally, we specify port mapping using the ports to make the container's application accessible from the host machine.

In this example, we also wait five seconds to let the container initialize before reloading the container object. This is necessary to get the container's IP address so we can make HTTP requests to it from the test runner.

To retrieve the container IP address, the container.reload() method must first be called. The network adapters can then be accessed via container.attrs['NetworkSettings']['Networks']. Since the network is specified using the network parameter, the IP address can be accessed using the following code:

container: Container = client.containers.run(**params)
sleep(5)
container.reload()

ip_address = container.attrs['NetworkSettings']['Networks'][settings.DBS_DOCKER_NETWORK]['IPAddress']
print(ip_address)

As we want to minimize the execution time of the tests, we run them asynchronously in the background. Since the tests are run on separate Docker containers, we need to make sure that the containers are stopped after the tests are completed. This can be achieved by using the container.stop() method, which stops the container, and then using the container.remove() method to remove the container from the Docker host.

Performing requests

In order to handle cases where the student's application might take longer to load, we use the requests package with a Retry object to perform HTTP requests. The Retry object is configured to attempt 6 requests with a 2 second backoff if an attempt fails. The Session object is used to persist parameters across requests and to take advantage of connection pooling.

from requests import Session, Request
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

s = Session()
retry = Retry(connect=6, backoff_factor=2)
adapter = HTTPAdapter(max_retries=retry)
s.mount('http://', adapter)
s.mount('https://', adapter)
req = Request(
    method=scenario.method,
    url=record.url,
)

Deployment

The application is currently deployed as a single Docker container, which utilizes supervisord to run all the necessary modules including the Django application and the workers. The Docker image is mounted to the path to the Docker socket from the host machine, which allows it to operate in a docker-in-docker manner.

The supervisord configuration file contains two programs - gunicorn and worker. The gunicorn program runs the Django application, while the worker program executes the workers using RQ. The number of worker processes can be specified using the numprocs parameter.

[supervisord]
nodaemon=true

[program:gunicorn]
directory=/usr/src/app
command=/opt/venv/bin/gunicorn -w 4 --log-level=debug --log-file=/var/log/gunicorn.log --bind unix:/var/run/gunicorn/gunicorn.socket --pid /var/run/gunicorn/gunicorn.pid dbs_tester.wsgi
autostart=true
autorestart=true
priority=900
stdout_logfile=/var/log/gunicorn.stdout.log
stderr_logfile=/var/log/gunicorn.stderr.log

[program:worker]
directory=/usr/src/app
command=/opt/venv/bin/python manage.py rqworker default
autostart=true
autorestart=true
priority=900
stdout_logfile=/var/log/rqworker.stdout.log
stderr_logfile=/var/log/rqworker.stderr.log
numprocs=4
process_name=%(program_name)s_%(process_num)02d

The application is executed using Systemd by running a Docker image. A unit file is used to configure the service. The Docker image is executed with parameters specifying the environment variables used in the Django settings file. The add-host parameter is used to make the host.docker.internal accessible from within the container. The unit file specifies that the docker.service is required and the network-online.target is wanted before the service is started. The ExecStartPre section is used to stop and remove the existing container before starting a new one. The ExecStartPost section is used to remove the container after stopping it. The dbs network is used to allow communication between the application container and other containers running in the same network.

[Unit]
Description=DBS Tester
Wants=network-online.target
After=network-online.target docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker stop -t 5 dbs-tester
ExecStartPre=-/usr/bin/docker rm dbs-tester
ExecStartPre=-/usr/bin/docker pull ghcr.io/fiit-databases/tester:master
ExecStart=/usr/bin/docker run -p 9200:9000 -v /var/run/docker.sock:/var/run/docker.sock -v ./logs:/var/log/ --env BASE_URL= --env ALLOWED_HOSTS= --env DATABASE_HOST= --env DATABASE_NAME= --env DATABASE_PASSWORD= --env DATABASE_PORT= --env DATABASE_USER= --env DJANGO_SETTINGS_MODULE=dbs_tester.settings.production --env REDIS_HOST= --env SECRET_KEY= --env GITHUB_TOKEN= --env GITHUB_USER= --name dbs-tester --network dbs --add-host=host.docker.internal:host-gateway ghcr.io/fiit-databases/tester:master
ExecStop=/usr/bin/docker stop -t 5 dbs-tester
ExecStopPost=-/usr/bin/docker rm dbs-tester

[Install]
WantedBy=multi-user.target

Conclusion

In conclusion, we have developed a web application for testing Docker images submitted by students using a series of predefined URL requests. We have used various technologies, such as Django, Redis, PostgreSQL, and Docker, to create a customisable and efficient system for automating the evaluation of student assignments.

The application includes several features to improve security, such as using LDAP authentication to restrict access to authenticated users from our university, and running student containers in a separate NAT Docker network.

The system is designed to handle a large number of simultaneous requests, and we have used the django-rq library to implement an asynchronous queue using Redis. This approach ensures that the system is scalable and can process a large number of requests efficiently.

Overall, the system provides an automated and efficient way to evaluate student assignments, reducing the workload of instructors and providing a more objective evaluation process.

The source code of the project is available on the GitHub repository FIIT-Databases/tester. Additionally, an example assignment for the project is also available on GitHub as FIIT-Databases/dbs-python-example. The project is being used in the education process at the Faculty of Informatics and Information Technologies STU in Bratislava. Anyone can access and use the project to create their own testing platform.

In addition to the features and problems that were mentioned, there are many more that were encountered during the development of this project. To get a more detailed overview, feel free to check the repositories provided or ask any questions in the comments. Your feedback and questions are appreciated, and I look forward to hearing from you!

Migrating data between two online MinIO instances

Jakub Dubec — Sun, 03 Jul 2022 16:28:45 +0000

Introduction

I use MinIO to store data for my personal Docker containers. It's super easy to use in applications and it solves problem data persistence in Docker (it's much more flexible than using volumes - sure, it also depends on the use-case).

Even though I am a quite satisfied developer I have to admit I struggled for a while trying to find out how to migrate data between two online MinIO instances. There is a GitHub issue for this situation but the response there is quite disappointing. I had to dig into documentation to find the response.

This article is for those who are lazy as I am and love quick straightforward solutions.

Preparation

We are going to use an official MinIO client called mc. I use macOS so installation is quite easy. The process supposes to be similar on different platforms.

In this example, we are going to migrate data from my_source to my_destination. We start with creating such a configuration. The default path is in your home directory of yours (~/.mc/config.json). The file is supposed to look like this (with your values).

{
    "version": "10",
    "aliases": {
        "my_source": {
            "url": "http://<remote_ip>:9001",
            "accessKey": "my_access_key",
            "secretKey": "my_secret",
            "api": "S3v2",
            "path": "auto"
        },
        "my_destination": {
            "url": "https://data.example.com",
            "accessKey": "my_access_key",
            "secretKey": "my_secret",
            "api": "S3v4",
            "path": "auto"
        }
    }
}

Migration

After configuration of your instances, the rest is quite simple. We are going to use a mc mirror command.

mc mirror my_source my_destination

Simple as that!

Creating containers for Django apps with periodical tasks

Jakub Dubec — Sun, 28 Nov 2021 01:41:57 +0000

As a Django application developer, you quite often encountered a situation where you would like to have the ability to perform periodical asynchronous background tasks. It comes in handy if you want to create some background checks, send notifications or build cache.

Motivation

My first choice was the installation of the django-celery-beats. I was quite satisfied with the result. I was able to dynamically configure my periodic task according to the user configuration because Celery read execution configuration from the database.

On the other hand, my application now has a dependency on the celery (which has to run as a separated service) and Redis server (which celery uses as message broker).

The containerization of the application became weird. I was not sure, if I am supposed to include the celery service inside of the container or as a dependency using docker-compose. I also thought it's ridiculous that I need to have Redis instance only to perform periodical tasks. I also wanted to reduce the size of the hypervisord configuration.

I wanted to keep flexibility but reduce the number of dependencies and configuration boilerplate.

Solution

I get rid of the celery in the applications where it was not necessary (I was not using the rest of the great celery features). I just wanted to load periodicity from the django.conf.settings or the database.

I use Alpine Linux as a base image for my applications. The base operating systems already can handle the execution of the periodical tasks. It's good old crond. The configuration is described in the Alpine Linux docs.

I created a simple Django management command called setup in my application. Let us also assume I have another Django management command called popularity which is supposed to run every five minutes. In this example I will read a configuration from the Django settings variable CRON_JOBS which can look like this:

CRON_JOBS = {
    'popularity': '*/5 * * * *'
}

The variable consists of the Django management command name and periodicity pairs. The command will create a crond configuration according to it using python-crontab library. This command is supposed to be run before the first run and every time if configuration changes (keep in mind as you are already creating CRON job rules from the management command, you can use ORM to read configuration from the database).

# management/commands/setup.py
from crontab import CronTab
from django.core.management import BaseCommand

class Command(BaseCommand):
    cron = CronTab(tabfile='/etc/crontabs/root', user=True)
    cron.remove_all()

    for command, schedule in settings.CRON_JOBS.items():
        job = cron.new(command='cd /usr/src/app && python3 manage.py {}'.format(command), comment=command)
        job.setall(schedule)
        job.enable()

    cron.write()

Keep in mind, if you are loading the data from the database, you have to perform this change once again using the setup management command. You can achieve this using call_command method.

from django.core import management

def save_config():
    # do whatever you want
    management.call_method('setup')

Creating container

As I said before, my Django applications are based on the Alpine Linux containers and are executed from the entrypoint.sh which is responsible for (in mentioned order):

executing migrations,
executing our setup management command which creates an initial CRON job configuration,
and initialize the supervisord service (which will manage gunicorn and crond service).

supervisord configuration

I use supervisord to manage the execution of the gunicorn application server and the crond service.

If the application is located in the /usr/src/app directory and gunicorn is installed in /root/.local/bin/gunicorn the could supervisor.conf look like this:

[supervisord]
nodaemon=true

[program:gunicorn]
directory=/usr/src/app
command=/root/.local/bin/gunicorn -b 0.0.0.0:8000 -w 4 my_app.wsgi --log-level=debug --log-file=/var/log/gunicorn.log
autostart=true
autorestart=true
priority=900

[program:cron]
directory=/usr/src/app
command=crond -f
autostart=true
autorestart=true
priority=500
stdout_logfile=/var/log/cron.std.log
stderr_logfile=/var/log/cron.err.log

If you are interested in the details of the configuration, don't hesitate to ask me in the comments or check the hypervisord documentation.

Dockerfile

The minimal Dockerfile has to contain at least the:

copying the application source code,
installing the dependencies,
copying the configuration,
execution of the entry-point.

FROM alpine:3.15

WORKDIR /usr/src/app

# Copy source
COPY . .

# Dependencies
RUN apk add --no-cache python3 supervisor 
RUN pip3 install --user gunicorn
RUN pip3 install --user -r requirements.txt

# Configuration
COPY conf/supervisor.conf /etc/supervisord.conf

# Execution
RUN chmod +x conf/entrypoint.sh
CMD ["conf/entrypoint.sh"]

entrypoint.sh

#!/bin/sh

# Wait until the PostgreSQL is ready
until PGPASSWORD=$DATABASE_PASSWORD psql -h "$DATABASE_HOST" -U "$DATABASE_USER" -c '\q'; do
  >&2 echo "Postgres is unavailable - sleeping"
  sleep 1
done

# Execute migrations
python3 manage.py migrate
# Execute our setup management command which installs CRON jobs
python3 manage.py setup

# Execute supervisord service
supervisord -c /etc/supervisord.conf

For complete example check the EvilFlowersCatalog/EvilFlowersCatalog repository.

Customizable database-based reporting

Jakub Dubec — Sun, 24 May 2020 18:30:16 +0000

My Final Project

I often struggled when I was trying to create customizable reporting tool for my projects. I wanted to have ability to have secure data mining tool which is simple and easy to use.

Of course, there are a lot of project like Metabase or Kibana. These tools are super cool, but sometimes to complicated or hard to customize for special use-case. You can also use GraphQL, but you still need to specify queries on the back-end.

I decited to create a simple JSON-based query language, which is converted into raw SQL query on the server side. Sounds too crazy? Yeah, because it is kinda crazy (because of security and stuff).

My idea was to create subset of DQL (data query language) functionality interpreted as JSON objects on the front-end application. To be sure about security, build query is checked on the API if it's valid and user is trying to access only his data.

This projects was subject to my final thesis.

Link to Code

I created library called duckql, which converts JSON into raw SQL.

For example, JSON object like this:

{
  "obj": "structures.Query",
  "entity": "users",
  "properties": [
    {
      "obj": "properties.Property",
      "name": "users.name"
    },
    {
      "obj": "properties.Property",
      "name": "users.surname"
    }
  ],
  "conditions": {
    "obj": "structures.Comparision",
    "properties": [
      {
        "obj": "properties.Property",
        "name": "users.age"
      },
      {
        "obj": "properties.Constant",
        "value": "15"
      }
    ],
    "operation": "gte"
  }
}

is converted into query like this:

SELECT users.name, users.surname FROM users WHERE (users.age >= 15);

My solution is based on awesome pydantic library.

Source code is avalaible on GitHub: duckql-python.