DEV Community: Pavel Loginov

Best Practices for Alembic and SQLAlchemy

Pavel Loginov — Wed, 30 Oct 2024 14:00:00 +0000

In this article, I’ll briefly go over some best practices that help keep projects organized, simplify database maintenance, and prevent common pitfalls when working with Alembic and SQLAlchemy. These techniques have saved me from trouble more than once. Here’s what we’ll cover:

Naming Conventions
Sorting Migrations by Date
Table, Column, and Migration Comments
Data Handling in Migrations without Models
Migration Testing (Stairway Test)
Service for Running Migrations
Using Mixins for Models

1. Naming Conventions

SQLAlchemy allows you to set up a naming convention that’s automatically applied to all tables and constraints when generating migrations. This saves you from manually naming indexes, foreign keys, and other constraints, which makes the database structure predictable and consistent.

To set this up in a new project, add a convention to the base class so that Alembic will automatically use the desired naming format. Here’s an example of a convention that works well in most cases:

from sqlalchemy import MetaData
from sqlalchemy.orm import DeclarativeBase

convention = {
    'all_column_names': lambda constraint, table: '_'.join(
        [column.name for column in constraint.columns.values()]
    ),
    'ix': 'ix__%(table_name)s__%(all_column_names)s',
    'uq': 'uq__%(table_name)s__%(all_column_names)s',
    'ck': 'ck__%(table_name)s__%(constraint_name)s',
    'fk': 'fk__%(table_name)s__%(all_column_names)s__%(referred_table_name)s',
    'pk': 'pk__%(table_name)s',
}

class BaseModel(DeclarativeBase):
    metadata = MetaData(naming_convention=convention)

2. Sorting Migrations by Date

Alembic migration filenames typically start with a revision tag, which can make the order of migrations in the directory appear random. Sometimes it’s useful to keep them sorted chronologically.

Alembic allows customizing the migration filename template in the alembic.ini file with the file_template setting. Here are two convenient naming formats for keeping migrations organized:

Based on date:

file_template = %%(year)d-%%(month).2d-%%(day).2d_%%(rev)s_%%(slug)s

Based on Unix timestamp:

file_template = %%(epoch)d_%%(rev)s_%%(slug)s

Using date or Unix timestamps in filenames keeps migrations organized, making navigation easier. I prefer using Unix timestamps, and an example will be provided in the next section.

3. Comments for Tables and Migrations

For those working in a team, commenting attributes is a good practice. With SQLAlchemy models, consider adding comments directly to columns and tables instead of relying on docstrings. This way, comments are available both in the code and the database, making it easier for DBAs or analysts to understand table and field purposes.

class Event(BaseModel):
    __table_args__ = {'comment': 'System (service) event'}

    id: Mapped[uuid.UUID] = mapped_column(
        UUID(as_uuid=True),
        primary_key=True,
        comment='Event ID - PK',
    )
    service_id: Mapped[int] = mapped_column(
        sa.Integer,
        sa.ForeignKey(
            f'{IntegrationServiceModel.__tablename__}.id',
            ondelete='CASCADE',
        ),
        nullable=False,
        comment='FK to integration service that owns the event',
    )
    name: Mapped[str] = mapped_column(
        sa.String(256), nullable=False, comment='Event name'
    )

It’s also helpful to add comments to migrations to make them easier to find in the file system. A comment can be added with -m <comment> when generating the migration. The comment will appear in the docstring and the filename. This naming makes it much easier to locate the required migration.

1728372261_c0a05e0cd317_add_integration_service.py 
1728372272_a1b4c9df789d_add_user.py
1728372283_f32d57aa1234_update_order_status.py  
1728372294_9c8e7ab45e11_create_payment.py
1728372305_bef657cd9342_remove_old_column_from_users.py

4. Avoid Using Models in Migrations

Models are often used for data manipulations, such as transferring data from one table to another or modifying column values. However, using ORM models in migrations can lead to issues if the model changes after the migration is created. In such cases, a migration based on the old model will break when executed, as the database schema may no longer match the current model.

Migrations should be static and independent of the current state of models to ensure correct execution regardless of code changes. Below are two ways to avoid using models for data manipulations.

Use raw SQL for data manipulation:

def upgrade():
   op.execute(
       "UPDATE user_account SET email = CONCAT(username, '@example.com') WHERE email IS NULL;"
   )

def downgrade():
   op.execute(
       "UPDATE user_account SET email = NULL WHERE email LIKE '%@example.com';"
   )

Define Tables Directly in the Migration: If you want to use SQLAlchemy for data manipulations, you can manually define tables directly in the migration. This ensures a static schema at the time of migration execution and will not depend on changes in the models.

from sqlalchemy import table, column, String

def upgrade():
    # Define the user_account table to work with data
    user_account = table(
        'user_account',
        column('id'),
        column('username', String),
        column('email', String)
    )

    # Get a connection to the database
    conn = op.get_bind()

    # Select all users without an email
    users = conn.execute(
        user_account.select().where(user_account.c.email == None)
    )

    # Update email for each user
    for user in users:
        conn.execute(
            user_account.update().where(
              user_account.c.id == user.id
            ).values(
              email=f"{@example.com">user.username}@example.com"
            )
        )

def downgrade():
    user_account = table(
        'user_account',
        column('id'),
        column('email', String)
    )

    conn = op.get_bind()
    # Remove email for users added in the upgrade
    conn.execute(
        user_account.update().where(
          user_account.c.email.like('%@example.com')
        ).values(email=None)
    )

5. Stairway Test for Migration Testing

The Stairway Test involves progressively testing upgrade/downgrade migrations step-by-step to ensure the entire migration chain works correctly. This ensures each migration can successfully create a new database from scratch and downgrade without issues. Adding this test to CI is invaluable for teams, saving time and frustration.

Integrating the test into your project can be done easily and quickly. You can find a code example in this repository. It also includes other valuable migration tests that may be helpful.

6. Migration Service

A separate service for performing migrations. This is just one way to carry out migrations. When developing locally or in environments similar to development, this method fits in well. I’d like to remind you about the conditional depends_on feature, which is relevant here. We take the application image with Alembic and run it in a separate container. We add a dependency on the database with the condition that migrations start only when the database is ready to handle requests (service_healthy). Additionally, a conditional depends_on (service_completed_successfully) can be added for the application, ensuring it starts only after migrations have completed successfully.

  db:
    image: postgres:15
    ...
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 10s
      start_period: 10s

  app_migrations:
    image: <app-image>
    command: [
      "python",
      "-m",
      "alembic",
      "-c",
      "<path>/alembic.ini",
      "upgrade",
      "head"
    ]
    depends_on:
      db:
        condition: service_healthy

  app:
    ...
    depends_on:
      app_migrations:
        condition: service_completed_successfully

The depends_on condition ensures migrations run only after the database is fully ready and that the application starts after migrations are completed.

7. Mixins for Models

While this may be an obvious point, it’s important not to overlook it. Using mixins is a convenient way to avoid code duplication. Mixins are classes that contain frequently used fields and methods, which can be integrated into any models where they're needed. For instance, we often need created_at and updated_at fields to track the creation and update times of records. It can also be useful to use an id based on UUID to standardize primary keys. All of this can be encapsulated in mixins.

import uuid
from sqlalchemy import Column, DateTime, func
from sqlalchemy.dialects.postgresql import UUID

class TimestampMixin:
    created_at = Column(
        DateTime,
        server_default=func.now(),
        nullable=False,
        comment="Record creation time"
    )
    updated_at = Column(
        DateTime,
        onupdate=func.now(),
        nullable=True,
        comment="Unique record identifier"
    )

class UUIDPrimaryKeyMixin:
    id = Column(
        UUID(as_uuid=True),
        primary_key=True,
        default=uuid.uuid4,
        comment="Unique record identifier"
    )

By adding these mixins, we can include UUID id and timestamps in any model where needed:

class User(UUIDPrimaryKeyMixin, TimestampMixin, BaseModel):
    __tablename__ = 'user'
    # Other columns...

Conclusion

Handling migrations can be challenging, but following these simple practices helps keep projects well-organized and manageable. Naming conventions, date sorting, comments, and testing have saved me from chaos and helped prevent mistakes. I hope this article proves helpful — feel free to share your own migration tips in the comments!

Wait for Services to Start in Docker Compose: wait-for-it vs Healthcheck

Pavel Loginov — Tue, 15 Oct 2024 13:00:00 +0000

When developing applications using Docker Compose, it's common to encounter situations where one service must wait for another to be ready before starting. For example, a web application might need to wait for a database or another dependent service to be up and running. In this post, we'll explore two popular ways to solve this problem:

Using the wait-for-it.sh script
Leveraging depends_on and healthchecks in Docker Compose

We'll create a simple Python application for demonstration, but don't worry if you're not familiar with Python—you can just copy the code.

Setting up the Environment

Let's start by creating a working directory:

mkdir waitforit
cd waitforit

Next, we’ll build a simple aiohttp-based web application with a /healthz endpoint for health checks. We’ll also add a start delay using an environment variable SLEEP_BEFORE_START, which allows us to simulate service startup delays. The delay can be set by passing the number of seconds in this variable.

File: app.py

import os
import time
from aiohttp import web

async def healthz(request):
    return web.Response(text="OK")

if __name__ == "__main__":
    sleep_time = int(os.getenv("SLEEP_BEFORE_START", 0))
    print(f"Sleeping for {sleep_time} sec before starting...")
    time.sleep(sleep_time)

    app = web.Application()
    app.add_routes([web.get("/healthz", healthz)])
    print("Starting...")
    web.run_app(app, host="0.0.0.0", port=8080)

Now, let's create a Dockerfile for the application.

File: Dockerfile

FROM python:3.9-slim

WORKDIR /app
RUN pip install aiohttp \
    && apt-get update \
    && apt-get install -y curl
COPY app.py .
CMD ["python", "-u", "app.py"]

Next, we’ll create a docker-compose.yml file to launch two services:

web – the web app with a startup delay.
dependent – a service that needs to wait until web is fully up.

File: docker-compose.yml

version: "3.8"

services:
  web:
    image: sleep-web-app
    build: .
    environment:
      SLEEP_BEFORE_START: 10

  dependent:
    image: sleep-web-app
    depends_on:
      - web

Let's start the containers and observe the result:

docker compose up --build

You’ll notice that dependent starts before web is fully ready:

web-1        | Sleeping for 10 sec before starting...
dependent-1  | Sleeping for 0 sec before starting...
dependent-1  | Starting...
web-1        | Starting...

Our goal is to configure Docker Compose so that dependent only starts after web is fully ready. The desired log should look like this:

web-1        | Sleeping for 10 sec before starting...
web-1        | Starting...
dependent-1  | Sleeping for 0 sec before starting...
dependent-1  | Starting...

Using `wait-for-it.sh`

wait-for-it.sh is a simple Bash script that blocks service execution until another service becomes available, checking the availability of a specific port. It’s useful for waiting on databases, web services, or any other TCP services.

The script needs to be run before the main command to start the application. This can be done in the entrypoint or by directly inserting it before the start command. In our example, we'll use the second method for simplicity.

As arguments, it is enough to pass the host:port whose availability we expect. The script will then wait for 15 seconds (the default value). To set your own wait time, you can add the argument --timeout <sec>. If the port becomes available within that time, the script will stop, and the service will continue to start. However, if the host:port remains unavailable after the timeout, the service will still start. To avoid this and stop the launch with an error status, you must pass the --strict flag.

Here’s how to download and use it:

curl -o wait-for-it.sh https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh
chmod +x wait-for-it.sh

Let's add a dependent volume to the service, through which we will pass the script. We will also add the application startup command using wait-for-it.sh. The script will wait for the web service to be ready on port 8080. The main command to start the service is executed after --. You can experiment with the parameters and different startup scenarios. We will consider an example where we have a rough idea of how long to wait for the web service (10 seconds) and will leave the timeout at the default (15 seconds).

  dependent:
    image: sleep-web-app
    command:
      - ./wait-for-it.sh
      - web:8080
      - --
      - python
      - -u
      - app.py
    volumes:
      - ./wait-for-it.sh:/app/wait-for-it.sh

Run the services again:

docker-compose up

The log shows that dependent now waits for web to be available before starting:

dependent-1  | wait-for-it.sh: waiting 15 seconds for web:8080
web-1        | Sleeping for 10 sec before starting...
web-1        | Starting...
dependent-1  | wait-for-it.sh: web:8080 is available after 11 seconds
dependent-1  | Starting...

Pros of `wait-for-it.sh`:

Easy to implement: just download and run before the main command.
Flexible: you can wait for any port on any service, even external ones.
Simple: no additional configuration needed.

Cons:

It only checks port availability, not the full readiness of the service (e.g., database schema initialization). The next approach can solve this.

Using `depends_on` and `healthcheck`

Another option is to use Docker Compose’s depends_on in combination with healthcheck. This approach waits for a service to pass a health check before starting dependent services.

In this approach, the service must have a healthcheck that checks its readiness to handle requests, while the dependent service should have depends_on with a condition based on the healthcheck of the first service. In simple terms, the dependent service starts when the first service is healthy (ready to handle requests).

Let's add a healthcheck for the web service to check its readiness via the /healthz endpoint. This way, we will know that not only is the port open, but the service is also ready to process requests. You can read more about healthcheck here. In test, we specify the command that checks the service's health. The command could simply check if the port is open, which would be equivalent to wait-for-it.sh. However, the essence of healthcheck is not just to check the port, but to verify that the service is ready to handle requests.

  web:
    image: sleep-web-app
    build: .
    environment:
      SLEEP_BEFORE_START: 10
    healthcheck:
      test: ["CMD", "curl", "http://web:8080/healthz"]
      interval: 10s # How often the status will be checked
      retries: 5 # How many times to check before considering it unavailable
      start_period: 10s # How long after startup to begin checks
      timeout: 10s # Timeout for each test run

We will add depends_on with a condition to the dependent service. The dependent service relies on web, but it will only start after the web service successfully passes the health check. You can read more about conditional depends_on here. It's worth mentioning that not all versions of Compose support conditional depends_on.

Final Compose:

version: "3.8"

services:
  web:
    image: sleep-web-app
    build: .
    environment:
      SLEEP_BEFORE_START: 10
    healthcheck:
      test: ["CMD", "curl", "http://web:8080/healthz"]
      interval: 10s
      retries: 5
      start_period: 10s
      timeout: 10s

  dependent:
    image: sleep-web-app
    depends_on:
      web:
        condition: service_healthy

Run the services:

docker-compose up

This time, the dependent service will start execution only after the health check of the web service via /healthz is successful.

web-1        | Sleep 10 sec before start...
web-1        | Starting...
dependent-1  | Sleep 0 sec before start...
dependent-1  | Starting...

Popular services have ready-made solutions for healthcheck:

postgresql: ["CMD", "pg_isready", "-U", "postgres"]
kafka: ["CMD-SHELL", "kafka-broker-api-versions.sh --bootstrap-server localhost:9092"]
redis: ["CMD", "redis-cli", "ping"]
mysql: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
mongodb: ["CMD", "mongo", "--eval", "db.adminCommand('ping')"]
elasticsearch: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"]

Pros of `depends_on` + `healthcheck`:

Health checks give a more reliable indication of service readiness compared to port checks.
It’s a built-in Docker Compose solution — no need for extra scripts.
Easier to manage chains of service dependencies.

Cons:

More configuration required.
Not all images come with health checks by default, and custom scripts may be needed.
It’s harder to check external services outside of Compose.

Which Method to Choose?

The choice between wait-for-it.sh and depends_on with healthcheck depends on your use case:

Use wait-for-it.sh for quick and simple port availability checks, especially in environments where you have more control over dependencies.
Use depends_on + healthcheck for a more reliable, built-in Docker Compose solution that ensures full service readiness.

Conclusion

Managing service readiness in Docker Compose is crucial for building reliable multi-service applications. Whether you opt for the simplicity of wait-for-it.sh or the more robust depends_on with health checks, both methods help ensure that services wait for dependencies to be fully ready before starting. By choosing the right strategy, you can avoid potential issues and make service startup smoother in your Docker-based applications.

User-Friendly Django Pagination

Pavel Loginov — Sun, 31 Dec 2023 18:02:26 +0000

Pagination is a common practice in web development used to divide large datasets into smaller and more manageable parts displayed on multiple pages.

Let's consider a successful example of pagination in Django using the Paginator class. This type of pagination is user-friendly and usually suitable for most cases.

This type of pagination allows you to:

Move to the next and previous pages using the Prev and Next buttons;
Move to the first and last pages from any page;
Navigate by skipping one page at a time.

Pagination in Django

You can find detailed information about pagination in Django from the following links:

In summary, pagination in Django is accomplished using the Paginator and Page classes. The Paginator class divides a large dataset into pages, taking a set of objects (list, tuple, or QuerySet) and the page size as parameters. The Page class represents the current page in pagination and contains the page objects and information about neighboring pages.

You can create a paginator manually or automatically using Class-Based Views. The template is passed the page object (page_obj), which contains the objects, page information, and the paginator itself.

Attributes used in the presented pagination:

page_obj.has_previous: Checks if there is a previous page.
page_obj.has_next: Checks if there is a next page.
page_obj.previous_page_number: Returns the number of the previous page.
page_obj.next_page_number: Returns the number of the next page.
page_obj.number: Returns the number of the current page.
page_obj.paginator: Returns an instance of the Paginator class.
page_obj.paginator.num_pages: Returns the total number of pages.
page_obj.paginator.page_range: Returns the range of pages.

The presented pagination also uses the add template filter, which allows adding values to variables directly in the template.

User-Friendly Pagination

The screenshot shows how the paginator will look on different pages. The paginator always displays the extreme pages and two neighboring pages, if they exist. The Prev and Next buttons are also displayed if the current page is not an extreme page.

The following listing shows the implementation of this pagination. Simply insert this code into your project, and it should work. We will examine this listing in more detail.

{% if page_obj.has_previous %}
  <li><a href="?page={{ page_obj.previous_page_number }}">(Prev)</a></li>

  {% if page_obj.number > 3 %}
      <li><a href="?page=1">(1)</a></li>
    {% if page_obj.number > 4 %}
      <li><a >...</a></li>
    {% endif %}
  {% endif %}
{% endif %}

{% for num in page_obj.paginator.page_range %}
  {% if page_obj.number == num %}
      <li><a href="?page={{ num }}">({{ num }})</a></li>
  {% elif num > page_obj.number|add:'-3' and num < page_obj.number|add:'3' %}
      <li><a href="?page={{ num }}">({{ num }})</a></li>
  {% endif %}
{% endfor %}

{% if page_obj.has_next %}
  {% if page_obj.number < page_obj.paginator.num_pages|add:'-3' %}
    <li><a >...</a></li>
    <li><a href="?page={{ page_obj.paginator.num_pages }}">
      ({{ page_obj.paginator.num_pages }})
    </a></li>
  {% elif page_obj.number < page_obj.paginator.num_pages|add:'-2' %}
    <li><a href="?page={{ page_obj.paginator.num_pages }}">
      ({{ page_obj.paginator.num_pages }})
    </a></li>
  {% endif %}

  <li><a href="?page={{ page_obj.next_page_number }}">(Next)</a></li>
{% endif %}

This code snippet is a Django template for creating a pagination control element. Let's go through what each part of this code does:

1. Link to Previous Page:

{% if page_obj.has_previous %}
  <li><a href="?page={{ page_obj.previous_page_number }}">(Prev)</a></li>

If page_obj.has_previous is True, it shows the Prev button to go to the previous page. The variable page_obj.previous_page_number contains the number of the previous page.

2. Links to First Pages and Ellipsis:

{% if page_obj.number > 3 %}
    <li><a href="?page=1">(1)</a></li>
  {% if page_obj.number > 4 %}
    <li><a >...</a></li>
  {% endif %}
{% endif %}

These lines of code handle the display of a link to the first page and an ellipsis (...) if the current page number is greater than 3 (or 4 for the ellipsis).

3. Current Page and Links to Neighboring Pages:

{% for num in page_obj.paginator.page_range %}
  {% if page_obj.number == num %}
      <li><a href="?page={{ num }}">({{ num }})</a></li>
  {% elif num > page_obj.number|add:'-3' and num < page_obj.number|add:'3' %}
      <li><a href="?page={{ num }}">({{ num }})</a></li>
  {% endif %}
{% endfor %}

This loop goes through the page range in the paginator. It checks two conditions:

If the page number (num) is the current page (page_obj.number), it creates a link for it. Typically, this is styled differently to indicate the current page.
If the page number is within three pages before or after the current page, it also creates links to these pages. This allows the user to navigate to the nearest two pages without viewing the entire range.

4. Ellipsis and Link to Last Page:

{% if page_obj.has_next %}
  {% if page_obj.number < page_obj.paginator.num_pages|add:'-3' %}
    <li><a >...</a></li>
    <li><a href="?page={{ page_obj.paginator.num_pages }}">
      ({{ page_obj.paginator.num_pages }})
    </a></li>
  {% elif page_obj.number < page_obj.paginator.num_pages|add:'-2' %}
    <li><a href="?page={{ page_obj.paginator.num_pages }}">
      ({{ page_obj.paginator.num_pages }})
    </a></li>
  {% endif %}

This part is similar to the second section, but for the end of the pagination range. It displays an ellipsis and a link to the last page if the current page is several pages away from the last page.

5. Link to Next Page:

{% if page_obj.has_next %}
...
<li><a href="?page={{ page_obj.next_page_number }}">(Next)</a></li>
{% endif %}

Similar to the first section, but for the button to go to the next page.

Conclusion

We have explored an example of a simple and user-friendly pagination in Django that allows users to navigate through pages and access the desired information easily.

The pagination implementation can be adapted to your needs, and you can make changes to the code to fit your project requirements. Good luck using pagination in your Django applications!

Typing in Python — Strong, Dynamic, Implicit

Pavel Loginov — Wed, 15 Nov 2023 16:00:00 +0000

Everyone knows that in Python, data types are divided into mutable and immutable, but how typing works in Python? To answer this question, we need to look at the characteristics of Python's type system.

Typically, when describing the type system of a programming language, three questions are addressed:

Does the language perform implicit automatic type conversions?
At what stage is the type of a variable determined?
Is it necessary to explicitly declare the type of variables?

By answering these questions, one can determine whether a language has strong or weak typing, static or dynamic typing, and explicit or implicit typing.

Python has strong, dynamic, implicit typing, and let's take a closer look at why.

1. Strong Typing

When we talk about "strong typing" in the context of programming, we're referring to a language's strict approach to handling variables of different types. In the Python programming language, which is characterized by strong typing, different data types are not automatically mixed. For example, the expression "some string" - 3 will cause an error because the language does not allow a string to be implicitly converted to a number to perform a mathematical operation. When encountering type-related errors, Python raises a TypeError exception.

Similarly, attempting to add a list [2, 1, 0] and a set set([2, 23, 2]) will also result in an error, as Python will not look for ways to automatically convert one data structure to another to perform the operation. By contrast, JavaScript, for example, allows for adding strings to numbers without issues: 3 + '1' // Results in the string: '31'.

Despite its strong typing, Python allows certain operations between different data types, but this is due to explicit implementation rather than automatic conversion:

# Sequence repetition:
# You can "multiply" a string or a list by a number,
# resulting in a repeating sequence.

print("word" * 3)  # wordwordword
print([1, 2] * 3)  # [1, 2, 1, 2, 1, 2]

# float and int
print(5 + 0.1)  # 5.1

# bool and number
print(2.2 + True)  # 3.2

# Note: The bool type inherits from int,
# so here the addition operation is straightforward.

# and others...

These operations are possible thanks to the language's internal mechanisms, which provide clear implementation for such cases. For instance, in your program, you can explicitly define the behavior of your type (class) when added to another object using magic methods. However, if you do not do this, you will get an error when attempting to add it to something else.

To illustrate, let's make a string behave like it does in JavaScript: converting the object it's being added to into a string and then concatenate the result.

class JavaScriptStr(str):
    def __add__(self, value):  # Explicit implementation of addition
        return super().__add__(str(value))


number = JavaScriptStr("3")
print(number + 1)  # "31"

2. Dynamic Typing

Dynamic typing in Python means that the data types of variables are determined at runtime, rather than at compile-time as in statically typed languages. This facilitates writing flexible code and allows, for example, the creation of functions that work with various types of data.

Consider the function find, which searches for an element required_element in a sequence sequence. In the language C, to implement the same logic, you would have to write several versions of the function for different data types, while in Python, one is sufficient:

def find(required_element, sequence):
    """Performs a search for an element within a sequence."""
    for index, element in enumerate(sequence):
        if element == required_element:
            return index
    return -1

print(find(2, [1, 2, 3]))  # Outputs: 1
print(find("c", ("a", "b", "c", "d")))  # Outputs: 2

print(find(1, 1))  # Raises a TypeError exception

The downside of dynamic typing is that it can lead to unexpected runtime errors. For example, if we pass a non-iterable object to sequence (which cannot be traversed using for), we will get an error. But we will know about it when the program execution reaches the specific line with the for loop over this object.

Dynamic typing also allows you to overwrite a single variable with data of different types.

number = 1  # int
number = "One"  # str

3. Implicit Typing

In languages with implicit typing, i*t isn't necessary to explicitly declare the type of a variable* — the interpreter determines the type based on the assigned value. This simplifies the code and makes it more readable:

var = 5  # int
var = "some string"  # now var is a str

Type Annotations in Python

Since version 3.5, Python introduced support for type annotations, allowing programmers to explicitly indicate the types of variables and the values returned by functions. This doesn't affect the interpreter's operation but aids in programming and code maintenance:

def plus(x: int, y: int) -> int:
    return x + y

print(plus(5, 10))  # Outputs: 15

# The function will perform string concatenation.
# This shows that annotation is not equal to explicit typing.
print(plus("Hello ", "world!"))  # Outputs: Hello world!

Specifying types through annotations makes the code more understandable for other developers and also allows for the detection of errors before the program is run, using additional static analysis tools. However, the language still retains the flexibility inherent in dynamic typing.

Conclusion

Typing in Python combines rigor and flexibility: on one hand, strong typing protects against type mismatch errors, on the other hand, dynamic typing allows for writing more versatile and easily readable code. Type annotations bring clarity to the program's structure, making it more predictable and safe.

How To Easily Install Docker and Compose on Ubuntu

Pavel Loginov — Tue, 14 Nov 2023 15:15:00 +0000

While Docker is an incredibly powerful tool, setting it up on Ubuntu can sometimes be a complex affair. But fear not! In this post, we'll go through the steps of installing Docker (and Docker Compose) on Ubuntu using a convenient script.

Before we begin the installation process, ensure that you are running on Ubuntu. The steps provided in this guide are written for Ubuntu, but could be applicable to other distributions with some modifications.

TL;DR:

wget https://gist.githubusercontent.com/welel/f80c96482e3b539487b9fa08bfcab86d/raw/90bc2330924d225aef7dc3178f5926bda7daff04/install_docker.sh

sudo chmod +x install_docker.sh

sudo ./install_docker.sh

Here's how to install Docker using a script on Ubuntu:

Step 1: Download the Installation Script

First, you need to download the script that automates the installation process. Open your terminal and enter the following command:

wget https://gist.githubusercontent.com/welel/f80c96482e3b539487b9fa08bfcab86d/raw/90bc2330924d225aef7dc3178f5926bda7daff04/install_docker.sh

This command fetches the necessary script from a Gist and saves it to your current directory.

The script available on Gist. The Gist includes both the script itself and a detailed explanation of its usage. You can view and download the script, as well as find instructions on how to use it, at the following link:

Install Docker on Ubuntu - Installation Script and Guide

This resource makes the installation process straightforward and clear, allowing you to get Docker up and running on your Ubuntu system with minimal fuss.

Step 2: Give the Script Execution Permissions

Before you can run the script, you need to give it permission to execute. Change the permissions with the chmod command:

sudo chmod +x install_docker.sh

Step 3: Execute the Script

Now it's time to run the script. You’ll do this with elevated privileges using sudo:

sudo ./install_docker.sh

Once you have executed the script, keep an eye on the terminal output. If everything goes according to plan, you should see a message stating Docker successfully installed. This confirmation signifies that Docker has been successfully installed on your system, and you're ready to start using it for your containerization needs.

Post-Installation

To ensure that it's working correctly, you can run:

docker --version

docker compose version

This command should output the version of Docker that you've just installed and Docker Compose version.

Furthermore, it's a good practice to add your user to the docker group, so you can execute Docker commands without using sudo:

sudo usermod -aG docker ${USER}

Remember to log out and log back in for this change to take effect.

With the above script, installing Docker on Ubuntu is almost as easy as pie. Remember to check the Docker documentation for more details on how to use Docker and its myriad of features. Happy Dockering!

SOLID Principles with Examples in Python

Pavel Loginov — Mon, 13 Nov 2023 15:00:00 +0000

Let's examine the SOLID principles through clear examples of Python code and simplify complex definitions into human language.

Note: The term “client” appears in the text. Client means a programming entity that uses another programming entity (e.g. one class uses another class inside, so the first class is a client).

SOLID is an acronym for a set of design principles created for developing software using object-oriented languages.

The SOLID principles are designed to encourage the creation of code that is simpler, more dependable, and easier to enhance. Each letter in SOLID stands for one design principle.

When implemented correctly, this makes your code more extensible, logical, maintainable, and easier to read.

To understand SOLID principles, you must have a good understanding of how interfaces are used.

Let's look at each principle one by one:

1. Single Responsibility Principle

The Single Responsibility Principle requires that each class should have a singular, clearly defined purpose.

The same applies to other programming entities. That is, it is necessary to decompose software entities so that each entity is responsible for the one job assigned to it. When a class assumes multiple responsibilities, it falls into an anti-pattern known as the God Object.

When a class takes on multiple responsibilities:

it becomes interdependent (modifying the behavior of one class operation leads to a change in another);
code readability declines;
testing becomes complex;
collaborative code development becomes challenging.

# Listing [1.1]
# An example of a class with many responsibilities.

class  User:
    def __init__(self, name):
        self.name = name

    def get_name(self):
        return self.name

    def save(self):
        ...

    def send(self):
        ...

    def log(self):
        ...

We have a User class that handles multiple responsibilities - managing user properties, database operations, data transmission, and logging. If you modify the functionality of one of these tasks within the application, it may necessitate adjustments in others to accommodate the new changes. It's like a domino effect, tip one bone and it will drop everything after it.

In this case, we simply decompose the class, create separate classes that will take on one responsibility.

# Listing [1.2]
# An example of the decomposition of the `User` class.

class User:
    def __init__(self, name):
            self.name = name

    def get_name(self):
        pass


class Storage:
    def save(self, user: User):
        ...


class HttpConnection:
    def send(self, user: User):
        ...


class Logger:
    def log(self, user: User):
        ...

Now our code is better structured. Individual entities are now smaller in size, making them easier to read and easier to work with. Now it is possible to give tasks to several developers to change different components at the same time, and no conflicts should arise.

2. Open-Closed Principle

Software entities (classes, modules, functions, etc.) must be open for extension but closed for modification.

Changing existing code is bad because it has already been tested and works. If we change the code, then we have to do regression testing. Therefore, when adding functionality, you should not change existing entities, but add new ones using composition or inheritance. Even with this approach, you may have to slightly edit the old code in order to prevent bugs or hacky code. But changing the old code should be avoided as much as possible.

Let's consider a scenario where you have an online store, and you offer a 20% discount to your preferred customers using the Discount class. If you decide to double the discount to 40% for VIP clients, you could extend the class as follows:

# Listing [2.1]
# An example of modifying a class when adding new functionality.

class Discount:
  def __init__(self, customer, price):
      self.customer = customer
      self.price = price

  def give_discount(self):
      if self.customer == 'favourite':
          return self.price * 0.2
      if self.customer == 'vip':
          return self.price * 0.4

However, this approach violates the Open-Closed Principle, as the OCP discourages it. For example, if we want to give a new discount to a different type of customer, then this requires adding new logic. To follow the OCP principle, we will add a new class that will extend Discount. And in this new class we implement this logic:

# Listing [2.2]
# An example of adding functionality via inheritance.

class Discount:
    def __init__(self, customer, price):
      self.customer = customer
      self.price = price

    def get_discount(self):
      return self.price * 0.2

class VIPDiscount(Discount):
    def get_discount(self):
      return super().get_discount() * 2

If you decide to give a discount to super VIP users, it will look like this:

# Listing [2.3]
# An example of adding functionality via inheritance (2).

class SuperVIPDiscount(VIPDiscount):
    def get_discount(self):
      return super().get_discount() * 2

Thus, we do not modify the existing code (closed for modification), but add a new one (open for extension).

When you're designing your entity structure, it's essential to identify, at the earliest stages, those system entities that might undergo changes or expansions in the future and create appropriate abstractions for them.

Let's explore another example: we have a Weapon class and a Character class. In this program, the character has a weapon and can perform attacks with it.

# Listing [3.1]
# Example program, not easily extensible.

class Weapon:
    def __init__(self, name, damage):
        self.name = name
        self.damage = damage

    def attack(self):
        print(f"{self.name} strikes: -{self.damage} hp")


class Character:
    def __init__(self, name, weapon: Weapon):
        self.name = name
        self.weapon = weapon

    def change_weapon(self, weapon: Weapon):
        self.weapon = weapon

    def attack(self):
        self.weapon.attack()


sword = Weapon("Needle", 24, 3)
aria = Character("Aria", sword)
aria.attack() # Output: Needle strikes: -24 hp

Now, we've made the decision to introduce a new weapon, the bow. This requires us to modify the Weapon.attack method and incorporate an additional type field to expand the output logic (changing "strikes" to "shoots" for the bow).

# Listing [3.2]
# An example of adding new functionality in violation of OCP.

class Weapon:
    def __init__(self, _type, name, damage):
        self.type = _type
        self.name = name
        self.damage = damage

    def attack(self):
        if self.type == "striking":
            print(f"{self.name} skrikes: -{self.damage} hp")
        elif self.type == "shooting":
            print(f"{self.name} shoots: -{self.damage} hp")


sword = Weapon("striking", "Needle", 24, 3)
aria = Character("Aria", sword)
aria.attack() # Output: Needle strikes: -24 hp

bow = Weapon("shooting", "Thread", 30, 100)
aria.change_weapon(bow)
aria.attack() # Output: Thread shoots: -30 hp

As we discussed above, this approach violates the OCP. When writing the Weapon class, its extension for another gun types was not provided in advance. From the very beginning it was worth creating more abstract code.

# Listing [3.3]
# An example of a program that is amenable to extension.

class Attacker:
    """Interface for attacking classes."""
    def attack(): raise NotImplementedError


class Weapon(Attacker):
    """Defines a general structure for weapons."""
    def __init__(self, name, damage):
        self.name = name
        self.damage = damage


class Sword(Weapon):
    """
    Inherits the structure of the weapon
    and implements the attack interface.
    """
    def attack(self):
        print(f"{self.name} stirkes: -{self.damage} hp")


class Bow(Weapon):
    def attack(self):
        print(f"{self.name} shoots: -{self.damage} hp")


sword = Sword("Needle", 24, 3)
bow = Bow("Thread", 30, 100)

aria = Character("Aria", sword)
aria.attack() # Output: Needle strikes: -24 hp

aria.change_weapon(bow)
aria.attack() # Output: Thread shoots: -30 hp

Code structured in this manner is more amenable to expansion, appears cleaner, and more professional. It should be noted that if you are absolutely sure that you will not have additional functionality in the future, then it is better to stick to the KISS (Keep It Short and Simple) principle and not create additional abstractions.

3. Liskov Substitution Principle

The main idea behind the Liskov Substitution Principle is that for any class, the client should be able to use any subclass of the base class without noticing the difference between them.

And therefore without any change in the execution behavior of the program. This means that the inherited class should complement, not replace, the behavior of the parent, and that the client is completely isolated and unaware of changes in the class hierarchy.

Subtype Requirement:
Let *φ(x)** be a property provable about objects x of type T. Then φ(y) should be true for objects y of type S where S is a subtype of T.*

In simpler terms, if you disrupt the logic of the parent class within the child class, you are violate the LSP principle.

# Listing [4]
# An example of a program that violates the LSP principle.

class Develpoer:
    def write_code(self): ...


class Backend(Developer):
    def configure_server(self): ...


class DevOps(Developer):
    """
    Let's imagine a scenario where our DevOps team lacks coding skills.
    """
    def monitor_resources(self): ...

    def write_code(self):
        """
        We alter the implementation, consequently violating the LSP.
        """
        raise UnableToDo("DevOps cannot write code.")

In listing above, the DevOps class violated the logic of his parent, thereby violating the LSP principle. According to this principle, a client who uses Developer should be able to replace it with any child class and not break the program. In the case of a child class DevOps, the program will raise an error.

The following example demonstrates the client's ability to use a class and its children without breaking program logic.

# Listing [5]
# An example of a program that follows the LSP principle.

from dataclasses import dataclass


@dataclass
class Position:
    x: int = 0
    y: int = 0

    def __str__(self):
        return f"({self.x}, {self.y})"


class Character:
    """Base class of characters."""
    def __init__(self, name):
        self.name = name
        self.position = Position()

    def move(self, destination: Position):
        print("{name} moves from {start} to {end}".format(
            name=self.name, start=self.position, end=destination
        ))
        self.position = destination


class Human(Character):
    """Child class that follows parent's logic."""
    def move(self, destination: Position):
        print("{name} goes from {start} to {end}".format(
            name=self.name, start=self.position, end=destination
        ))
        self.position = destination

    def buy(self):
        """Adds aditional logic."""
        print("Buys an item.")


class Dragon(Character):
    """Child class that follows parent's logic."""
    def move(self, destination: Position):
        print("{name} flies from {start} to {end}".format(
            name=self.name, start=self.position, end=destination
        ))
        self.position = destination

    def attack(self):
        """Adds aditional logic."""
        print("Spews fire at the enemy.")


def move(character: Character, destination: Position):
    """
    A client that uses `Character` and its subclasses
    without noticing the difference.
    """
    character.move(destination)


spirit = Character("Spirit")
john = Human("John")
drogon = Dragon("Drogon")

meeting_point = Position(x=300, y=250)

move(spirit, meeting_point)
move(john, meeting_point)
move(drogon, meeting_point)

# Output:
# Spirit moves from (0, 0) to (300, 250)
# John goes from (0, 0) to (300, 250)
# Drogon flies from (0, 0) to (300, 250)

As we can see, the move function can work both with Character and its subclasses without errors.

LSP is the basis of good object-oriented software design because it aligns with one of the fundamental principles of Object-Oriented Programming (OOP): polymorphism. The point is to create correct hierarchies such that classes derived from the base are polymorphic for their parent in relation to the methods of its interfaces. It is also interesting to note how this principle relates to the example of the previous principle. If we try to extend a class with a new incompatible class, then everything will break. Interaction with the client will be broken, and as a result, such an extension will not be possible (or, in order to make this possible, we would have to violate another principle and modify the client code, which should be closed for modification, this is highly undesirable and unacceptable).

Thoughtfully considering new classes in accordance with LSP facilitates the proper expansion of class hierarchies. Furthermore, LSP contributes to the Open-Closed Principle.

4. Interface Segregation Principle

Clients should not depend on interfaces they do not use. You shouldn't force a client to implement an interface that it does not use.

Create thin interfaces: many client-specific interfaces are better than one general-purpose interface. This principle eliminates the disadvantages of implementing large interfaces.

To illustrate this, let's take the following example. Suppose we were tasked with creating a Smartphone. We initially created the Device interface to accommodate it and future devices. Later, we needed to add a Laptop that couldn't make phone calls. At this point, we should recognize that our Device interface contradicts the Interface Segregation Principle, and it should be split. However, if we were unaware of ISP, we might have written Laptop as shown in Listing [6.1]. Then, when the task arose to add a Phone, we would also violate the principle. The resulting code would look like this:

# Listing [6.1]
# An example of a program that violates ISP.

# In Listing 6.*, by ... we mean a missing method implementation.

class Device:
    def call(self): raise NotImplementedError
    def send_file(self): raise NotImplementedError
    def browse_internet(self): raise NotImplementedError


class Smartphone(Device):
    def call(self): ...
    def send_file(self): ...
    def browse_internet(self): ...


class Laptop(Device):
    def call(self):
        raise BadOperation("A laptop cannot make calls.")

    def send_file(self): ...
    def browse_internet(self): ...


class Phone(Device):
    def call(self): ...

    def send_file(self):
        raise BadOperation("A phone cannot send files.")

    def browse_internet(self):
        raise BadOperation("A phone cannot access internet.")

This is a clear illustration of the dependence of the Laptop and Phone clients on the Device interface, which they only partially implement.

A nice trick is that in our business logic, a single class can implement multiple interfaces when necessary. This allows us to provide a unified implementation for all shared methods across interfaces. In Python, this is easily achieved through multiple inheritance:

# Listing [6.2]
# An example of a program that complies with ISP.

class CallDevice:
    def call(self): raise NotImplementedError


class FileTransferDevice:
    def send_file(self): raise NotImplementedError


class InternetDevice:
    def send_file(self): raise NotImplementedError


class Smartphone(CallDevice, FileTransferDevice, InternetDevice):
    def call(self): ...
    def send_file(self): ...
    def browse_internet(self): ...


class Laptop(FileTransferDevice, InternetDevice):
    def send_file(self): ...
    def browse_internet(self): ...


class Phone(CallDevice):
    def call(self): ...

We now see fine-grained interfaces and eliminate methods within software entities that they do not use. This results in more predictable behavior, and the code becomes less tightly coupled.

Segregated interfaces force us to think more about our code from the client's point of view, which will lead us to less dependency and easier testing. This way, not only did we make our code better for the client, but it also made it easier for us to understand, test, and implement the code for ourselves.

5. Dependency Inversion Principle

Dependency should be on abstractions, not on specifics.

Modules at higher levels should not depend on modules at low levels. Both upper and lower level classes must depend on the same abstractions. Abstractions should not depend on details. Details must depend on abstractions.

As development progresses, there comes a point where our application predominantly consists of modules. At this stage, it becomes essential to enhance our code using dependency injection. The functionality of high-level components relies on low-level components. You can utilize inheritance or interfaces to achieve specific behaviors.

Let's look at a bad example first. Suppose we have a Post entity, and we've assigned three programmers to implement various storage solutions for posts. Unfortunately, they didn't agree on naming conventions and created storages with different method names. This is problematic, because the entity that will use the storages to save posts is highly dependent on the specific storage implementation and will have to adapt to each repository each time they change.

# Listing [7.1]
# An example of unorganized code.

class Post:
    title: str
    content: str


class PostLocalStorage:
    def fetch_all(self): ...
    def get_one(self): ...
    def save(self): ...


class PostCacheDict:
    def get_all(self): ...
    def get(self): ...
    def set(self): ...


class PostDBStorage:
    def select_all(self): ...
    def select_one(self): ...
    def insert(self): ...

The first step to structuring your code and getting rid of dependencies is to create a common interface for storages.

# Listing [7.2]
    # An example of a common interface for storage classes.

class Storage:
    def get_all(self): raise NotImplementedError
    def get(self): raise NotImplementedError
    def save(self): raise NotImplementedError


class PostLocalStorage(Storage):
    def get_all(self): ...
    def get(self): ...
    def save(self): ...

# Other storages also inherit from `Storage`.
...

Now all storages use the same method names, which allows the client to use the repository without knowing its type. However, an even better approach is to introduce an abstraction and interact directly with it. This abstraction will receive a storage object and delegate calls to specific methods to the underlying storage.

# Listing [7.3]
# An example of introducing a general abstraction for storage.

class StorageClient(Storage):
    def __init__(self, storage: Storage):
        self.storage = storage

    def get_all(self):
        return self.storage.get_all()

    def get(self, *args):
        return self.storage.get(*args)

    def save(self, *args):
        return self.storage.save(*args)

With this approach:

the client always works with the storage abstraction StorageClient;
it offers a clear and transparent interface;
the client remains independent of the specific storage implementation.

Note: storage client != StorageClient (storage client is the one who saves posts using StorageClient).

If we reflect the final program on the definition, then the client now depends on the StorageClient abstraction rather than on specific implementations like PostLocalStorage, and so on. The top-level module (storage client) is independent of lower-level modules (storage implementations). Both upper (storage client) and lower (storage implementation) classes depend on the same abstraction - StorageClient. StorageClient does not depend on storage implementation details, it simply delegates the execution of common interface methods. The implementation details depend on and are guided by StorageClient.

By incorporating these principles into your software development process, you can enhance the quality of your codebase, making it easier to scale and adapt to evolving requirements. Embracing SOLID principles is not just a best practice; it's a path to crafting software that stands the test of time.

DEV Community: Pavel Loginov

Best Practices for Alembic and SQLAlchemy

1. Naming Conventions

2. Sorting Migrations by Date

3. Comments for Tables and Migrations

4. Avoid Using Models in Migrations

5. Stairway Test for Migration Testing

6. Migration Service

7. Mixins for Models

Conclusion

Wait for Services to Start in Docker Compose: wait-for-it vs Healthcheck

Setting up the Environment

Using wait-for-it.sh

Pros of wait-for-it.sh:

Cons:

Using depends_on and healthcheck

Pros of depends_on + healthcheck:

Cons:

Which Method to Choose?

Conclusion

User-Friendly Django Pagination

Pagination in Django

User-Friendly Pagination

1. Link to Previous Page:

2. Links to First Pages and Ellipsis:

3. Current Page and Links to Neighboring Pages:

4. Ellipsis and Link to Last Page:

5. Link to Next Page:

Conclusion

Typing in Python — Strong, Dynamic, Implicit

1. Strong Typing

2. Dynamic Typing

3. Implicit Typing

Type Annotations in Python

Conclusion

How To Easily Install Docker and Compose on Ubuntu

Step 1: Download the Installation Script

Step 2: Give the Script Execution Permissions

Step 3: Execute the Script

Post-Installation

SOLID Principles with Examples in Python

1. Single Responsibility Principle

2. Open-Closed Principle

3. Liskov Substitution Principle

4. Interface Segregation Principle

5. Dependency Inversion Principle

Using `wait-for-it.sh`

Pros of `wait-for-it.sh`:

Using `depends_on` and `healthcheck`

Pros of `depends_on` + `healthcheck`: