Sitri = Vault + Pydantic: continuation of the saga, local development.

Background

In the previous article I wrote about how to configure your application using Sitri, however, I missed the point with local development, since you will agree that it is not very convenient deploying a Vault locally, and storing a local config in a common Vault, especially if several people are working on a project, is doubly inconvenient.

In Sitri this problem is solved quite simply - using the local mode for your settings classes, that is, you do not even have to rewrite anything or duplicate code, and the structure json file for local mode will almost completely repeat the structure of secrets.

So, well, now, let's add literally a couple of lines of code to our project + I'll show you how you can work with all this if your project is run locally in docker-compose...

Preparing the code

To begin with, let's agree that local_mode is true when ENV = "local" :)

Next, I propose to slightly edit our provider_config.py and create a BaseConfig class there from which we will inherit in our Config settings classes. We will do this in order not to duplicate the code, that is, the settings classes themselves will contain only what is specific to them.

import hvac
from sitri.providers.contrib.system import SystemConfigProvider
from sitri.providers.contrib.vault import VaultKVConfigProvider
from sitri.settings.contrib.vault import VaultKVSettings

configurator = SystemConfigProvider(prefix="superapp")
ENV = configurator.get("env")

is_local_mode = ENV == "local"
local_mode_file_path = configurator.get("local_mode_file_path")


def vault_client_factory() -> hvac.Client:
    client = hvac.Client(url=configurator.get("vault_api"))

    client.auth_approle(
        role_id=configurator.get("role_id"),
        secret_id=configurator.get("secret_id"),
    )

    return client


provider = VaultKVConfigProvider(
    vault_connector=vault_client_factory,
    mount_point=f"{configurator.get('app_name')}/{ENV}",
)


class BaseConfig(VaultKVSettings.VaultKVSettingsConfig):
    provider = provider
    local_mode = is_local_mode
    local_provider_args = {"json_path": local_mode_file_path}

A bit about local_provider_args in this field we specify the arguments for creating an instance of JsonConfigProvider, they will be validated and this dictionary must match the schema, so don't worry - this is not some dirty trick. However, if you want to create an instance of the local provider yourself, then you just put it in the optional local_provider field.

Now, we can easily inherit the config classes from the base one. For example, a settings class for connecting to Kafka would look like this:

from typing import Any, Dict

from pydantic import Field
from sitri.settings.contrib.vault import VaultKVSettings

from superapp.config.provider_config import BaseConfig, configurator


class KafkaSettings(VaultKVSettings):
    mechanism: str = Field(..., vault_secret_key="auth_mechanism")
    brokers: str = Field(...)
    auth_data: Dict[str, Any] = Field(...)

    class Config(BaseConfig):
        default_secret_path = "kafka"
        default_mount_point = f"{configurator.get('app_name')}/common"

        local_mode_path_prefix = "kafka"

As you can see, the required changes are minimal. local_mode_path_prefix we specify that the structure of the general config is saved in our json file. Now, let's write this json-file for the local configuration:

{
    "db":
    {
        "host": "testhost",
        "password": "testpassword",
        "port": 1234,
        "user": "testuser"
    },
    "faust":
    {
        "agents":
        {
            "X":
            {
                "concurrency": 2,
                "partitions": 5
            }
        },
        "app_name": "superapp-workers",
        "default_concurrency": 5,
        "default_partitions_count": 10
    },
    "kafka":
    {
        "auth_data":
        {
            "password": "testpassword",
            "username": "testuser"
        },
        "brokers": "kafka://test",
        "mechanism": "SASL_PLAINTEXT"
    }
}

... Well, or just copy and paste it from the end of the last article. As you can see, everything is very simple here. For our further research, rename main.py in the root of the project to __main__.py so that you can run the package with a command from docker-compose.

Put the application into the container and enjoy the build

The first thing we should do is write a small Dockerfile:

FROM python:3.8.3-buster

ENV PYTHONUNBUFFERED=1 \
    POETRY_VIRTUALENVS_CREATE=false \
    POETRY_VIRTUALENVS_IN_PROJECT=false \
    POETRY_NO_INTERACTION=1


RUN pip install poetry

RUN mkdir /superapp/
WORKDIR /superapp/

COPY ./pyproject.toml ./poetry.lock /superapp/
RUN poetry install --no-ansi

WORKDIR /

Here we just install the dependencies and that's it, since it is for local development, we do not copy the project code.

Next, we need an env file with the variables required for local-mode:

SUPERAPP_ENV=local
SUPERAPP_LOCAL_MODE_FILE_PATH=/config.json
SUPERAPP_APP_NAME=superapp

As you can see, nothing superfluous, no configuration information is needed for Vault, since in local mode, the application will not even try to "knock" on Vault.

And the last thing we need to write is the docker-compose.yml file itself:

# docker-compose config for local development
version: '3'
services:
  superapp:
    command: python3 -m superapp
    restart: always
    build:
      context: ./
      dockerfile: Dockerfile
    volumes:
      - ./superapp:/superapp
      - ./config.json:/config.json
    env_file:
      - .env.local

Everything is simple here too. We put our json file in the root, as write above in the environment variable for the container.

Now, launch:

docker-compose up

Creating article_sitri_vault_pydantic_superapp_1 ... done
Attaching to article_sitri_vault_pydantic_superapp_1

superapp_1  | db=DBSettings(user='testuser', password='testpassword', host='testhost', port=1234) faust=FaustSettings(app_name='superapp-workers', default_partitions_count=10, default_concurrency=5, agents={'X': AgentConfig(partitions=5, concurrency=2)}) kafka=KafkaSettings(mechanism='SASL_PLAINTEXT', brokers='kafka://test', auth_data={'password': 'testpassword', 'username': 'testuser'})

superapp_1  | {'db': {'user': 'testuser', 'password': 'testpassword', 'host': 'testhost', 'port': 1234}, 'faust': {'app_name': 'superapp-workers', 'default_partitions_count': 10, 'default_concurrency': 5, 'agents': {'X': {'partitions': 5, 'concurrency': 2}}}, 'kafka': {'mechanism': 'SASL_PLAINTEXT', 'brokers': 'kafka://test', 'auth_data': {'password': 'testpassword', 'username': 'testuser'}}}

As you can see, everything started successfully and the information from our json file successfully passed all checks and became settings for the local version of the application, yuhhu!

Code of this "continuation" I put in a separate branch of the repository, so you can take a look at how it all looks after the changes: branch