DEV Community: Libor Jelinek

VSCode language mode association for subfolders and special characters folders

Libor Jelinek — Thu, 31 Oct 2024 07:45:35 +0000

Visual Studio Code (VSCode) has tons of file format supported. Based on the filename and extension, VSCode finds language mode. Often, provided sensible defaults are okay. But what if you want different language mode for different folders?

VSCode has in its .vscode/settings.json file a files.associations object. It's a map of glob pattern to language mode. See supported glob pattern syntax in VSCode docs.

How to change language mode for all files anywhere

The simplest variant. All files need different language mode.

For example, when styling with Tailwind CSS and you install Tailwind CSS extension for VSCode, you want to change all *.css files to tailwindcss mode instead of plain css.

{
  "files.associations": {
    "*.css": "tailwindcss"
  }
  ...
}

How to change language mode for specific paths

It's also not uncommon, that files under certain subfolder needs different language mode.

For example, when theming Sphinx documentation, you create a lot of HTML files which are not pure HTML, but Jinja HTML. I install Better Jinja extension which provides jinja-html mode.

However, the path relative to workspace will not work:

"source/_templates/*.html": "jinja-html``

It's because when a path separator (/) appears in a glob pattern, VSCode interpret it as absolute path.

"/Users/liborjelinek/git-dm-oss/sphinx-themes/sphinx_*_theme/src/sphinx_*_theme/styles/input.css": "tailwindcss",

Mmm very impractical. Trick here is to place ** at the very beginning. Two asterisk means any path, including none. Fixed:

"**/source/_templates/*.html": "jinja-html``

Being creative with ** is beneficial. E.g., to associate all *.html anywhere but only under certain path? Use ** again:

{
  "files.associations": {
    "*.css": "tailwindcss",
    "**/source/_templates/**/*.html": "jinja-html`
  }
  ...
}

How to escape special characters for language mode `files.associations`?

Now the tricky part which I found during preparing starter template new Sphinx Themes based on Cookiecutter template generator.

Cookiecutter projects uses folder named {{ cookiecutter.project_slug }}. Only under it, I want to change association, e.g. for *.py from py to jinja-py.

Escaping glob pattern special characters (like *, {, etc.) isn't described in the VSCode docs, but it's possible.

Firstly, these will NOT work:

"**/{{ cookiecutter.project_slug }}/**/*.py": "jinja-py" because { and } are glob pattern's grouping characters.
"**/{{{{ cookiecutter.project_slug }}}}/**/*.py": "jinja-py" because doubling to escape isn't VSCode glob pattern escaping mechanism
**/\{\{ cookiecutter.project_slug \}\}/**/*.py": "jinja-py" because for backslash escape, the \ is escape character but to JSON itself (for example, \n).

The escape the backslash will do the trick ("**/\\{\\{ cookiecutter.project_slug \\}\\}/**/*.css": "tailwindcss"). For example:

{
  "files.associations": {
    // CSS files are Tailwind
    "**/\\{\\{ cookiecutter.project_slug \\}\\}/**/*.css": "tailwindcss",
    // HTML files are Jinja
    "**/\\{\\{ cookiecutter.project_slug \\}\\}/**/*.html": "jinja-html",
    "**/\\{\\{ cookiecutter.project_slug \\}\\}/**/*.py": "jinja-py",
    "**/\\{\\{ cookiecutter.project_slug \\}\\}/**/*.toml": "jinja-toml",
    "**/\\{\\{ cookiecutter.project_slug \\}\\}/**/*.json": "jinja-json"
  },
  ...
}

Pytest and PostgreSQL: Fresh database for every test (part II)

Libor Jelinek — Sun, 01 Sep 2024 08:34:29 +0000

In the previous post, we created the Pytest fixture which will create/drop Postgres database before/after the test method. In this part, I want to improve the fixture to be more flexible and configurable with the help of Pytest factory fixtures.

Limits of static fixture

For example, if you have more than one database to mock in the test

def test_create_user(test_db1, test_db2):
    ...

you must create almost two identical fixtures:

TEST_DB_URL = "postgresql://localhost"
TEST_DB1_NAME = "test_foo"
TEST_DB2_NAME = "test_bar"

@pytest.fixture
def test_db1():
    with psycopg.connect(TEST_DB_URL, autocommit=True) as conn:
        cur = conn.cursor()

        cur.execute(f'DROP DATABASE IF EXISTS "{TEST_DB1_NAME}" WITH (FORCE)')
        cur.execute(f'CREATE DATABASE "{TEST_DB1_NAME}"')

        with psycopg.connect(TEST_DB_URL, dbname=TEST_DB1_NAME) as conn:
            yield conn

        cur.execute(f'DROP DATABASE IF EXISTS "{TEST_DB1_NAME}" WITH (FORCE)')

@pytest.fixture
def test_db2():
    with psycopg.connect(TEST_DB_URL, autocommit=True) as conn:
        cur = conn.cursor()

        cur.execute(f'DROP DATABASE IF EXISTS "{TEST_DB2_NAME}" WITH (FORCE)')
        cur.execute(f'CREATE DATABASE "{TEST_DB2_NAME}"')

        with psycopg.connect(TEST_DB_URL, dbname=TEST_DB2_NAME) as conn:
            yield conn

        cur.execute(f'DROP DATABASE IF EXISTS "{TEST_DB2_NAME}" WITH (FORCE)')

Pytest fixture factories

"Static" fixtures are a bit limiting here. When needed almost the same with just a slight difference, you need to duplicate a code. Hopefully, the Pytest has a concept of factories as fixtures.

Factory fixture is a fixture which returns another fixture. Because, like every factory, it is a function, it can accept arguments to customize returned fixtures. By convention, you can prefix them with make_*, like make_test_db.

Specialized fixtures

The only argument to our fixture factory make_test_db will be a test database name to create/drop.

So, let's create two "specialized" fixtures based on the make_test_db factory fixture.

The usage will looks like:

@pytest.fixture
def test_db_foo(make_test_db):
    yield from make_test_db("test_foo")

@pytest.fixture
def test_db_bar(make_test_db):
    yield from make_test_db("test_bar")

Sidenote: `yield from`

Did you notice the yield from? There is a key difference between yield and yield from on how they handle the flow of data and control within generators.

In Python, both yield and yield from are used within generator functions to produce a sequence of values, but

yield is used to pause a generator function's execution and return a single value to the caller.
while yield from is used to delegate the generation of values to another generator. It essentially "flattens" the nested generator, passing its yielded values directly to the caller of the outer generator.

I.e., we don't want to "yield" from a specialized fixture but from a fixture factory. Therefore yield from is required here.

Fixture factory to create/drop database

Changes required to our original fixture creating/dropping database are actually almost none except the wrapping the code to the inner function.

@pytest.fixture
def make_test_db():
    def _(test_db_name: str):
        with psycopg.connect(TEST_DB_URL, autocommit=True) as conn:
            cur = conn.cursor()

            cur.execute(f'DROP DATABASE IF EXISTS "{test_db_name}" WITH (FORCE)') # type: ignore
            cur.execute(f'CREATE DATABASE "{test_db_name}"') # type: ignore

            with psycopg.connect(TEST_DB_URL, dbname=test_db_name) as conn:
                yield conn

            cur.execute(f'DROP DATABASE IF EXISTS "{test_db_name}" WITH (FORCE)') # type: ignore

    yield _

Bonus: Rewrite migrations fixture as factory fixture

In the previous part, I also had a fixture applying Yoyo migrations to just created empty database. It was also not very flexible. Let's do the same and wrap actual code to the inner function.

In this case, because the code doesn't need to do cleanup after return from test method (no yield in it), the

factory fixture returns (not yield) inner function
specialized fixture calls (not yield from) factory fixture

@pytest.fixture
def make_yoyo():
    """Applies Yoyo migrations to test DB."""
    def _(test_db_name: str, migrations_dir: str):
        url = (
            urlparse(TEST_DB_URL)
            .
            _replace(scheme="postgresql+psycopg")
            .
            _replace(path=test_db_name)
            .geturl()
        )

        backend = get_backend(url)
        migrations = read_migrations(migrations_dir)

        if len(migrations) == 0:
            raise ValueError(f"No Yoyo migrations found in '{migrations_dir}'")

        with backend.lock():
            backend.apply_migrations(backend.to_apply(migrations))

    return _

@pytest.fixture
def yoyo_foo(make_yoyo):
    migrations_dir = str(Path(__file__, "../../foo/migrations").resolve())
    make_yoyo("test_foo", migrations_dir)

@pytest.fixture
def yoyo_bar(make_yoyo):
    migrations_dir = str(Path(__file__, "../../bar/migrations").resolve())
    make_yoyo("test_bar", migrations_dir)

A test method which need two databases and apply migrations to them:

from psycopg import Connection

def test_get_new_users_since_last_run(
        test_db_foo: Connection,
        test_db_bar: Connection,
        yoyo_foo,
        yoyo_bar):
    test_db_foo.execute("...")
    ...

Conclusion

Building your own fixture factory creating and dropping databases for the Pytest method is actually a good exercise to practice Python generator and yield/yield from operators.

I hope this article helped you with your own database test suite. Feel free to leave me your question in the comments and happy coding!

Pytest and PostgreSQL: Fresh database for every test (part I)

Libor Jelinek — Sun, 18 Aug 2024 16:03:14 +0000

In Pytest, everyone's favorite Python testing framework, a fixture is a reusable piece of code that arranges something before the test enters, and cleanup after it quits. For example, a temporary file or folder, setups environment, starting a web server, etc. In this post, we will look at how to create a Pytest fixture which creates a test database (empty or with known state) that gets cleaned up, allowing each test to run on a completely clean database.

The goals

We will create a Pytest fixture using Psycopg 3 to prepare and clean up the test database. Because an empty database is rarely helpful for testing, we will optionally apply Yoyo migrations (at the time of writing website is down, go to archive.org snapshot) to fill it up.

So, requirements for the Pytest fixture named test_db created in this blogpost are:

drop test database if exists before the test
create an empty database before the test
optionally apply migrations or create a test data before the test
provide a connection to the test database to the test
drop test database after the test (even in the case of failure)

Any test method which requests it by listing it a test method argument:

def test_create_admin_table(test_db):
    ...

Will receive a regular Psycopg Connection instance connected to the test DB. Test can do whatever it need as with plain Psycopg common usage, e.g.:

def test_create_admin_table(test_db):
    # Open a cursor to perform database operations
    cur = test_db.cursor()

    # Pass data to fill a query placeholders and let Psycopg perform
    # the correct conversion (no SQL injections!)
    cur.execute(
        "INSERT INTO test (num, data) VALUES (%s, %s)",
        (100, "abc'def"))

    # Query the database and obtain data as Python objects.
    cur.execute("SELECT * FROM test")
    cur.fetchone()
    # will return (1, 100, "abc'def")

    # You can use `cur.fetchmany()`, `cur.fetchall()` to return a list
    # of several records, or even iterate on the cursor
    for record in cur:
        print(record)

Motivation & alternatives
It looks like there are some Pytest plugins which promise PostgreSQL fixtures for tests that rely on databases. They might work well for you.

I have tried pytest-postgresql which promises the same. I have tried it before writing my own fixture but I was not able to make it work for me. Maybe because their docs were very confusing to me.

Another, pytest-dbt-postgres, I haven't tried at all.

Project file layout

In classic Python project, the sources lives in src/ and tests in tests/:

├── src
│   └── tuvok
│       ├── __init__.py
│       └── sales
│           └── new_user.py
├── tests
│   ├── conftest.py
│   └── sales
│       └── test_new_user.py
├── requirements.txt
└── yoyo.ini

If you use migrations library like fantastical Yoyo, migration scripts are likely in migrations/:

├── migrations
    ├── 20240816_01_Yn3Ca-sales-user-user-add-last-run-table.py
    ├── ...

Configuration

Our test DB fixture will need a very little configuration:

connection URL - (without database)
test database name - will be recreated for every test
(optionally) migrations folder - migrations scripts to apply for every test

Pytest has a natural place conftest.py for sharing fixtures across multiple files. The fixture configuration will go there too:

# Without DB name!
TEST_DB_URL = "postgresql://localhost"
TEST_DB_NAME = "test_tuvok"
TEST_DB_MIGRATIONS_DIR = str(Path(__file__, "../../migrations").resolve())

You can set these values from the environment variable or whatever suits your case.

Create `test_db` fixture

With knowledge of PostgreSQL and Psycopg library, write the fixture in conftest.py:

@pytest.fixture
def test_db():
    # autocommit=True start no transaction because CREATE/DROP DATABASE
    # cannot be executed in a transaction block.
    with psycopg.connect(TEST_DB_URL, autocommit=True) as conn:
        cur = conn.cursor()

        # create test DB, drop before
        cur.execute(f'DROP DATABASE IF EXISTS "{TEST_DB_NAME}" WITH (FORCE)')
        cur.execute(f'CREATE DATABASE "{TEST_DB_NAME}"')

        # Return (a new) connection to just created test DB
        # Unfortunately, you cannot directly change the database for an existing Psycopg connection. Once a connection is established to a specific database, it's tied to that database.
        with psycopg.connect(TEST_DB_URL, dbname=TEST_DB_NAME) as conn:
            yield conn

        cur.execute(f'DROP DATABASE IF EXISTS "{TEST_DB_NAME}" WITH (FORCE)')

Create migrations fixture

In our case, we use Yoyo migrations. Write apply migrations as another fixture called yoyo:

@pytest.fixture
def yoyo():
    # Yoyo expect `driver://user:pass@host:port/database_name?param=value`.
    # In passed URL we need to
    url = (
        urlparse(TEST_DB_URL)
        .
        # 1) Change driver (schema part) with `postgresql+psycopg` to use
        # psycopg 3 (not 2 which is `postgresql+psycopg2`)
        _replace(scheme="postgresql+psycopg")
        .
        # 2) Change database to test db (in which migrations will apply)
        _replace(path=TEST_DB_NAME)
        .geturl()
    )

    backend = get_backend(url)
    migrations = read_migrations(TEST_DB_MIGRATIONS_DIR)

    if len(migrations) == 0:
        raise ValueError(f"No Yoyo migrations found in '{TEST_DB_MIGRATIONS_DIR}'")

    with backend.lock():
        backend.apply_migrations(backend.to_apply(migrations))

If you want to apply migrations to every test database, require yoyo fixture for test_db fixture:

@pytest.fixture
def test_db(yoyo):
    ...

To apply migration to some test only, require yoyo individually:

def test_create_admin_table(test_db, yoyo):
    ...

Conclusion

Building own fixture to give your tests a clean database was a rewarding experience for me allowing me to delve deeper into both Pytest and Postgres.

I hope this article helped you with your own database test suite. Feel free to leave me the your question in comments and happy coding!

DEV Community: Libor Jelinek

VSCode language mode association for subfolders and special characters folders

How to change language mode for all files anywhere

How to change language mode for specific paths

How to escape special characters for language mode files.associations?

Pytest and PostgreSQL: Fresh database for every test (part II)

Limits of static fixture

Pytest fixture factories

Specialized fixtures

Sidenote: yield from

Fixture factory to create/drop database

Bonus: Rewrite migrations fixture as factory fixture

Conclusion

Pytest and PostgreSQL: Fresh database for every test (part I)

The goals

Project file layout

Configuration

Create test_db fixture

Create migrations fixture

Conclusion

How to escape special characters for language mode `files.associations`?

Sidenote: `yield from`

Create `test_db` fixture