DEV Community

Cover image for Lessons learned from writing my first python package
Tobias Haindl
Tobias Haindl

Posted on

Lessons learned from writing my first python package

A short summary of lessons I learned from writing my first python packages: taggercore and taggercli.
More infos about them can be found here:

Dependency management

Dependency managemant is crucial for any project.
I could not find a very satisfying solution for separating package dependencies from dev dependencies until I stumbled across this stackoverflow post.
I specified all package dependencies in the
The setup method provides an optional parameter called extras_require. This parameter can help with separating package from dev dependencies.

    extras_require={"dev": ["pytest", "tox", "pytest-cov", "pytest-mock", "black"]}
Enter fullscreen mode Exit fullscreen mode

As you can see above, I specified pytest, tox, black and two pytest plugins as my dev extras.
Now you can simple install the package dependencies including the dev extras by running pip install -e .[dev](or in zsh: pip install -e ".[dev]").
If you do not need the dev dependencies run pip install -e .


extras_require shows its full power when combined with tox.
You only need to specify the key (e.g. "dev") of the extra you want to install. Tox makes sure to install all the dependencies in the environment.


envlist = py38

extras = dev
commands = pytest 
Enter fullscreen mode Exit fullscreen mode

In my opinion this method of dependency management is very comfortable. It was important for me to keep the package dependencies separated from the dev dependencies.
How are you managing your python dependencies?


Get to know mock assertions

At the start of the tagger project I was pretty new to writing tests in python.
It took me quite a bit of time to understand the power of python mock objects.
There are many useful assertions provided by the mock object:

- assert_called
- assert_called_with_once
- assert_any_call
Enter fullscreen mode Exit fullscreen mode

For a full list see the documentation.

If you need to assert multiple calls you can use:
and pass in an array of objects e.g.:


# shortened for readability
Enter fullscreen mode Exit fullscreen mode

Understand how patching works

Patching allows you to manipulate behaviour of an object at runtime. It is a very powerful way to mock out dependencies during testing.

If you are coming from a Java background, like me, you might know the handy mockito library which can be used to create mock objects.
In python this can be achieved with the patch method. However it is a bit tricky to get the patch target right.
I can highly recommend this pycon talk about Demystifying the Patch Function.
The gist: "patch where the object is used - not where it is created"
Let me clarify this statement with an example:

The taggercli package uses classes and functions created in taggercore.
In taggercli/commands/tag you can find:
from taggercore.usecase import scan_region_and_global
How would we mock this call ?
As it is used: taggercli.commands.tag.scan_region_and_global and NOT by patching taggercore.usecase.scan_region_and_global

scan_mock = mocker.patch(
Enter fullscreen mode Exit fullscreen mode

(mocker is a pytest fixture providing a nice API to the mock package)

If you are stuck with debugging a wrongly patched mock object, the dir() command can be helpful.
It can provide hints about how the functions and classes are accessed in your python file.

Embrace the power of pytest fixtures

Inject fixtures into tests

As you can see, especially in taggerlambda/test/, I made extensive use of pytest-fixtures.
I used it to provide simple "fake" data:


def tagging_result(regional_resources, global_resources) -> TaggingResult:
    successful_arns = [resource.arn for resource in regional_resources] + [
        resource.arn for resource in global_resources
    yield TaggingResult(successful_arns, {})
Enter fullscreen mode Exit fullscreen mode

Now the fixture can be used in tests by adding its name to the test parameter list:


def test_lambda_in_tag_mode_env(
        tagging_result # the object is injected by pytest automatically

    mocked_perform_tagging = mocker.patch("src.tagging_lambda.perform_tagging")
    mocked_perform_tagging.return_value = tagging_result

Enter fullscreen mode Exit fullscreen mode

Inject fixtures into fixtures

Do you need a more complex object in your tests consisting of other subobjects?
This can easily be done by reusing a fixture in another fixture.
Take a look at the following example:
Each resource objects can have a list of tag objects:


def tags() -> List[Tag]:
    yield [
        Tag("Project", "CoolProject"),
        Tag("Owner", "Fritz")
Enter fullscreen mode Exit fullscreen mode

The tag fixture can then be reused in another fixture:

def regional_resources(tags) -> List[Resource]:
    yield [
        Resource("some-arn-1", "someq", "queue", tags),
        Resource("some-arn-2", "someq2", "queue", tags)
Enter fullscreen mode Exit fullscreen mode

Manipulate env variables

If you are using environment variables in the code under test, you can easily provide a fully configured environment via pytest-fixture.

def env_for_tag_mode_env(monkeypatch):
    config = {
        "ACCOUNT_ID": "111111111111",
        "TAG_MODE": "ENV"
    monkeypatch.setenv("ACCOUNT_ID", config["ACCOUNT_ID"])
    monkeypatch.setenv("TAG_MODE", config["TAG_MODE"])
    yield config
Enter fullscreen mode Exit fullscreen mode

I'm yielding the config dict to let the assertion part of the test code know how the environment currently is configured.

The monkeypatch fixture is included in pytest and comes with useful methods like setenv and delenv

That's all from me for now!
Thanks for reading my article and I hope you can profit from my summary :)

Top comments (0)