DEV Community: Izabela Kowal

What is the best string formatting technique for logging in Python?

Izabela Kowal — Fri, 23 Apr 2021 22:10:55 +0000

First, let's do a quick recap of all of the possible string formatting techniques in Python:

printf-style, % operator, oldschool



# example with %s format specifier
>>> name = "Guido"
>>> "Hello %s" % name
'Hello Guido'

# version with a mapping
>>> year = 2021
>>> "Hello %(name)s from year %(year)d" % {"name": name, "year": year}
'Hello Guido from year 2021'

Template Strings, since Python 2.4



from string import Template

t = Template("Hello $name")

>>> t.substitute(name=name)
'Hello Guido'

t = Template("Hello $name from year $year")

>>> t.substitute(name=name, year=year)
'Hello Guido from year 2021'

str.format, since Python 3.0



>>> "Hello, {}".format(name)
'Hello, Guido'

>>> "Hello {name} from year {year}".format(name=name, year=year)
'Hello Guido from year 2021'

f-Strings, Python 3.6+



>>> f"Hello {name}"
'Hello Guido'

>>> f"Hello {name} from year {year}"
'Hello Guido from year 2021'

The f-Strings method is the most recent, the fastest and usually the most convenient way of formatting strings in Python.

Here's a great article where you can learn more about string formatting in Python:

https://realpython.com/python-string-formatting/

The logging module

The logging module was added to the Python standard library way back when the only way of formatting strings was to use the printf formatting technique. As the documentation states, at least for now, there's no way to change that while maintaining backwards compatibility. The logger methods (debug(), info(), error() etc.) take positional parameters for the logging message. Therefore, the recommended way of string formatting is to pass the actual message with string formatting specifiers (%s, %r, %x etc.) and the other arguments we want to append to the main message.



import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

name = "Guido"

>>> logger.info("Hello %s from logger!", name)
INFO:__main__:Hello Guido from logger!

# NOTE: not this way, you need to pass args to the info method:
# logger.info("Hello %s from logger!" % name)

Although the logging methods accept other formatting styles and the official Python logging cookbook gives us a few workarounds, printf technique, in particular, is recommended by the docs. Let's focus now on the two main advantages of this approach.

Performance

Passing the message arguments using the printf-style enables postponing the formatting to the very last moment before the message is outputted via the handler. Therefore, the interpolation won't be applied unless the logging level is sufficient enough. This might come quite handy when dealing with some heavy computational data:



import logging
from dataclasses import dataclass

logging.basicConfig(level=logging.ERROR)
logger = logging.getLogger(__name__)


@dataclass
class Developer:
    name: str

    def __str__(self) -> str:
        print("Some super heavy calculation going on")
        return self.__repr__()


developer = Developer(name="Guido")

# info won't be logged since we've set up log level to error
>>> logger.info(f"Hello {developer} from logger!")
Some super heavy calculation going on

>>> logger.info("Hello %s from logger!", developer)
# prints nothing

On the other hand, it's also questionable. If you need to avoid an automatic call of the str() method of your object, it might be an indicator that something is wrong with your code, and you ultimately need to perform some refactoring. Let's move on to the second advantage of printf-style logging. This one is more obvious and straightforward.

Working with log aggregators

If you're using a log aggregator in your project, like, for example, Sentry (which is awesome!), the printf-style logging becomes quite helpful. If you call the same log statement many times with immediate string interpolation, you will get multiple occurrences of the same log message:



logger.error(f"Something went wrong with {obj}")

However, if you use the printf technique, all of the records will be grouped as occurrences of the same log message, with different values for %s:



logger.error("Something went wrong with %s", obj)

Summary

Considering all the advantages of the printf-style log formatting, is it worth giving up super clean and modern f-Strings? Or is it an overkill? Or maybe you have found some new third party library that handles the string formatting for logging better? Let me know your thoughts below!

7 ways to implement DTOs in Python and what to keep in mind

Izabela Kowal — Mon, 19 Apr 2021 21:54:44 +0000

Data Transfer Objects are simply data structures typically used to pass data between application layers or between services.

The simplest form of DTO in Python can be just a dictionary:

itemdto = {
    "name": "Potion",
    "location": Location("Various"),
    "description": "Recover 20 HP",
}

Issues (among others): mutable, lack of typing, lack of specificity.

We will use here the Location class, which might be a model class managed by an ORM in the real-life scenario:

class Location:
    def __init__(self, name: str) -> None:
        self.name = name

    def __repr__(self) -> str:
        return f"Location(name={self.name})"

    def __eq__(self, other: Location) -> bool:
        return self.name == other.name

If we want to set the attributes of a DTO more precisely, we can define a separate class:

class ItemDto:
    def __init__(self, name: str, location: Location, description: str = "") -> None:
        self.name = name
        self.location = location
        self.description = description

Alternatively, we can use kwargs and a .get() method for optional parameters:

class ItemDto:
    def __init__(self, **kwargs) -> None:
        self.name = kwargs["name"]
        self.location = kwargs["location"]
        self.description = kwargs.get("description", "")

itemdto = ItemDto(
    name="Super Potion",
    location=Location("Various"),
    description="Recover 70 HP"
)

Well defined DTOs can give us more benefits, such as making it easier to perform serialization or validation. Here are a few examples of using different features of Python standard library and 3rd party packages to create better DTOs.

stdlib solutions

dataclasses

added to Python 3.7 (and later backported to Python 3.6)
created using a @dataclass decorator
by default add automatically generated dunder methods __init__, __repr__ and __eq__
__init__ method takes all fields as method parameters and sets their values to instance attributes with the same names:

from dataclasses import dataclass

@dataclass
class ItemDto:
    name: str
    location: Location
    description: str = ""


# support both positional and keyword args
itemdto = ItemDto(
    name="Old Rod",
    location=Location("Vermillion City"),
    description="Fish for low-level Pokemon",
)

generated __repr__ method returns a string containing class name, field names and field representation

>>> print(itemdto)
ItemDto(name='Old Rod', location=Location(name=Vermillion City), description='Fish for low-level Pokemon')

generated __eq__ method compares the class tuples containing field values of the current and the other instance

itemdto2 = ItemDto(
    name="Old Rod",
    location=Location("Vermillion City"),
    description="Fish for low-level Pokemon",
)

>>> itemdto == itemdto2
True

__eq__ method works the same as if we would explicitly declare it this way:

def __eq__(self, other):
    if other.__class__ is self.__class__:
        return (self.name, self.location, self.description) == (other.name, other.location, other.description)
    return NotImplemented

it might be a good idea to make DTO instances immutable. It is possible by setting the argument frozen to True:

@dataclass(frozen=True)
class ItemDto:
    name: str
    location: Location
    description: str = ""

not iterable:

...: for field in itemdto:
...:     print(field)
...: 
TypeError: 'ItemDto' object is not iterable

NamedTuples

NamedTuple is a subclass of regular tuple
introduced in Python 3.0 as a factory method in the collections module:

from collections import namedtuple

ItemDto = namedtuple("ItemDto", ["name", "location", "description"])

added to Python 3.5 as a typed version in typed module and later enhanced with variable annotations syntax in Python 3.6:

from typing import NamedTuple

class ItemDto(NamedTuple):
    name: str
    location: Location
    description: str = ""


# support both positional and keyword args
itemdto = ItemDto(
    "X Speed", "Temporarily raise Speed in battle", Location("Celadon Dept. Store")
)

>>> print(itemdto)
ItemDto(name='X Speed', location='Temporarily raise Speed in battle', description='Celadon Dept. Store')

immutable
__repr__ and __eq__ handled
iterable

...: for field in itemdto:
...:     print(field)
...: 
X Speed
Temporarily raise Speed in battle
Location(name=Celadon Dept. Store)

support default values, although these must be defined after any fields without default values.

TypedDicts

available since Python 3.8:

from typing import TypedDict

class ItemDto(TypedDict):
    name: str
    location: Location
    description: str


itemdto = ItemDto(
    name="Escape Rope,",
    location=Location("Various"),
    description="Teleport to last visited Pokemon Center",
)

>>> print(itemdto)
{'name': 'Escape Rope,', 'location': Location(name=Various), 'description': 'Teleport to last visited Pokemon Center'}

mutable
__repr__ and __eq__ handled
iterable in dict kind of way
don't support default values
can provide typing for existing dictionaries
since those are still dictionaries, after all, they can be directly serialized to JSON data structures (although in this example, we should provide a custom encoder for the Location class).

3rd party packages

attrs

a pytest dependency, so there's a chance you might already have it in your project
similar to the dataclasses, in fact, the attrs library was the basis for designing the dataclasses:

import attr

@attr.s(frozen=True)
class ItemDto:
    name: str = attr.ib()
    location: Location = attr.ib()
    description: str = attr.ib(default="")

# also, the dataclasses syntax!
@attr.dataclass(frozen=True)
class ItemDto:
    name: str
    location: Location
    description: str = ""

attrs provide some extra functionality on top of those that the dataclasses offer, like runtime validation and memory optimization (slotted classes).

Here's an example of runtime validation:

@attr.s(frozen=True)
class PokemonDto:
    name: str = attr.ib()
    type: str = attr.ib(
        validator=attr.validators.in_(
            [
                "Fire",
                "Water",
                "Electric",
                "Poison",  # ...
            ]
        )
    )

>>> PokemonDto("Charmander", "Fire")
PokemonDto(name='Charmander', type='Fire')

>>> PokemonDto("Charmander", "Gyarados")
ValueError: 'type' must be in ['Fire', 'Water', 'Electric', 'Poison'] (got 'Gyarados')

Whether to choose attrs or dataclasses - it ultimately depends on your specific use case and if you are able to use 3rd party packages in your project.

pydantic

FastAPI uses pydantic for schema definition and data validation
pydantic enforces type hints at runtime
the recommended way for creating pydantic models is to subclass pydantic.BaseModel, therefore all models inherit some methods:

from pydantic import BaseModel

class PokemonDto(BaseModel):
    name: str
    type: str

    class Config:
        allow_mutation = False


# enforced keyword arguments in case of BaseModel subclass
pokemondto = PokemonDto(name="Charizard", type="Fire")

like attrs, pydantic also supports vanilla Python dataclasses:

import pydantic

@pydantic.dataclasses.dataclass(frozen=True)
class PokemonDto:
    name: str
    type: str

# in this case positional args are allowed
PokemonDto("Charizard", "Fire")

enables (recursive) data validation:

from enum import Enum

from pydantic import BaseModel

class TypeEnum(str, Enum):
    fire = "Fire"
    water = "Water"
    electric = "Electric"
    poison = "Poison"
    # ...


class PokemonDto(pydantic.BaseModel):
    name: str
    type: TypeEnum

    class Config:
        allow_mutation = False


>>> PokemonDto(name="Charizard", type="Fire")
PokemonDto(name='Charizard', type=<TypeEnum.fire: 'Fire'>)

>>> PokemonDto(name="Charizard", type="Charmeleon")
ValidationError: 1 validation error for PokemonDto
type
  value is not a valid enumeration member; permitted: 'Fire', 'Water', 'Electric', 'Poison' (type=type_error.enum; enum_values=[<TypeEnum.fire: 'Fire'>, <TypeEnum.water: 'Water'>, <TypeEnum.electric: 'Electric'>, <TypeEnum.poison: 'Poison'>])

enables JSON (de)serialization:

>>> PokemonDto(name="Charizard", type="Fire").json()
'{"name": "Charizard", "type": "Fire"}'

Summary

Your choice on how to implement the DTOs depend on multiple circumstances - whether you need, among others:

immutability
default values support
iterability
serialization
runtime type checking
performance optimization
other, more advanced configurability.

Post initially inspired by this Reddit thread.