DEV Community: Alexey

Pydantic: The end of manual validations! ✨

Alexey — Sat, 23 Nov 2024 11:50:21 +0000

Pydantic is a data validation and settings management library for Python. It uses Python type hints to validate and parse data, ensuring that your code works with properly structured and typed data. By leveraging Python’s dataclass-like model structure, Pydantic makes it easy to define schemas for complex data and automatically validate and serialize/deserialize data in a clean, Pythonic way. Let's explore the main features:

Data Validation

Automatically validate input data against a schema using Python's type hints.

from pydantic import BaseModel, ValidationError

class User(BaseModel):
    id: int
    name: str
    email: str

# Valid input
user = User(id=1, name="John Doe", email="john@example.com")
print(user)

# Invalid input
try:
    user = User(id="not-an-integer", name="Jane", email="jane@example.com")
except ValidationError as err:
    print(err)

Whenever you want to define data model, use pydantic.BaseModel!

Function Validation

Pydantic provides powerful tools for validating not just data models but also the input and output of functions. This is achieved using the @validate_call decorator, allowing you to enforce strict data validation for function arguments and return values. If the provided arguments or return type don’t match the expected types, a ValidationError is raised.

from pydantic import validate_call

@validate_call
def greet(name: str, age: int) -> str:
    return f"Hello {name}, you are {age} years old."

# Valid input
print(greet("Alice", 30))  # Output: Hello Alice, you are 30 years old.

# Invalid input
try:
    greet("Bob", "not-a-number")
except Exception as e:
    print(e)

By enabling the validate_return flag in @validate_call, Pydantic will also validate the return value of the function against its annotated return type. This ensures the function adheres to the expected output schema.

from pydantic import validate_call

@validate_call(validate_return=True)
def calculate_square(number: int) -> int:
    return number ** 2  # Correct return type

# Valid input and return
print(calculate_square(4))  # Output: 16

# Invalid return value
@validate_call(validate_return=True)
def broken_square(number: int) -> int:
    return str(number ** 2)  # Incorrect return type

try:
    broken_square(4)
except Exception as e:
    print(e)

Parsing

Pydantic can parse complex nested structures, including JSON data, into model objects.

from pydantic import BaseModel
from typing import List

class Item(BaseModel):
    name: str
    price: float

class Order(BaseModel):
    items: List[Item]
    total: float

# JSON-like data
data = {
    "items": [
        {"name": "Apple", "price": 1.2},
        {"name": "Banana", "price": 0.8}
    ],
    "total": 2.0
}

order = Order(**data) 
print(order) # items=[Item(name='Apple', price=1.2), Item(name='Banana', price=0.8)] total=2.0

Serialization and Deserialization

Pydantic models can be serialized into JSON or dictionaries and reconstructed back.

from pydantic import BaseModel

class User(BaseModel):
    id: int
    name: str
    email: str

# Create a model instance
user = User(id=1, name="Alice", email="alice@example.com")

# Serialize to dictionary and JSON
user_dict = user.model_dump()
user_json = user.model_dump(mode='json')

print("Dictionary:", user_dict)
print("JSON:", user_json)

# Deserialize back to the model
new_user = User.model_validate(user_json)
print("Parsed User:", new_user)

Flexible Validation

Data validation is not force-type validation. For example, if you define a model with id, due_date, and priority fields of types int, bool, and datetime respectively, you can pass:

numerical string as id
ISO-8601, UTC or strings of the other date formats as due_date
'yes'/'no', 'on'/'off', 'true'/'false', 1/0 etc. as priority

from sensei import APIModel
from datetime import datetime


class Task(APIModel):
    id: int
    due_date: datetime
    priority: bool


task = Task(due_date='2024-10-15T15:30:00', id="1", priority="yes")
print(task)

The result will be

Task(id=1, due_date=datetime.datetime(2024, 10, 15, 15, 30), priority=True)

Custom Validation

You can also define custom validation logic in your model using validators. They allow you to apply more complex validation rules that cannot be easily expressed using the built-in types or field constraints. Validator is defined through the field_validator decorator or Field object. You can pass one or more field names to field_validator, to determine what fields will use this validator, or '*' to apply validator for every field.

from typing import Any
from pydantic import Field, field_validator, EmailStr, BaseModel

class User(BaseModel):
    id: int
    username: str = Field(pattern=r'^\w+$')
    email: EmailStr
    age: int = Field(18, ge=14)
    is_active: bool = True
    roles: list[str]

    # Define validator executed 'before' internal parsing
    @field_validator('roles', mode='before')
    def _validate_roles(cls, value: Any):
        return value.split(',') if isinstance(value, str) else value

user = User(id=1, username='john', email='john@example.com', roles='student,singer')
print(user) # id=1 username='john' email='john@example.com' age=18 is_active=True roles=['student', 'singer']

Open-source Projects

There are a lot of open-source projects powered by Pydantic. Let's explore the best of them:

FastAPI

One of the most prominent use cases of Pydantic is in FastAPI, a modern web framework for building APIs with Python. FastAPI uses Pydantic models extensively for request body validation, query parameters, and response schemas.

Source: https://github.com/fastapi/fastapi
Docs: https://fastapi.tiangolo.com

Sensei

While FastAPI is designed for building APIs, Sensei is designed for wrapping these APIs quickly and easy. API Clients powered by Sensei ensure users they will get relevant data models and will not get confusing errors.

Source: https://github.com/CrocoFactory/sensei
Docs: https://sensei.crocofactory.dev

SQLModel and Typer

SQLModel and Typer are two remarkable projects developed by Sebastián Ramírez, the creator of FastAPI.

SQLModel is a library designed to streamline database interactions in Python applications. Built on top of SQLAlchemy and Pydantic, SQLModel combines the power of an ORM with the convenience of data validation and serialization.

Source: https://github.com/fastapi/sqlmodel
Docs: https://sqlmodel.tiangolo.com

Typer is a framework for creating command-line interface (CLI) applications using Python. It simplifies the process by using Python's type hints to automatically generate user-friendly CLI commands and help text.

Source: https://github.com/fastapi/typer
Docs: https://typer.tiangolo.com

HTTPX: Dump requests library in a junkyard 🚀

Alexey — Fri, 22 Nov 2024 21:56:26 +0000

The most common mistake in HTTP requests in Python is still using the requests library. The httpx library is a more advanced, versatile, and modern HTTP client for Python, supporting both synchronous and asynchronous workflows. It's easy to transition from the outdated requests library, which lacks critical features like native async support.

Install httpx:

pip install httpx

Async/Sync

httpx supports both synchronous and asynchronous clients, offering flexibility for various project environments.

requests is sync-only, making it unsuitable for high-performance applications.
aiohttp is async-only, which lacks simplicity for synchronous needs.

Synchronous Example

import httpx

# Synchronous GET request
response = httpx.get('https://api.example.com/data')
print(response.json())  # Parse JSON data

# Using a persistent Client for better performance
with httpx.Client(base_url='https://api.example.com') as client:
    response = client.get('/data')
    print(response.json())

Asynchronous Example

import httpx
import asyncio

async def fetch_data():
    # Asynchronous GET request
    async with httpx.AsyncClient(base_url='https://api.example.com') as client:
        response = await client.get('/data')
        print(response.json())

# Run the async function
asyncio.run(fetch_data())

Similar to `requests`

For those familiar with requests, transitioning to httpx is straightforward, as both libraries share similar APIs.

With requests:

import requests

response = requests.get('https://api.example.com/data')
data = response.json()

With httpx:

import httpx

response = httpx.get('https://api.example.com/data')
data = response.json()

Both libraries share similar method names (get, post, etc.), making migration easy.

Client Management

The concept of a Client in httpx is analogous to a Session in requests. However, httpx.Client is more powerful and efficient. You can read the article from the httpx documentation, to learn more about httpx.Client.

Features

Performance: Clients reuse underlying connections, making multiple requests faster.
Fine-grained control: Manage headers, cookies, timeouts, and other settings.
Asynchronous support: Async clients (httpx.AsyncClient) are optimized for modern async workflows.

Perfomance Explanation

When you make a simple request, like the following:

import httpx

response = httpx.get('https://example-api.com', params={'page': 1})
print(response.json())

httpx follows this algorithm:

If you make dozens of requests, like this code:

import httpx

urls = [...]
params_list = [{...}, ...]

for url, params in zip(urls, params_list):
    response = httpx.get(url, params=params)
    print(response.json())

httpx will open the client for each request. It slows your application. The better solution is to use a single client instance. You can close it whenever you want.

import httpx

urls = [...]
params_list = [{...}, ...]

with httpx.Client() as client:
    for url, params in zip(urls, params_list):
        response = client.get(url, params=params)
        print(response.json())

Sensei Integration

The sensei Python framework uses httpx as its HTTP client for creating efficient, robust, and flexible API Clients (or API Wrappers). You can inject your own client into each request and close it whenever you want, to achieve the best performance:

from sensei import Manager, Router, Client

manager = Manager()
router = Router('httpx://example-api.com', manager=manager)

@router.get('/users/{id_}')
def get_user(id_: int) -> User:
    pass

with Client(base_url=router.base_url) as client:
    manager.set(client)

    for i in range(10):
        user = get_user(i)
        print(user)

    manager.pop()

Visit sensei repository to read more: https://github.com/CrocoFactory/sensei

Documentation

httpx provides rich, well-organized documentation site with examples and guides, making learning and troubleshooting easier and code docstrings explaining every function and parameter.

With these advantages and code examples, it's clear why httpx is the modern choice for HTTP requests in Python. Replace your requests code with httpx today and enjoy the benefits of a cutting-edge library! 🚀

Sensei: Simplify API Client Generation 🚀

Alexey — Wed, 20 Nov 2024 15:10:22 +0000

Sensei simplifies the process of creating API Clients by handling routing, data validation, and response mapping automatically. This reduces the complexity of HTTP requests, making it easier to integrate APIs into your projects without writing boilerplate code.

Sensei uses type hints to generate API clients, providing clear interfaces and robust validation for interacting with APIs. Its syntax is very similar to the framework FastAPI

Documentation: https://sensei.crocofactory.dev
Source Code: https://github.com/CrocoFactory/sensei

Code Example

from typing import Annotated
from sensei import Router, Path, APIModel

router = Router('https://pokeapi.co/api/v2/')


class Pokemon(APIModel):
    name: str
    id: int
    height: int
    weight: int


@router.get('/pokemon/{name}')
def get_pokemon(name: Annotated[str, Path(max_length=300)]) -> Pokemon: 
    pass

pokemon = get_pokemon(name="pikachu")
print(pokemon) # Pokemon(name='pikachu' id=25 height=4 weight=60)

Didn't it seem to you that the function doesn't contain the code? Sensei writes it instead of you! The result of the call get_pokemon(name="pikachu") is the object Pokemon(name='pikachu' id=25 height=4 weight=60) containing pokémon's data.

There is a wonderful OOP approach proposed by Sensei:

class User(APIModel):
    email: EmailStr
    id: PositiveInt
    first_name: str
    last_name: str
    avatar: AnyHttpUrl

    @classmethod
    @router.get('/users')
    def query(
            cls,
            page: Annotated[int, Query()] = 1,
            per_page: Annotated[int, Query(le=7)] = 3
    ) -> list[Self]:
        pass

    @classmethod
    @router.get('/users/{id_}')
    def get(cls, id_: Annotated[int, Path(alias='id')]) -> Self: 
        pass

    @router.post('/token')
    def login(self) -> str: 
        pass

    @login.prepare
    def _login_in(self, args: Args) -> Args:
        args.json_['email'] = self.email
        return args

    @login.finalize
    def _login_out(self, response: Response) -> str:
        return response.json()['token']

user = User.get(1)
user.login() # User(id=1, email="john@example.com", first_name="John", ...)

When Sensei doesn't know how to handle a request, you can do it yourself, using preprocessing as prepare and postprocessing as finalize

Comparison

Sensei 👍: It provides a high level of abstraction. Sensei simplifies creating API wrappers, offering decorators for easy routing, data validation, and automatic mapping of API responses to models. This reduces boilerplate and improves code readability and maintainability.

Bare HTTP Client 👎: A bare HTTP client like requests or httpx requires manually managing requests, handling response parsing, data validation, and error handling. You have to write repetitive code for each endpoint.

Features

Sensei provides features that are useful for both standard and messy APIs:

Validation 🛡️
Rate Limiting Handling ⏳
Automatic Handling of Return Types 🔄
DRY Architecture Without Duplications 🧹
Async Support ⚡
Case Conversion and Aliases 🔠
Own Client for Fast Requests 🚀

Target Audience

Developers Working with APIs, Data Scientists and Analysts, etc.

DEV Community: Alexey

Pydantic: The end of manual validations! ✨

Data Validation

Function Validation

Parsing

Serialization and Deserialization

Flexible Validation

Custom Validation

Open-source Projects

FastAPI

Sensei

SQLModel and Typer

HTTPX: Dump requests library in a junkyard 🚀

Async/Sync

Synchronous Example

Asynchronous Example

Similar to requests

Client Management

Features

Perfomance Explanation

Sensei Integration

Documentation

Sensei: Simplify API Client Generation 🚀

Code Example

Comparison

Features

Target Audience

Similar to `requests`