Developer Service

Posted on Aug 24 Originally published at developer-service.blog

Your Data Structure Is Too Flexible

#python #pydantic #data #structures

Python makes it easy to pass data around as dicts, nested JSON blobs, and “whatever the client sent”.

That flexibility is useful when you own the shape. It gets expensive when the data crosses a trust boundary - an HTTP body, a webhook, a file from another team - and you treat it like it already matches your assumptions.

You may already know Pydantic. This article is not a tour of fields and validators.

It is about a design rule that is easy to skip when the happy path works: validate at the boundary, then move on.

Keep flexibility only where you chose it.

Staging lied (inbound API JSON)

Imagine a small HTTP handler that accepts a JSON body as a Python dict and charges a user’s account. ledger below is a stand-in for whatever typed client or service call sits deeper in your stack.

In staging, every client you control sends the same clean shape, so the happy path is the only path you ever see.

def charge_user(payload: dict) -> None:
    user_id = payload["user"]["id"]
    amount = payload["amount"]
    # Somewhere deeper in the stack — expects an int user_id
    ledger.debit(user_id=user_id, amount=amount)

Staging payload:

{
    "user": {"id": 42, "email": "dev@example.com"},
    "amount": 19.99,
}

That works. user.id is an int, amount is a float, and ledger.debit never complains.

Then production gets a request from a mobile client, a partner integration, or a retry queue that re-serialized the body. The JSON is still “valid”.

The shape is almost right:

{
    "user": {"id": "42", "email": "buyer@example.com"},
    "amount": 19.99,
}

Nothing fails in the handler. It forwards user_id as whatever JSON gave it.

The failure shows up later - in ledger.debit, a SQL parameter binder, or a comparison that assumes an int. If that deeper call type-checks or binds user_id as int, a str raises there.

Worse: the wrong type slips into storage and you notice even later.

The stack trace points at infrastructure code. The bug was at the door: user.id crossed the trust boundary as a string, and a flexible dict let it through.

Staging did not lie about the feature. It lied about the contract.

The cost of flexible-by-default

A dict is patient. It will carry a string where you meant an int, a missing nest where you meant a required object, or a None where you meant a value.

And it will do it without complaining at the boundary.

Type annotations on the parameter help your editor and your teammates. They do not stop the wrong shape from entering the process.

“Almost the right shape” is worse than obviously wrong data. Staging taught everyone what the payload usually looks like.

Optional fields, defaulted nests, and defensive .get() calls then paper over the cases that don’t match. The contract becomes tribal knowledge instead of something the code can reject.

The consequence arrives later. You debug three layers down while the handler that accepted the body already returned.

Catching bad data at the door costs a validation error. Catching it downstream costs time, a misleading stack trace, and confidence in every other flexible boundary you still have.

The rule: strict at trust boundaries

ONE THING: Strict at edges; flexibility only where you chose it.

A trust boundary is any place data enters your process from something you do not fully control.

For this article, that means inbound API JSON. The same idea applies to config files, environment variables, and messages from another service - not shown here.

Inside the app, after the boundary has done its job, you can pass richer objects around, reshape data, or keep temporary dicts for a transform. That flexibility is earned. It is not the default for untrusted input.

If you already use Pydantic and it still feels like boilerplate before the “real” code, flip the framing.

The model is the contract.

The handler that runs after validation is the easy part, because the hard part already happened at the door.

One refactor

Same payload, different door. Instead of accepting a naked dict, define the contract once and validate before anything else runs.

from pydantic import BaseModel, ValidationError


class User(BaseModel):
    id: int
    email: str


class ChargeRequest(BaseModel):
    user: User
    amount: float


def charge_user(payload: dict) -> None:
    request = ChargeRequest.model_validate(payload)
    ledger.debit(user_id=request.user.id, amount=request.amount)

Feed it the production body where user.id was "42". In Pydantic v2’s default (lax) mode, that string is coerced to int at the boundary.

Strict mode would reject it instead - either way, the decision happens at the door. For this example: lax turns "42" into 42; a value like "not-an-id" fails in both modes:

try:
    ChargeRequest.model_validate(
        {
            "user": {"id": "not-an-id", "email": "buyer@example.com"},
            "amount": 19.99,
        }
    )
except ValidationError as e:
    print(e)

You get a ValidationError immediately, pointing at user.id, before ledger.debit is called. Trimmed, the message looks like this:

1 validation error for ChargeRequest
user.id
  Input should be a valid integer, unable to parse string as an integer
  [type=int_parsing, input_value='not-an-id', input_type=str]

By the time ledger.debit runs, request.user.id is already an int - the conversion happened at the boundary, not three layers down in a binder that was never meant to diagnose your API contract.

That is the whole point: fail at the trust boundary, with a message about the field that broke the contract. In an HTTP API, map that to a 400 or 422 with the validation details, not a 500 from deeper in the stack.

If you use FastAPI, the framework can wire the same models to the request body for you. The design rule does not change, only who calls model_validate.

What this isn't

Skip the conclusion that you should never use dicts. Dicts are fine for owned data, temporary transforms, and code that runs after a boundary has already checked the shape. The problem is treating untrusted JSON as if it were already trusted.

This article is also not a complete Pydantic course. You saw one nested model, model_validate, and a ValidationError at the door. That is enough to prove the design rule. It is not enough to cover custom validators, settings, serialization, or performance.

TypedDict and dataclasses are not the enemy either. Those tools document and structure data in useful ways. They do not, by themselves, give you the same runtime contract check at a trust boundary. That check is the point of this article.

Go deeper

If that design rule makes sense and you want the longer path - custom validators, nested models, settings, APIs, and how Pydantic compares to the alternatives - then read Practical Pydantic.

It is written for Python developers, FastAPI users, and anyone tired of bad data surviving staging and failing in production.

Keep the rule either way: validate at the trust boundary, then move on. Stay flexible only where you chose it.

DEV Community