DEV Community: Olivier

Guardrails for AI-Generated Python Code: A pytest + Hypothesis Setup That Catches What Review Misses

Olivier — Mon, 28 Sep 2026 08:48:11 +0000

AI assistants are very good at writing Python that looks right. It's typed, it's formatted, it comes with tests. The problem is that the tests are usually written by the same model, from the same assumptions, so they pass for the same reasons the code might be wrong.
I've spent a lot of time with Python code that processes sensor and telemetry data, and at some point I stopped trusting review alone to catch this. Here's the setup I use now. It's three layers, and each one catches a different kind of mistake.

The failure mode

A typical example: you ask an assistant for a function that converts power readings to watts. You get this:

python
# telemetry.py
from decimal import Decimal

UNIT_FACTORS = {"W": Decimal("1"), "kW": Decimal("1000"), "MW": Decimal("1000000")}


def to_watts(value: Decimal, unit: str) -> Decimal:
    if value < 0:
        raise ValueError("power reading cannot be negative")
    try:
        return value * UNIT_FACTORS[unit]
    except KeyError:
        raise ValueError(f"unknown unit: {unit}") from None

This version is fine. The versions that aren't fine look almost identical: a float instead of Decimal, a missing negative check, "kw" instead of "kW". Example-based tests with three hand-picked values rarely catch these. Properties do.

Layer 1: property-based tests with Hypothesis

Instead of asserting specific outputs, assert things that must always be true:

python
# test_telemetry.py
from decimal import Decimal

import pytest
from hypothesis import given, strategies as st

from telemetry import UNIT_FACTORS, to_watts

readings = st.decimals(
    min_value=0, max_value=10**6, places=3, allow_nan=False, allow_infinity=False
)


@given(value=readings, unit=st.sampled_from(list(UNIT_FACTORS)))
def test_larger_units_never_produce_smaller_values(value, unit):
    assert to_watts(value, unit) >= to_watts(value, "W")


@given(value=readings)
def test_kw_round_trip_is_exact(value):
    assert to_watts(value, "kW") / 1000 == value


@given(
    value=st.decimals(
        max_value=Decimal("-0.001"), allow_nan=False, allow_infinity=False
    )
)
def test_negative_readings_are_rejected(value):
    with pytest.raises(ValueError):
        to_watts(value, "W")

The round-trip test fails immediately if someone "simplifies" the function to use floats. The negative-reading test fails if the guard disappears in a refactor. Neither needs you to guess the breaking input.

I write properties by hand, not with the assistant. That's the point: the tests encode what a human believes must be true, independently of what the model generated.

Layer 2: schema snapshots for data contracts

AI refactors love to "improve" models: an int becomes float, an optional field becomes required. On a system talking to thousands of devices, that's a silent breaking change.

I snapshot the JSON schema of every Pydantic model that crosses a boundary:

python
# test_contracts.py
import json
from pathlib import Path

from models import Reading  # your Pydantic v2 model

SNAPSHOT = Path(__file__).parent / "snapshots" / "reading.schema.json"


def test_reading_schema_is_stable():
    current = Reading.model_json_schema()
    expected = json.loads(SNAPSHOT.read_text())
    assert current == expected, "Reading schema changed: update the snapshot deliberately"
When the schema changes on purpose, the developer updates the snapshot in the same PR, and the diff makes the contract change visible to reviewers. When it changes by accident, CI stops it.

Layer 3: mutation testing, but only where it matters

Mutation testing (I use mutmut) changes your code in small ways and checks whether any test fails. If a mutant survives, your tests don't really cover that logic, no matter what the coverage report says.

It's slow, so I don't run it on every PR. I limit it to critical modules and run it nightly:

toml

# pyproject.toml
[tool.mutmut]
paths_to_mutate = ["src/telemetry.py"]
bash
mutmut run
mutmut results

(Config options differ between mutmut versions, so check the docs for yours.) Surviving mutants go into the backlog as test gaps.

Wiring it into CI

Hypothesis profiles keep local runs fast and CI runs thorough:

python
# conftest.py
from hypothesis import settings

settings.register_profile("dev", max_examples=50)
settings.register_profile("ci", max_examples=300)
yaml
# .github/workflows/tests.yml
name: tests
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-dev.txt
      - run: pytest --hypothesis-profile=ci

Who writes what

The layers only work if the split between human and assistant is explicit. Mine looks like this:

I also keep a short rules file in the repo that the assistant reads (a CLAUDE.md or equivalent): Decimal for all power and money values, no sync DB calls inside async def, never edit files under snapshots/. It doesn't replace the tests, but it reduces how often they fail.

A checklist for your repo

Pick two or three modules where a wrong value costs real money or safety.
Write three to five properties for each, by hand.
Snapshot the schemas of every model that crosses a service boundary.
Run mutation testing nightly on those modules only.
Add a rules file for your assistant based on the bugs the layers catch.

What this doesn't solve

It won't catch wrong architecture. A well-tested function in the wrong place is still in the wrong place. That still needs a senior reviewer.
Good properties are hard to write. Expect the first few to take longer than the code they test.
Hypothesis adds CI time. In my experience, 300 examples per property adds seconds, not minutes, but heavy fixtures can change that.

For me the trade-off has been worth it on systems where a wrong number in production means a wrong invoice or a wrong charging decision. On a throwaway prototype, it probably isn't.

What does your setup look like? I'm curious whether others gate AI-generated code differently, especially around async code.

If you'd rather not build this from scratch: Boldare, a Python software house in Poland, runs exactly this kind of guardrail setup on production backends for IoT and energy systems, with senior review on every merge. Worth a look if you need a team that takes AI-generated code seriously.

Why AI Test Automation Matters More as Your SaaS Codebase Grows

Olivier — Thu, 10 Sep 2026 08:41:06 +0000

TL;DR: Traditional test automation solves test creation once, then the maintenance cost scales roughly proportionally with the codebase until the suite itself becomes a project. AI-assisted automation targets that maintenance cost specifically: generation tied to real changes, anomaly detection instead of brittle assertions, and coverage that adjusts instead of accumulating. Four components, one clear limit, and a quick way to tell if it's worth it yet.

The problem, stated precisely

A regression suite is cheap to justify when it's new. It's expensive later, not because automation stopped working, but because the maintenance cost scales with the codebase. New features mean new test cases. New test cases mean a longer regression cycle. At some point the suite costs nearly as much to keep green as it saves.

This is not the same problem as "can AI write tests." Test generation alone doesn't touch the maintenance side of the equation. The interesting part is the combination below.

The pattern

[testing pipeline]
-> assessment (map where time is actually lost)
-> test generation (tied to real code changes)
-> anomaly detection (behavior, not exact-match)
-> adjusting coverage (stale tests flagged, not accumulated)
-> human review (bug vs. intended change vs. noise)
-> [suite whose maintenance cost stops scaling 1:1 with the codebase]

Assessment before automation. Map which parts of the suite take longest to maintain, which failures are false positives being manually triaged every release, and where coverage gaps exist despite a large test count. Skipping this step tends to mean automating the wrong 20% first.
Test generation tied to actual changes. Instead of a fixed, hand-expanded script, cases get generated from what changed in a given PR. This keeps generation proportional to actual code churn instead of growing an ever-larger static list.
Anomaly detection instead of brittle exact-match assertions. A test that fails because a CSS class changed for styling reasons is noise, not signal. Detecting meaningful behavioral differences cuts the false-positive rate that erodes trust in a suite over time.
Coverage that adjusts instead of accumulating. As features get deprecated or refactored, associated tests get flagged for review instead of running forever against dead code paths.

Why generation alone doesn't solve this

Generating tests once, even well, doesn't address the part of the problem that actually compounds: keeping the suite aligned with a codebase that keeps changing. A one-time generation pass is still a fixed snapshot the moment it's written. The maintenance tax shows up exactly the same way it does with hand-written tests, just with a different starting point.
The combination of generation-on-change, anomaly detection, and adjusting coverage is what keeps the suite's maintenance cost from scaling in lockstep with the codebase. Any one piece alone leaves a gap.

Where this actually breaks

Anomaly detection can flag that something changed. It can't tell you whether the change was intentional. That judgment call doesn't go away, and shouldn't, since it depends on product context no detector has access to.

It's also weaker at catching absence than presence. Generation tied to code changes works well for regressions in existing behavior. It's a poor tool for noticing that no one wrote a test for a scenario no one anticipated in the first place, since there's no diff to generate a test from.

And scale matters. A small product with a modest, well-maintained suite may not have enough proportional maintenance cost yet to justify the setup. This earns its cost once regression cycles are visibly stretching release timelines, not as a default for every test suite regardless of size.

Quick gut check

Stretching regression cycles, a false-positive rate high enough that failures get triaged rather than trusted, and suite maintenance time comparable to or exceeding time saved? Probably worth it. Stable cycle length, low false positives, maintenance time small relative to value delivered? The current setup is doing its job.

Curious what other teams have hit trying to reduce false positives specifically. Exact-match assertions are the obvious brittle case, but I'd guess there's a long tail of less obvious ones (timing-dependent tests, ordering assumptions in supposedly unordered data) that don't get talked about as much. What's broken your suite in a way that wasn't really a bug?

Anatomy of an AI Agent Pipeline for Document Generation

Olivier — Wed, 26 Aug 2026 13:45:04 +0000

TL;DR: A recurring pattern for turning structured content (Markdown, CMS entries) into finished, branded documents (Google Docs, Word) using an AI coding agent plus a dedicated export script. Four components, one common failure mode, and a decision point on whether it's worth building at all.

The problem, stated precisely

Content gets authored in a clean format. It has to leave as a formatted document with a cover page, consistent typography, and tables that don't fall apart. The step in between is usually manual, and usually falls on whoever wrote the content.
This is not the same problem as "intelligent document processing" (IDP), which is about extracting structured data out of existing documents (think OCR on invoices). This is the reverse direction: generating a document from content that's already structured. Worth separating clearly, because the tools and failure modes don't overlap.

The four-part pattern

structured content → agent (workflow definition) → formatting rules (encoded in editor context) → reusable content blocks → export script (native API) → branded document

Agent with a defined workflow, not a single prompt.
The agent isn't told "format this document" from scratch each time. It's given a workflow: what to validate in the source, what to assemble, when to pause for human review, what to flag for manual cleanup. This is what makes the setup reusable across document types instead of a one-off script.
Formatting rules that live where the work happens.
Heading-to-style mappings, table structure conventions, which Markdown patterns export cleanly: encoded as context the agent picks up automatically, not documentation someone has to read first. New team member, new agent session, same correct output.
Reusable content blocks.
Recurring sections (disclaimers, methodology, standard intros) are parametrized templates, not copy-paste targets. Consistency stops depending on who's writing that week.
A dedicated export script against the native API.
Not a generic converter. Working directly against the target platform's API (Google Docs, Word's OOXML) is what gives full control over branding, and it's also where most of the interesting bugs live (see below).

Why not just a script

A script handles the happy path fine. It falls over on anything that doesn't match the expected shape: an unusual table, a heading pattern nobody anticipated, a section that needs a human call rather than an automatic export. An agent, working from intent rather than a fixed rule set, can reason through those cases instead of failing silently.
The trade-off: agent output needs the same review discipline as any automated pipeline. Correctness and speed are separate properties. Don't assume one from the other.

Where this actually breaks

Document APIs weren't built for this use case, and their documented behavior doesn't always match reality once you're in the weeds. A few patterns worth watching for if you're building something similar: inserting content in one place can shift positioning of everything downstream of it, and global style changes (spacing, margins) can silently affect elements you wouldn't think of as "text," like headers and footers. None of this shows up in the docs. It shows up when you actually run the pipeline against real content.
Because of this, a realistic pipeline still budgets for a short human pass before anything ships. The win isn't zero manual work, it's manual work measured in minutes of polish instead of hours of rebuilding.

Is it worth building

Quick gut check: recurring document type, produced weekly or more, structured and version-controlled source content, specific branding requirements a generic tool can't hit? Probably worth it, pays back within a few weeks once you count revisions. One-off documents, source content scattered across files with no consistent shape? Not worth the engineering time. Use an existing tool.

Curious whether others have hit the same API quirks building something similar (Google Docs, Notion API, Confluence, wherever). What broke for you that the documentation didn't warn about?

Outsourcing software development to Poland: a practical look at what actually matters

Olivier — Thu, 20 Aug 2026 08:59:53 +0000

If you're a CTO or engineering lead evaluating outsourcing partners, "just go with Poland" is advice you've probably heard without much explanation of why. This post breaks down the actual factors worth checking — talent depth, communication overhead, legal alignment, time zone math — rather than repeating the usual marketing pitch. Structured so it's easy to scan.

First, what do we mean by "outsourcing"?

Outsourcing software development means hiring an external team to design, build, test, and often maintain your product, instead of hiring in-house. Two variants matter for planning purposes:

Offshoring — outsourcing to a distant country/time zone.
Nearshoring — outsourcing to a nearby country with significant time zone overlap.

For a team in Western/Central Europe, Poland is nearshore. For a team in the US or Middle East, it's technically offshore, but the overlap is usually workable for at least a few real-time hours a day.

Why do teams outsource in the first place?

Usually one or more of:

No in-house technical skill — software isn't the core business.
Cost — hiring, onboarding, and retaining an in-house team doesn't pencil out for the scope needed.
Deadline pressure — there isn't time to build a team from zero.
Risk — the product is high-stakes and picking the wrong delivery partner is expensive to undo.
None of these are Poland-specific — they're the reasons to outsource anywhere. I've sat in enough of these conversations with clients to know the Poland-specific pitch usually comes after this part, not before. The interesting question is what Poland actually brings once you've decided to outsource.

What's actually different about the Polish market?

Talent pool size and depth
Poland has 18 public and 5 private technical universities, concentrated in a handful of major cities, with STEM as a popular field of study. The country has been reported to produce tens of thousands of IT graduates annually, and its IT outsourcing workforce is in the hundreds of thousands, with continued growth.
Skill level, not just headcount
A large talent pool doesn't automatically mean good engineers (I'd argue this is the weakest data point in most "why Poland" pitches, ours included). One somewhat more useful signal: Polish developers consistently place well in competitive programming — Microsoft's Imagine Cup, Google's Code Jam, TopCoder. Doesn't prove any given team is strong, but it says something about the baseline coming out of the education system.
English proficiency (the thing that actually breaks projects)
Communication overhead is probably the single biggest hidden cost in outsourcing — bigger than hourly rate, in my experience. As of 2024, Poland ranked among the top countries globally on the EF English Proficiency Index. Many graduates are also fluent in German, French, or Spanish, which matters if your stakeholders aren't all English-native.
Working model
Most established Polish dev shops run agile/Scrum by default — sprints, direct access to individual engineers rather than a single account manager as a bottleneck, and a discovery phase up front to nail down scope before code gets written. If a vendor doesn't do this, worth asking directly why not.
Time zone overlap
CET/CEST gives full-day overlap with the rest of Europe, a few real-time hours with US East Coast mornings, and workable overlap with Middle East business hours. Compare that to outsourcing to APAC from the US, where overlap can be close to zero.
Legal and regulatory alignment
As an EU member, Poland operates under the same regulatory framework as other EU countries — GDPR being the relevant one for most software projects. If you're handling EU user data, this removes a category of legal risk that shows up when outsourcing to non-EU jurisdictions.
Physical access
Central European location with well-connected airports (Warsaw, Gdańsk, Kraków, Katowice, Poznań) — relevant if your team actually wants to visit for a kickoff or a critical phase, not just video call.

Is it actually cheaper?

Not the cheapest — worth being clear-eyed about that. India and China generally beat Poland on raw hourly rate. What Poland tends to offer is a quality-to-cost ratio: salaries roughly half of comparable Western European rates, with lower communication and process risk than the lowest-cost markets. Whether that trade-off makes sense depends on how much you value delivery predictability over absolute lowest cost. (Every vendor will tell you their trade-off is the right one. Mine included. Do your own diligence.)

FAQ

Is Poland good for nearshoring from Western Europe?
Yes — full time zone overlap, EU legal alignment, and similar working-week/holiday calendar make it low-friction.

Does the time zone gap work for US or Middle East clients?
It's workable with adjusted hours, not full-day overlap. Teams like Boldare handle this regularly for clients in the US, Canada, and Australia.
**
Do Polish outsourcing vendors default to agile?**
Most established ones do — Scrum, sprints, direct engineer access. Worth confirming explicitly rather than assuming.
**
Is Poland the cheapest place to outsource?**
No. It's a mid-cost, high-reliability option — cheaper markets exist, but usually with higher communication or process risk.