DEV Community: Janne Sinivirta

Meet Daffy: A Lightweight Guardian for Your DataFrames

Janne Sinivirta — Mon, 05 Jan 2026 15:39:33 +0000

The Data Validation Dilemma

Most DataFrame breakages are boring: a column got renamed, a join introduced nulls, a dtype changed, or a value shows up that you didn’t expect.

In notebooks, we do validate — just informally. We inspect .head(), run .info(), do a quick value_counts(), and add a couple of ad-hoc asserts when something looks suspicious. That’s often enough to move forward.

The problem is what happens when the notebook turns into “real code”. Those checks either:

stay behind in the exploration phase, or
get mixed into the transformation logic itself.

Either way, the assumptions become hard to see later:

What columns are required on input?
What constraints do we assume (non-null, ranges, allowed values)?
What does the function guarantee on output?

That’s the gap Daffy is trying to close: keep the transformation code clean, while making the DataFrame “contract” explicit at the function boundary.

What Daffy does

Daffy is a small library for validating pandas and Polars DataFrames at runtime using decorators.

You annotate a data-processing function with what you expect to receive (input) and what you promise to return (output). When the function runs, Daffy checks those expectations and raises a clear error if something doesn’t match — close to where the data is transformed, not several steps later.

Key features:

Column and type checks: Ensure required columns exist and have expected dtypes.
Value constraints: Enforce rules like non-null columns, unique keys, allowed categories, or numeric ranges.
Row-level validation: For cross-column business rules, validate rows with Pydantic models.
Multiple backends: Works with pandas, Polars, Modin, and PyArrow tables.

The main idea is separation of concerns:

Your function stays focused on the transformation.
The assumptions live next to it, but outside the transformation code.

Before and After: A Validation Story

To make this concrete, here’s a simple example: apply a discount to a products DataFrame.

Before Daffy – manual checks mixed into code:

def apply_discount(df):
    # Manual validation
    assert "Price" in df.columns, "Missing Price column!"
    assert "Brand" in df.columns, "Missing Brand column!"
    # We assume Price should be numeric;
    # you might add more checks here

    # Perform transformation
    df = df.copy()
    df["Discount"] = df["Price"] * 0.1
    return df

This works, but the validation and business logic are coupled. If this function grows (or gets copied), the checks tend to drift, get removed, or become inconsistent.

After Daffy – validation declared at the boundary:

from daffy import df_in, df_out

@df_in(columns=["Brand", "Price"])
@df_out(columns=["Brand", "Price", "Discount"])
def apply_discount(df):
    df = df.copy()
    df["Discount"] = df["Price"] * 0.1
    return df

You can also add constraints without turning the function into a pile of checks:

@df_in(columns={
    "Price": {"checks": {"gt": 0}},
    "Brand": {"checks": {"notnull": True}},
})
def apply_discount(df):
    ...

Why Choose Daffy over Pandera or Others?

You might ask: why not use Pandera or Great Expectations?

They’re both good tools — they’re just aimed at slightly different workflows:

Pandera is strong when you want a schema-first approach with rich validation, and you’re okay maintaining schemas/classes alongside the code.
Great Expectations is great for broader pipeline / warehouse-style data quality, expectation suites, reporting, and monitoring.

Daffy is intentionally narrower in scope. It’s for cases where you want lightweight checks right where you transform data, with minimal ceremony:

define the input/output expectations next to the function
keep the function body focused on transformations
fail early with an error that points to the violated assumption

Final Thoughts

If you’re mostly doing DataFrame work in Python (notebooks, scripts, ETL steps, small-to-medium pipelines) and your pain point is “assumptions are scattered and easy to forget”, Daffy is a practical middle ground.

It won’t replace heavier validation/monitoring frameworks for every scenario — and it shouldn’t try to. But if you want clearer function boundaries and faster feedback when your DataFrame shape changes, it fits nicely.

Try it out here: https://github.com/vertti/daffy

Stop Writing Shell Scripts for Container Health Checks

Janne Sinivirta — Sat, 03 Jan 2026 12:16:02 +0000

It started as one of those “this will take 30 seconds” moments.

We ship a container that includes a tiny helper binary — something we compile in a builder stage and COPY into the runtime image. Think: config-render, migrate-db, probe, whatever your service depends on at startup.

I just wanted to make sure the image really contained the helper I thought it contained, so I added:

RUN config-render --version

Green build. Ship it. Done.

..until a pipeline failed later with an error that told me basically nothing useful.

Was it missing from PATH?
Did we copy it into the wrong directory?
Did it lose the executable bit?
Wrong architecture (hello exec format error)?
Or did it run but print something unexpected?

So I did what we all do: I started “hardening” the check.

First: “Is it on PATH?”
Then: “Can it execute?”
Then: “What version is it?”
Then: “Is that version acceptable?”

And suddenly my “simple validation” became a little bash pipeline that tried to be clever:

command -v config-render >/dev/null 2>&1 && \
  config-render --version 2>/dev/null | \
  grep -oE '[0-9]+\.[0-9]+\.[0-9]+' && \
  # ...some version comparison logic... \
  true || \
  (echo "config-render missing or wrong version"; exit 1)

It worked... for now.

But this is where validation quietly becomes a maintenance trap:

Output parsing breaks when formatting changes (“v1.2.3”, “1.2.3 (build abc)”, extra lines, etc.)
Redirect chains swallow the exact error you needed (permission denied vs exec format error vs missing)
Version comparisons drift into “good enough”
Every repo grows its own flavor of the same fragile scripts

At some point I realized: I don’t want “a shell script that hopefully detects the problem.”
I want a single check that tells me exactly why it failed.

One command, clear failure reasons

That’s what Preflight is for.

Instead of assembling checks out of grep + awk, you run:

preflight cmd config-render --min 1.4.0

When it passes:

[OK] cmd: config-render
     path: /usr/local/bin/config-render
     version: 1.6.2

When it fails, you get the reason:

not found in PATH
failed to execute (permission denied / exec format error / etc.)
version too old (with an explicit comparison)

Example:

[FAIL] cmd: config-render
       failed to execute: exec format error

or:

[FAIL] cmd: config-render
       version 1.2.0 < minimum 1.4.0

No guessing. No “why did this randomly break today”.

Why predictable checks matter (containers, CI, everywhere)

This isn’t just a “containers are minimal” problem. It’s a reliability problem.

Shell-based checks are notoriously sensitive to environment:

bash vs sh differences (and whatever /bin/sh happens to be today)
GNU vs BSD tool differences (grep, sed, awk behave just differently enough)
Linux vs macOS quirks in CI runners
inconsistent error messaging when commands fail inside pipelines

A small binary with a narrow job tends to behave the same way everywhere.

That consistency is what you actually want from validation: predictable pass/fail and predictable output.

Multi-stage builds bonus: executable documentation

There’s another benefit I didn’t appreciate until later: in a multi-stage Dockerfile, these checks become documentation that runs.

When you see:

RUN preflight cmd config-render --min 1.4.0
RUN preflight env DATABASE_URL
RUN preflight file /app/config.yaml --not-empty

you’re not just “testing stuff”. You’re encoding expectations:

“This image must contain this helper”
“This version is the minimum supported”
“This env var must be present”
“This config file must exist and be non-empty”

It reads like a contract - and it fails like one too.

What Preflight checks

The above command check was just the beginning. Preflight now supports:

cmd — exists on PATH, executes, extracts version, compares semver
env — required env vars, allowed values / patterns
file — existence, permissions, “not empty”, basic content checks
http / tcp — connectivity with retry + timeout
hash — checksum verification

Three places it immediately helps

1) Build-time validation (fail fast)

COPY --from=ghcr.io/vertti/preflight:latest /preflight /usr/local/bin/preflight

RUN preflight cmd config-render --min 1.4.0
RUN preflight env DATABASE_URL
RUN preflight file /app/config.yaml --not-empty

2) CI validation (same checks outside Docker)

preflight cmd config-render --min 1.4.0
preflight env DATABASE_URL

3) Healthchecks without `curl` (distroless-friendly)

HEALTHCHECK CMD ["/preflight", "http", "http://localhost:8080/health"]

When shell is fine — and when it bites you

Shell is totally fine when:

the check is genuinely trivial
you don’t care about consistent error reporting
you’re happy maintaining the script

Preflight pays off when:

you’re copying/downloading helper binaries into images
you want the same behavior in CI and in containers
you need real version constraints
you want consistent output across checks
you want checks that double as executable documentation

Repo: https://github.com/vertti/preflight

My Must-Read List of Software Process Books

Janne Sinivirta — Thu, 09 May 2019 06:33:27 +0000

Recently I was asked to give a talk on Continuous Deployment. I guess I was expected to talk about the technical challenges involved in building, testing and deploying software. But to be honest, those challenges are a small and relatively trivial part of CD. So I decided to talk about what I call holistic continuous deployment.

To get the full benefits of continuous deployment I argue that you need to improve the performance of the entire business system. You need to:

shorten and amplify feedback loops
enable the environment for continual experimentation
encourage taking risks and learning from failures.

In the presentation I went through why and how to change the way to plan, communicate, collaborate, learn and lead your whole business.

I’ve given the presentation now in few places and one of the things people always ask me is “where can I read more about the topic”? I read a lot and love giving book suggestions. But this is a difficult question because how I see it, I’m asked “how to do modern software development well?” Much of it has to do with what could be called DevOps. But that too has become a topic that encompasses everything under the sun, depending on who you ask from.

Regardless, here’s my attempt to list some of the books that have had the biggest impact on my thinking on software development methods and processes. I will also briefly explain why I’ve picked each book. The list is also in the order I would read the books.

As with many seemingly technical efforts, successful holistic continuous deployment is 20% technology and 80% people. Picking the wrong technology, tool or cloud provider is rarely the real reason for failure. You are much more likely to fail because of communication failures, lack of empathy, lack of alignment or cumbersome organization structure. So what my book suggestion list is missing is probably 7 books on psychology and human behaviour research. But I don’t feel qualified to give advice on those topics.

I hope you find new ideas and inspiration from these books, or the research required to back your gut feelings up like I did. And do let me know your own picks!

THE GOAL: A PROCESS OF ONGOING IMPROVEMENT - ELIYAHU M. GOLDRATT

The Goal is written by the Israeli business management guru Eliyahu M. Goldratt. It is a fictional novel about a manager in charge of a troubled manufacturing operation. The book is used in management colleges to teach students about the importance of strategic capacity planning and constraint management.

Time Magazine listed the book as one of "The 25 Most Influential Business Management Books”. The book is a quick read but gives you plenty to think about.

The Theory of Constraints presented in the book is easier to apply in manufacturing than software development. However I find the idea and process of flow and hunting bottlenecks important concepts when working in organisations.

THE PHOENIX PROJECT - GENE KIM, KEVIN BEHR, GEORGE SPAFFORD

The Phoenix Project is a book directly based on Goldratt's “The Goal”. The fictional story is about an IT organisation and its struggles. For me the story is a bit anxiety inducing as it captures so vividly many typical pain points in large IT companies. What stuck with me forever, since I read the book, is the fact that most organisations have “key person problems”. Key person is that one and only guy who “knows this system” or the only guy who “can do this database maintenance”.

You should do everything you can to distribute information and avoid creating these key persons. But when you already have them, you should make sure you allow them to have slack (these people usually work at 120% capacity) and that they should focus more on teaching others than executing.

BEYOND THE PHOENIX PROJECT: THE ORIGINS AND EVOLUTION OF DEVOPS - GENE KIM, JOHN WILLIS

I think Beyond the Phoenix Project is only available as an audiobook. It’s a quite freely flowing discussion between the authors of the Phoenix Project on how the book was created. They share interesting bits and pieces of how they learned what they did, what motivated their writing and how they eventually ended up writing the DevOps Handbook.

DEVOPS HANDBOOK - GENE KIM, JEZ HUMBLE, PATRICK DEBOIS, JOHN WILLIS, JOHN ALLSPAW

By the time I read the DevOps Handbook, it had very little new to offer me. But at that time I had been developing software as my day job for more than 20 years. Even so, it is a well written and extremely well researched book. It deals with all the mandatory topics on and around DevOps and I would encourage you to read it even if you feel very familiar with the topic. Also, I would encourage reading “The Goal” and “Phoenix Project” before this book, not after.

ACCELERATE: THE SCIENCE OF LEAN SOFTWARE AND DEVOPS - NICOLE FORSGREN, JEZ HUMBLE, GENE KIM

Many processes and methods applied in software development today are used solely because they were suggested by Martin Fowler or Kent Beck said it’s good in their respective blog. Bringing in some more scientific approach is very welcome and my last two books do just that. The State of DevOps report has now been published for 7 years. In their own words “The State of DevOps Report is the longest standing, most widely referenced and largest body of DevOps research on the planet.”

The “Accelerate” book takes a deep dive into how the report is researched, how it has evolved over the years, the results of the report and the statistical methods used and their justifications. In general it’s very nice to have research data to back up your arguments on why some practices should be considered in your organization.

THE PRINCIPLES OF PRODUCT DEVELOPMENT FLOW: SECOND GENERATION LEAN PRODUCT DEVELOPMENT - BY DONALD G. REINERTSEN

With this book I’m torn. It is probably the best book on software development processes I’ve ever read, but the format (splitting everything into principles) is a tedious choice and the book is a pretty lengthy read. Despite its shortcomings I very much encourage you to read the book. I think it contains an endless amount of little golden nuggets and probably warrants several rereads.

The book argues in a very pragmatic way that many of our typical project management and product development practices are broken or plain wrong. Then the book reaches out to other fields of science and pulls in the necessary research to prove its points. We get to read research from telecommunication networks, queuing theory, the military etc. The examples show how things we tend to do in projects have been proven unsuccessful in many other fields.

This was originally published in my company's blog.

Great User Stories for Continuous Deployment

Janne Sinivirta — Wed, 31 May 2017 09:53:20 +0000

Unity Ads delivers targeted video advertisement to hundreds of millions of mobile phones at staggering rates. We have six teams building the platform at our Helsinki office. All the teams are able to deliver all parts of the product to production several times a day every day.

We have wholeheartedly embraced the uncertainty in product development. We work in a fast paced industry where new companies and technologies come and go and each can drastically alter the whole landscape. We realize we need to innovate and implement accordingly. We admit that we can only make educated guesses on what will work and what will not. So we experiment, measure results and learn. Everything in our process aims to make experimentation fast, economic and safe.

To make this work, one of the most important things we focus on is making the batch size of work as small as possible on all levels. For teams this means small and high quality user stories. These will be the focus of this blog post.

Definition of Good User Story

Continuous Deployment means that all changes made to the software are deployed to production as soon as they are ready. We deploy software to production several times a day.

My definition of good user story in continuous deployment environment is:

“The smallest increment we can make, from working software to working software, that still brings value or proves a hypothesis.”
Breaking down my definition, it contains two challenges: bringing value and being a small increment. Developers often struggle with focusing or defining the value in each story. Product people struggle with the fact that most of their feature requests are bad ideas. To get to a working idea, the fastest way is to make small hypothesis, get it to production and prove it, and then iterate.

Value Hypothesis

When discussing the goals behind features, it’s important to switch from calling them requirements to calling them hypothesis. When we are in an uncertain or fast changing environment, you can’t really have requirements. This naming change encourages experimentation. We should be allowed (even encouraged) to say that “we are unsure if this feature will bring the benefits we expect, so let’s find a way to verify the assumptions with minimal effort”.

The teams here works towards goals not features. Getting there usually starts with the teams asking for the goals behind feature requests. You can use a technique like the annoying but sometimes effective “Five Whys or just more probing discussions. In the end, you should clearly understand the reasons why a specific feature is beneficial.

Safe Increments

When the goal is clear, the next thing we hunt for is “the smallest increment that still brings value or proves a hypothesis”. It’s easy to split stories on a technical level so they make sense to implement but so that alone they don’t bring any value. I strongly discourage this.

It is very easy to have the whole team busy completing technical bits and pieces for weeks only to realize that first time the whole thing is usable for the customer will still take few weeks or months of integration work. The most common culprit I see is the split of story to “frontend part and “backend part”. Neither brings any value alone and often when implemented apart it requires many iterations and neither implementer gets the satisfaction of “delivering anything.

Generally the aim is to figure out “how can I throw away 80% of the story and still deploy working software that we can learn from”? You should elaborate the original story and make all the steps and constraints it has explicit. If it turns out the feature describes a workflow or a “wizard style, consider if it’s possible to deliver only one or few of the workflow steps. Are there variations in the feature? Consider implementing just one of the variations first. Richard Lawrence describes similar patterns in his blog with more detailed examples.

A great way to encourage the developers to shift their focus from delivering code to delivering value was to make team decision to have a product demo every two weeks. In each demo only show working software from staging environment. No excuses, no showing stories or burndown charts. This has the effect of having developers consider “what will I have to show for this in the next demo when planning each story.

More Resources

There is relatively little good resources on the topic of good user stories and methods to arriving to those. This is surprising, considering the importance of the topic for the success of your whole organization and ability to do meaningful continuous deployment. Agile Alliance lists the following resources:

Twenty Ways to Split Stories, 2005
How You'll Probably Learn to Split Features, 2008
Patterns for Splitting User Stories, 2009
InfoQ: How to Split User Stories, 2011
Splitting User Stories: the Hamburger Method, 2012

Let me know if you have others you have found useful!

Big thanks to Heikki Tunkelo and Samuel Husso for all the feedback while writing this article. This was originally published on my LinkedIn page.