Stop adding print statements to debug your data pipeline — use watcher instead

Abinesh N — Wed, 20 May 2026 16:19:50 +0000

I built a Python decorator that watches your DataFrame pipelines automatically

You know this moment:

Input rows  : 1,000,000
Output rows :   263,979

Somewhere in your pipeline, 736k rows disappeared.

Which step caused it?

A bad merge?
A silent dropna()?
A duplicate join key?
A dtype issue?
A filter you forgot existed?

So you start adding:

print(df.shape)
print(df.columns)
print(df.isnull().sum())

…everywhere.

Then rerun the entire pipeline again.

That frustration is why I built dfwatcher.

What is dfwatcher?

dfwatcher is a lightweight decorator for pandas pipelines that automatically tracks:

row count changes
null deltas
schema drift
dtype changes
join explosions
memory usage
pipeline summaries

with zero config.

Just decorate your functions.

from watcher import watch

@watch
def clean(df):
    return df.dropna()

That’s it.

Example

@watch
def merge_orders(df):
    return df.merge(orders, on="customer_id", how="left")

Output:

merge_orders()  964,203 → 1,069,104  ▲ +104,901 rows (+10.9%) ⚠

  columns added : +tier

  💥 join explosion · duplication ratio 10.9%

  key column     top value    repeat count
  customer_id    9182               184

Instead of just telling you rows increased…

…it tells you why.

Why I built it

Most pipeline bugs are not syntax bugs.

They’re data drift bugs.

The code runs successfully.
The tests pass.
The pipeline completes.

But the data quietly changes shape somewhere in the middle.

Those are the hardest bugs to debug because:

they’re silent
they propagate downstream
and they’re usually discovered hours later

I wanted something that behaves like:

“git diff for DataFrames”

but automatically during execution.

Features

Row tracking

clean()  1,000,000 → 964,203  ▼ -35,797 rows

Null tracking

nulls -35,797  status  (35,797 → 0)

Schema drift detection

columns added : +revenue_band

Dtype change detection

dtype change : customer_id  int64 → object

Join explosion detection

💥 join explosion

Threshold guards

@watch(
    warn_on_loss=0.05,
    raise_on_loss=0.20
)

Turn silent data corruption into CI failures.

Session summaries

with session("nightly ETL"):
    df = clean(df)
    df = merge(df)
    df = score(df)

At the end you get a full pipeline summary automatically.

What surprised me while building it

The hardest part wasn’t row tracking.

It was making the output useful without becoming noisy.

A debugging tool that prints too much becomes another thing developers ignore.

So I focused heavily on:

readable terminal formatting
meaningful warnings
showing only the most important changes
zero-config defaults

The goal was:

install → decorate → immediately useful

Roadmap

Currently:

pandas support
memory tracking
custom handlers
CI-friendly summaries

Planned:

Polars backend
DuckDB backend
HTML / notebook renderer
structured JSON logging
global config system

Install

pip install dfwatcher

GitHub:
https://github.com/Abineshabee/watcher

PyPI:
https://pypi.org/project/dfwatcher/

I’d genuinely love feedback from data engineers, ML engineers, analytics engineers, and pandas users.

Especially:

features you wish pipeline tools had
debugging pain points
weird merge bugs you’ve experienced
ideas for Polars / DuckDB support

DEV Community: Abinesh N