Three ways my grouped train/test split leaked anyway...

CayllahuaPedro — Mon, 10 Aug 2026 00:19:47 +0000

I spent two weeks building a computer vision component to estimate how full a plastic
container is from drone imagery. Translucent white containers, whitish chemical product
inside, shot obliquely from a drone during field inspections.

The headline number looked good: mean absolute error of 0.055 on fill fraction, Pearson
correlation of 0.97. Then I audited my own evaluation and found that 38 of my 46 test
crops had the same physical container sitting in the training set.

The arithmetic was fine. The problem was the sentence I had wrapped around it: I was
presenting 0.055 as the error on containers the model had never seen before.

What makes this worth writing about is that I had the guardrail in place from day one,
and it failed three separate times for three unrelated reasons. Each one is easy to
reproduce in any project that trains on frames extracted from video.

Why grouping matters here at all

A drone flies over a site and captures a burst. In my case, 12 frames over 12 seconds.
The same physical container appears in every frame of that burst, from slightly different
angles and distances.

If you shuffle those crops randomly into train and test, you are asking the model to
recognize a container it has already memorized. The metric you get back describes
interpolation between frames of one burst. It says nothing about a container the model
has never seen.

This is the most common failure in applied ML and everyone knows about it. Which is
exactly why the next part is worth reading.

The guardrail I wrote on day one

My dataset module reads the grouping column from config and does not offer a random
option at all:

split:
  group_column: skid_id   # never random

The code path for a random split does not exist. You cannot pass a flag to get one.
I wrote it that way on purpose, on the first day, before there was any data to split.

I still leaked. Three times.

Leak 1: the group column held the wrong ID

group_column was set to skid_id, which is what you want. Group by physical
container.

The problem was upstream. When the level labels came back from annotation, the skid_id
column in labels.csv had been populated with the scene ID, not a container ID. A
scene in my pipeline is a temporal cluster of captures, formed by grouping images taken
less than 120 seconds apart.

So the config said skid_id, the code faithfully grouped by whatever was in the column
named skid_id, and the actual grouping was by scene. Several distinct containers share
a scene. The guardrail did exactly what it was told and the result was still wrong.

Lesson: group_column: skid_id is a claim about the contents of a column, not a
property the code can enforce. Nothing in my pipeline verified that the column named
after physical containers actually distinguished physical containers.

Leak 2: the evaluation scripts built their own split

This is the one that stung.

I wrote a fix for leak 1: a module that clusters crops into physical containers by
spatial proximity of bounding box centers within a scene. Then I moved on, believing the
partitions were now grouped by container.

When I went back to the code weeks later to correct a report, I checked which modules
imported that clustering function. Exactly one did, and it was the script that renders a
visual verification sheet. The grouping module never entered a partition.

The actual evaluation code was doing this:

GroupKFold(...).split(X, y, groups=sub["src"])

src is the source frame. So the split separated frames, and since the same
container appears in all 12 frames of a burst by construction, separating frames
separates nothing.

The alternative cross-validation I also reported grouped by filename prefix, which has
its own problem (see leak 3). Neither partition ever grouped by container.

I had written the safe split path, wired the config, removed the unsafe option, and then
my evaluation scripts quietly went around all of it.

Lesson: a safe path is only safe if it is the only path. My dataset module could
not be misused. My eval scripts never called it.

Leak 3: my cross-site evaluation was not cross-site

Separately from the container problem, I had been reporting a stronger result: train on
one site, evaluate on another. That is the evaluation that actually answers "will this
work somewhere else."

I inferred the site from the filename prefix. Some images carried the drone's default
camera naming, others carried what looked like a site-specific label. Two prefixes, two
sites, or so I assumed.

When I finally tabulated the scenes:

Scene	n	Time window	File prefixes present
SCENE-01	15	12:03:27 – 12:03:39 (12 s)	both
SCENE-02	38	12:08:06 – 12:09:12 (66 s)	both
SCENE-03	1	12:13:33	one
SCENE-04	46	16:02:41 – 16:02:53 (12 s)	both

The drone-default prefix appears in all four scenes, interleaved in time with the other
one. It is a second camera or a second naming convention, and it does not identify a
location.

My "train on site A, test on site B" evaluation was training on some images from
SCENE-04 and testing on other images from SCENE-04. Same place, same 12 seconds, two
filename conventions.

Lesson: if a grouping variable is inferred from a string rather than recorded as
metadata, verify it against something physical before you build a claim on it. I now
ask for the site to be recorded at capture time.

How bad was it, measured

Once I had a container-level estimate, I counted how many evaluation crops had their
container also present in training:

Partition	Eval crops whose container is also in training
GroupKFold by frame	38 of 46
Prefix cross-validation, fold A	12 of 15
Prefix cross-validation, fold B	24 of 31

And that is a lower bound. The clustering heuristic that produced these container IDs
over-splits: it reported 19 distinct containers where my manual count on the verification
sheet found 10. When the drone drifts mid-burst, the bounding box center of the same
container moves more than the 50-pixel threshold and the heuristic files it as two
containers. With the true count of 10, the leakage is close to total.

The part people skip: what survived

When you find something like this, the temptation is to either bury it or to announce
that everything is invalid. Neither is accurate, and being precise about the boundary is
most of the value.

What broke. One published claim. My validation report stated that the partitions
ruled out evaluating on frames or containers already seen during training. The "frames"
half was true. The "containers" half was false, and it appeared in three separate
documents. It was a false statement rather than an optimistic one, and I corrected it in
all three places.

What held.

Relative comparisons. Every experiment ran on the same contaminated partition, so the ranking between approaches stays valid. Classical CV baseline versus frozen backbone plus regression head, small backbone versus base, rectified crops versus raw. Those comparisons are what I needed to choose a technical direction, and they survive intact.
The feasibility answer. The question on the table was whether the visual signal exists at all. A correlation of 0.97 answers that even if part of it comes from leakage.
The decision not to re-run everything. With 46 crops from a single 12-second window, no partition of this dataset measures generalization. Fixing the grouping would produce a different wrong number. The honest move is to declare the limitation and ask for data from other days.

The only thing that died is reading 0.055 as error on unseen containers. It never was
that.

The fix that actually prevents a repeat

I did not fix the grouping heuristic. With this dataset it would not change any decision.

What I changed is how the limitation is stated. The leakage count is now computed inside
the evaluation run and emitted into the header of the generated results table. It is not
written by hand anywhere.

The reason is specific. The first time I corrected the report, I fixed the section that
explained the cross-site claim and left four other sections that repeated it. A number
that lives only in prose goes stale the moment the code moves, and nobody notices,
because prose does not fail loudly.

That is now a house rule for the project: no number in a report may be typed by a
human. It comes out of the evaluation script or it does not appear.

A checklist, if you train on frames

Print the group sizes and the target variance per group before you trust a split. If your target barely varies inside a fold, a constant predictor wins that fold and you will conclude there is no signal. I hit this too, and it sent me down a wrong path for a week.
Verify that the group column contains what its name says. Render a contact sheet of the groups and look at it. A grouping error is obvious in five seconds visually and invisible in a metric.
Grep for who imports your safe split. If your evaluation scripts construct their own GroupKFold, the safe path is decoration.
Never infer a grouping variable from a filename. Ask for it as metadata at capture time.
Compute the leakage count as a metric. Number of test samples whose group appears in training. Emit it next to your headline number, in every run.
When a correction invalidates a number, search the whole corpus for it, not just the place where you explain it.

Item 5 is the one I would install first in any project. It turns "we grouped correctly"
from something you believe into something you measure, and it costs about fifteen lines
of code.

DEV Community: CayllahuaPedro