DEV Community

rhea hollis
rhea hollis

Posted on

Action Recognition Field Notes: Boundary Relabeling, Near-Miss Pairs, and Splits That Don't Lie

#computervision #deeplearning #machinelearning #tutorial

An action recognition run that stalls at "decent" usually isn't starved of architecture. It's starved of honest labels. This post is the code for the three fixes that moved accuracy more than any tuning: boundary relabeling, near-miss sampling, and source-aware splits.

1. Relabel to temporal boundaries

Fixed 16-frame windows make "opening a door" and "closing a door" nearly the same clip. Trim to the detected action, keep context padding on both sides:

def trim_to_action(clip, onset, offset, fps, pad=0.5):
    lo = max(0, int(onset * fps - pad * fps))
    hi = min(len(clip), int(offset * fps + pad * fps))
    return clip[lo:hi]   # boundaries carry the signal, not the window
Enter fullscreen mode Exit fullscreen mode

The relabel of ~200 near-miss clips outperformed a week of architecture experiments.

2. Oversample the near-miss pairs

Aggregate class balance hides the pairs that matter (sit down vs. bend-to-pick-up). Mine the confusion matrix and oversample against nearest neighbors:

def pair_weights(cm, labels, boost=3.0):
    pairs = {}
    for i, j in zip(*cm.topk_offdiagonal(k=5)):   # top confusable pairs
        pairs[(labels[i], labels[j])] = boost
    return pairs   # feed a WeightedRandomSampler, not a blanket class weight
Enter fullscreen mode Exit fullscreen mode

Blanket class weights lift everything evenly; pair boosts lift exactly what's broken.

3. Split by scene and source, never randomly

Random splits leak the same actor and room into train and test. The split needs provenance:

def honest_split(rows, by=("scene_id", "source_id"), ratios=(0.7, 0.15, 0.15)):
    groups = defaultdict(list)
    for r in rows:
        groups[tuple(r[k] for k in by)].append(r)  # provenance fields drive the split
    return random_group_split(groups, ratios)
Enter fullscreen mode Exit fullscreen mode

This only works when every row knows where it came from — which is why lineage labels belong in the schema. Collections that ship source metadata per clip (Thordata's video datasets, for one — free trial here: https://www.thordata.com/?ls=dev&lk=dev-1) make the honest split a one-liner instead of a reconstruction project.

The pattern across all three fixes: the data work is the model work. Boundaries in the labels, pairs in the sampler, provenance in the split.

Top comments (0)