Build a Self-Supervised Pretext Task in 25 Lines of Python (No Labels Required)

#python #deeplearning #machinelearning #ai

Most teams don't have a data problem. They have a label problem: millions of raw documents or images and only a few thousand annotated ones. Self-supervised learning (SSL) gets around this by making the data label itself.

The easiest way to understand that is to build the label generator yourself. Here's a minimal masked-token pretext task, the same idea BERT is pretrained with, in plain Python.

Step 1: Start with raw, unlabeled text

corpus = [
    "the engineer deployed the model to production",
    "the server returned an unexpected error",
    "the model detected a damaged component",
]

No sentiment tags, no categories, no annotators.

Step 2: Turn every sentence into (input, target) pairs automatically

import random

MASK = "[MASK]"

def make_pretext_examples(sentence, mask_prob=0.15, seed=None):
    rng = random.Random(seed)
    tokens = sentence.split()
    inputs, targets = [], []
    for tok in tokens:
        if rng.random() < mask_prob:
            inputs.append(MASK)
            targets.append(tok)      # the hidden word is the label
        else:
            inputs.append(tok)
            targets.append(None)     # no loss on visible tokens
    return inputs, targets

for i, s in enumerate(corpus):
    print(make_pretext_examples(s, mask_prob=0.3, seed=i))

Output (Python 3, exactly as printed):

(['the', 'engineer', 'deployed', '[MASK]', 'model', 'to', 'production'], [None, None, None, 'the', None, None, None])
(['[MASK]', 'server', 'returned', '[MASK]', 'unexpected', 'error'], ['the', None, None, 'an', None, None])
(['the', 'model', '[MASK]', '[MASK]', 'damaged', 'component'], [None, None, 'detected', 'a', None, None])

Notice the first two sentences mostly hid easy words like "the" and "an". That's exactly the shortcut problem covered below; real pipelines mask subword tokens and use smarter masking strategies.

That's the whole trick. The target came from the original sentence. Nobody wrote it by hand.

Step 3: What a real model does with these pairs

In a real pipeline you would:

Tokenize with a subword tokenizer instead of split()
Feed inputs to a Transformer encoder
Compute cross-entropy only on the masked positions (the Nones are ignored)
Repeat over millions of sentences

The encoder can't predict "detected" in the third example without understanding that models detect damaged components, so it ends up learning syntax and meaning as a side effect. That side effect, a reusable representation, is what you actually want.

Step 4: Throw away the head, keep the encoder

After pretraining, you discard the token-prediction head and fine-tune the encoder on your small labeled dataset (say, 2,000 support tickets tagged by intent). This is where SSL pays off: far fewer labels for the same accuracy.

Three gotchas we learned the hard way

Too-easy masks teach shortcuts. Masking only stop-words ("the", "a") gives the model nothing to learn.
Masking is a design decision. In images, a random crop can cut out the exact defect or tumour you care about. Choose transformations that keep the meaning you need downstream.
Low pretraining loss ≠ a useful model. Evaluate with a linear probe and with fine-tuning on the real task.

Beyond masking

Masked modeling is one of four big SSL objective families. The others are contrastive learning (SimCLR), teacher-student methods without negatives (BYOL), and redundancy reduction (Barlow Twins, VICReg). We cover all four, plus a decision framework and how to evaluate SSL models properly, in our complete guide: Self-Supervised Learning Explained: How AI Learns Without Labeled Data.

What would you mask in your own dataset? Tell us in the comments.

— The AheadMint Team

DEV Community