I trained a language model that writes by filling in blanks

Every language model you've used writes the same way. Left to right, one token at a time, each word chosen after the one before it.

I wanted to know what happens if you don't do that. So I trained a small model that starts with a sentence that's entirely blanks, and fills them in.

How it writes

The model is a BERT-style transformer, which means it looks at the whole sequence in both directions at once, not just at what came before.

Generation works like this:

1. Start with a sequence where every position is [MASK].
2. The model predicts a token for every masked position at the same time.
3. Keep the predictions it's most confident about. Put the mask back on the rest.
4. Repeat, until nothing is masked.

[MASK] [MASK] [MASK] [MASK] [MASK] [MASK]
[MASK] little [MASK] [MASK] [MASK] park .
The little [MASK] went to the park .
The little dog went to the park .

It's diffusion, the same idea behind image generators, applied to words. Image models start from noise and remove it step by step. This one starts from blanks and fills them in step by step.

Decision 1: diffusion instead of left to right

The appeal is that nothing forces the model to commit to the first word before it knows the last. It can settle the shape of a sentence early, the subject here and the full stop there, and fill in the middle once the edges make sense. And it fills many positions per step instead of one, which is where the speed promise of diffusion language models comes from.

Why it's awkward

This is where it got interesting, and honest.

Image diffusion works with gentle noise. A pixel can be slightly wrong and get nudged towards right. Words don't work like that. A token is either "dog" or it isn't, and there's no such thing as slightly-dog. So "noise" for text becomes a blunt on or off mask, and a lot of the smoothness that makes diffusion work for images just isn't there.

Other awkward parts showed up quickly. You have to decide the length of the output before you start, because you're filling a fixed row of blanks. And deciding which predictions to keep at each step matters enormously. Commit too early and the model locks in a mistake it can never undo. Commit too late and it barely makes progress.

Decision 2: train it on a laptop

I trained it on TinyStories, a dataset of simple short stories, locally on Apple Silicon. No cloud cluster.

That was deliberate. A small model on a small dataset is something you can actually finish, and finishing is where you learn. You get to watch it train, break it, change one thing and see what happens, all in an afternoon instead of a week of waiting on rented GPUs. It's the same instinct as every other small version of a big thing I've built: shrink it until you can hold all of it, then look closely.

What I took from it

Left-to-right generation isn't a law of nature. It's one choice, and seeing the alternative up close makes it much clearer what that choice buys you and what it costs.

The model writes worse stories than the ones you're used to. It taught me more than any of them.