Ever wondered how LLMs format code so perfectly? How do they know when to break to the next line? If you've followed my series, you know that AI can't visually see text like we do. It can't say "this word is too long for the line, put it on a new one.", since it operates in tokens. So what does it do? Spoiler, it's way more sophisticated than you'd expect.
The Process
- Counts characters in the current line
- References the maximum line width
- Calculates remaining space
- Compares with next word's length
- Decides whether to break the line
Simple enough. But how it does step 1, counting characters,is where things get interesting.
The Geometry
Researchers found that LLMs build geometric structures called manifolds to handle character counting. This is a curved surface in six-dimensional space with reference points representing different character counts. But six dimensions is hard to wrap your brain around, so instead, picture an old spiral staircase. Each step represents a different character count: step 1 = one character, step 2 = two characters, and so on.
The staircase is made of creaky wood. When you step on step 5, steps 4 and 6 might creak too. This happens when LLMs aren't 100% certain—instead of "definitely 5 characters," it says "somewhere between 4 and 6 characters."
Why Build a Complex Structure?
Why not just count 1, 2, 3, 4 like humans?
The spiral structure is incredibly space-efficient and robust. Instead of separate storage for every character count, it uses a few dimensions to store hundreds of different counts. The overlapping "creaks" provide flexibility to operate with uncertainty.
Just like human perception, precision decreases with larger numbers. You'd struggle to distinguish between 1,000 and 1,001 items, but easily tell apart 1 and 2.
Boundary Detection
Once the model has the character count, it needs to detect if it's approaching the line boundary. It uses specialized components called boundary heads.
Picture two spiral staircases moving together:
- Each step of the first staircase represents current character count
- Each step of the second staircase represents a line width
When you're on step 35 of the current count and approaching a line width of 40 characters, a boundary heads activates: "Hey, we're getting close to the line width. Pay attention."
The Final Decision
For the last step, the model needs the next word's length. Say it's planning to add "aluminum" (8 characters).
Instead of simple subtraction, it plots both values on a two-dimensional graph to decide whether the word fits or needs a line break.
The Biology Connection
Here's the most fascinating part to me: LLMs independently discovered solutions remarkably similar to biology. The boundary-detecting features work like boundary cells in animal brains, which are the same neural mechanisms that help us navigate physical space. Pretty cool, huh?
You can read the complete research paper here.
The content in this post is part of my "Learning Out Loud" LinkedIn series, where I share things I've learned recently. Watch the video version on LinkedIn.
Top comments (0)