Have you ever wondered how high-performance LLM deployment frameworks like vLLM, TensorRT-LLM, or Hugging Face TGI actually optimize model serving? While you wait for tokens to stream into your chat window, the infrastructure under the hood is executing a fragile balancing act: scheduling prompt pre-computation, paging memory segments, verifying speculative token chains, and dodging system-stalling bottleneck crashes.
To teach you how LLMs manage GPU memory under high concurrent loads, I built an interactive Tetris-style puzzle game:
๐งฑ PagedAttention: VRAM Tetris
Play in Fullscreen Mode (if the embed sizing is tight)
๐ ๏ธ Choose Your Allocation Mode
Your journey as a memory scheduler is split into two distinct memory allocation modes:
- ๐ข Contiguous Mode (Easy/Vanilla): Stacking falling token sequence blocks (Tetrominos) into solid rows. Any gaps you leave behind are trapped, creating unusable External Memory Fragmentation that blocks new incoming allocations.
- ๐ Paged Mode (PagedAttention - Hard): Play with paged virtualization. Pressing Shift or P triggers a Page Split, shattering the active falling block into individual 1x1 memory pages that cascade down independently to fill any available fragmentation holes below.
๐งฌ Playable ML Concepts Explained
This isn't just standard Tetrisโevery shape, block placement, and allocation rule represents a real-world concept in modern machine learning infrastructure. Here is how the in-game mechanics map directly to how large language models allocate GPU memory:
1. ๐พ Memory Allocation & Contiguity (Standard Stacking)
- In-Game: You must rotate and slide falling token block shapes to pack them together contiguously. Complete horizontal rows of memory blocks represent completed inference requests, which are garbage-collected to free up VRAM.
2. ๐๏ธ External Memory Fragmentation (The Stacking Gaps)
- In-Game: Leaving empty spaces under your placed blocks represents external fragmentation. If VRAM fill spikes or blocks stack to the top, the engine crashes, throwing a CUDA OUT OF MEMORY (OOM) error.
3. ๐ PagedAttention Virtualization (The Page Split)
- In-Game: In Paged Mode, triggering a Page Split shatters the falling shape into individual 1x1 blocks that automatically drop down to seek out and fill the smallest hidden gaps in the memory grid.
๐ ๏ธ The Under-the-Hood Engineering Journey
Creating an educational puzzle game designed for embedded platforms presented some fascinating web development challenges:
1. Optimizing for the 600px Embed Limit
Dev.to embeds are capped at a strict maximum height of 600px. Fitting a complex tycoon dashboard with side panels, scoreboards, next-piece canvases, and a 20-row Tetris grid inside 600px required serious spatial compression.
-
The Solution: We shrank the cell block size (
BLOCK_SIZE) to 22px (yielding a 440px canvas height), converted the left panel stats list into a compact 2x2 grid, resized preview boxes to 70px, and relocated the system logs console from a horizontal footer directly into the left sidebar. The final layout fits completely inside exactly 580px, preventing vertical clipping.
2. Physics of the Paged Cascading Split
Splitting a rigid grid structure into individual falling particles in real-time required careful synchronization.
- The Solution: When the split is triggered, the engine parses the active Tetromino shape, decomposes it into coordinate objects relative to the grid columns, calculates the lowest-available free cell index per column, and translates each block to its destination slot before recalculating line-clear sweeps.
Click to see the Page Split Javascript logic
// --- Special Mechanic: Paged Memory Split ---
function executePageSplit() {
if (abilityCharge < 100) {
addSystemLog("PagedAttention Split not fully charged yet!", "warning");
SOUNDS.warning();
return;
}
SOUNDS.split();
addSystemLog(`Executing PagedAttention: Splitting ${currentPiece.name} into virtual pages...`, "info");
// Get all filled cells of the falling piece
const pages = [];
for (let r = 0; r < currentPiece.shape.length; r++) {
for (let c = 0; c < currentPiece.shape[r].length; c++) {
if (currentPiece.shape[r][c]) {
pages.push({
x: currentPiece.x + c,
y: currentPiece.y + r,
color: currentPiece.color
});
}
}
}
// Drop each page individually down its column to the lowest free cell
pages.forEach(page => {
let lowestY = page.y;
while (lowestY + 1 < ROWS && !grid[lowestY + 1][page.x]) {
lowestY++;
}
if (lowestY >= 0) {
grid[lowestY][page.x] = page.color;
}
});
abilityCharge = 0; // Consume charge
clearMemoryLines();
spawnPiece();
}
๐ฌ Let's Discuss:
- What is your high score in Paged Mode utilizing the Page Split ability?
- Did you notice how quickly a contiguous stack triggers a CUDA OOM compared to a paged system?
- How does VRAM Tetris change your perspective on memory allocation bottlenecks?
UnitBuilds-CC
/
PAGED-ATTENTION
A Tetris inspired game to teach how LLMs use VRAM
๐งฑ PagedAttention: VRAM Memory Allocation Tetris
An educational retro-cyberpunk Tetris-style simulator that maps classic block-packing gameplay directly to GPU memory allocation, external memory fragmentation, and virtual paging concepts.
๐ Play the Live Demo here! (Will be updated after deploy)
๐ฎ The Concept
In PagedAttention Tetris, you play as a GPU memory scheduler. Incoming requests of varying token sizes (represented by falling Tetris shapes) must be allocated in the GPU's memory registers. Gaps left behind represent External Memory Fragmentation. If memory becomes too cluttered and blocks stack to the top, you trigger a CUDA Out of Memory (OOM) crash.
Playable Memory Allocation Engines:
- ๐ข Contiguous Mode: Falling block sequences remain solid. If gaps are left underneath, they cannot be filled, causing fragmentation and system bloat.
- ๐ Paged Mode (PagedAttention): Pressing
ShiftorPtriggers a Page Split. The active falling block shatters into individual 1x1 block pages thatโฆ
Disclaimer: AI was used throughout this project, it is just fitting that it would co-author with me, so special thanks to the Foundry for its tireless hours toiling away and Gemini for producing the cover image.
Top comments (20)
@xulingfeng The red flag catch you again?
๐คฃKFC is so good ๐
Now that was mean ๐ญ
My bad ๐ Next time I'll save you a screenshot of the order receipt so you can at least look at it.
... I'll get you back someday... Wont be today, might not be tomorrow, but I'll get you back for that 1... ๐๐ญ
How you liking the tetris btw?
Dude this is genuinely cool โ a Tetris game that actually teaches PagedAttention while you play? The Contiguous vs Paged mode switch is brilliant. ๐ Mad respect for building this. ๐
Thanks for the feedback! I was worried it doesnt play well, cuz they havent implemented my embeddings upgrade yet ๐ซ So the windows are a bit restrictive... But I think the streak is going well ๐ 6 days of daily games, each covering a different aspect of LLMs, it's been alot, but I'm having fun with it! (My cloud bill is surviving for now...) Kinda cool that LLMs are Demented got over 200 reads ๐ฅณ Lets hope I can keep the games interesting without repeat topics...
You should write a milestone post too โ a peek behind the curtain. I think a lot of us would love to see how the system behind the system actually works.
Also, keep grinding man ๐ My top post is about to hit 3K reads. Catch up.
Not good ๐๐ป
This is correct. I usually only eat KFC when I'm tired of everything else โ fresh veggies and meat are always the better choice for a healthy diet.๐
As for vegetarians, they can only eat vegan food.
Fair point. I'm actually polishing up Stratagem #7 right now โ should be out in a few hours. Also waiting for UnitBuilds' next article ๐ Either way, UnitBuilds is never going to escape the KFC joke at this point.
Until what time?
Probably around 1-2 AM your time ๐ So you'll wake up to it tomorrow. No rush.
But it's only 10:31 in the morning.
Oh wait โ you're in India! My bad. So it'll be out in a few hours, you'll see it this afternoon ๐
Published it while pretending to work. Just in time to clock out and catch the subway. ๐๐จ
Nice model of the fragmentation half. The piece the Tetris framing can't easily show, and the part that actually made vLLM a step change, is what paging unlocks beyond packing: copy-on-write sharing of KV pages. Two requests with the same system prompt, or N parallel samples from one prompt, can point at the same physical pages and only fork on write. Contiguous allocation can't express that at all - every sequence owns its own slab. So the paged table isn't only dodging external fragmentation, it's turning the KV cache into shared virtual memory, and prefix reuse is where a lot of the real throughput comes from once production traffic has fat shared system prompts. Would make a fun hard-mode level: two falling pieces allowed to overlap because they're the same prefix.
True, can work, but the problem is controlling the 2 pieces at once. Could set it up for wasd and arrows separately, but for now the series is still there to just teach the basics, I think from 10 on, I'll start covering the more advanced nuances, eg. having a ghost tetris block that fills fields it crosses, so if you're fast, you can use it to clear up older rows. Or system instructions that anchors a row and pushes it to the bottom, so it progressively fills the vram with set sequences that stay persistent. Though that would be re-using the tetris concept, instead of experimenting with something new.