Have you ever wondered how high-performance LLM deployment frameworks like vLLM, TensorRT-LLM, or Hugging Face TGI actually optimize model serving? While you wait for tokens to stream into your chat window, the infrastructure under the hood is executing a fragile balancing act: scheduling prompt pre-computation, paging memory segments, verifying speculative token chains, and dodging system-stalling bottleneck crashes.
To teach you how LLMs manage GPU memory under high concurrent loads, I built an interactive Tetris-style puzzle game:
🧱 PagedAttention: VRAM Tetris
Play in Fullscreen Mode (if the embed sizing is tight)
🛠️ Choose Your Allocation Mode
Your journey as a memory scheduler is split into two distinct memory allocation modes:
- 🏢 Contiguous Mode (Easy/Vanilla): Stacking falling token sequence blocks (Tetrominos) into solid rows. Any gaps you leave behind are trapped, creating unusable External Memory Fragmentation that blocks new incoming allocations.
- 🔋 Paged Mode (PagedAttention - Hard): Play with paged virtualization. Pressing Shift or P triggers a Page Split, shattering the active falling block into individual 1x1 memory pages that cascade down independently to fill any available fragmentation holes below.
🧬 Playable ML Concepts Explained
This isn't just standard Tetris—every shape, block placement, and allocation rule represents a real-world concept in modern machine learning infrastructure. Here is how the in-game mechanics map directly to how large language models allocate GPU memory:
1. 💾 Memory Allocation & Contiguity (Standard Stacking)
- In-Game: You must rotate and slide falling token block shapes to pack them together contiguously. Complete horizontal rows of memory blocks represent completed inference requests, which are garbage-collected to free up VRAM.
2. 🗜️ External Memory Fragmentation (The Stacking Gaps)
- In-Game: Leaving empty spaces under your placed blocks represents external fragmentation. If VRAM fill spikes or blocks stack to the top, the engine crashes, throwing a CUDA OUT OF MEMORY (OOM) error.
3. 🔋 PagedAttention Virtualization (The Page Split)
- In-Game: In Paged Mode, triggering a Page Split shatters the falling shape into individual 1x1 blocks that automatically drop down to seek out and fill the smallest hidden gaps in the memory grid.
🛠️ The Under-the-Hood Engineering Journey
Creating an educational puzzle game designed for embedded platforms presented some fascinating web development challenges:
1. Optimizing for the 600px Embed Limit
Dev.to embeds are capped at a strict maximum height of 600px. Fitting a complex tycoon dashboard with side panels, scoreboards, next-piece canvases, and a 20-row Tetris grid inside 600px required serious spatial compression.
-
The Solution: We shrank the cell block size (
BLOCK_SIZE) to 22px (yielding a 440px canvas height), converted the left panel stats list into a compact 2x2 grid, resized preview boxes to 70px, and relocated the system logs console from a horizontal footer directly into the left sidebar. The final layout fits completely inside exactly 580px, preventing vertical clipping.
2. Physics of the Paged Cascading Split
Splitting a rigid grid structure into individual falling particles in real-time required careful synchronization.
- The Solution: When the split is triggered, the engine parses the active Tetromino shape, decomposes it into coordinate objects relative to the grid columns, calculates the lowest-available free cell index per column, and translates each block to its destination slot before recalculating line-clear sweeps.
Click to see the Page Split Javascript logic
// --- Special Mechanic: Paged Memory Split ---
function executePageSplit() {
if (abilityCharge < 100) {
addSystemLog("PagedAttention Split not fully charged yet!", "warning");
SOUNDS.warning();
return;
}
SOUNDS.split();
addSystemLog(`Executing PagedAttention: Splitting ${currentPiece.name} into virtual pages...`, "info");
// Get all filled cells of the falling piece
const pages = [];
for (let r = 0; r < currentPiece.shape.length; r++) {
for (let c = 0; c < currentPiece.shape[r].length; c++) {
if (currentPiece.shape[r][c]) {
pages.push({
x: currentPiece.x + c,
y: currentPiece.y + r,
color: currentPiece.color
});
}
}
}
// Drop each page individually down its column to the lowest free cell
pages.forEach(page => {
let lowestY = page.y;
while (lowestY + 1 < ROWS && !grid[lowestY + 1][page.x]) {
lowestY++;
}
if (lowestY >= 0) {
grid[lowestY][page.x] = page.color;
}
});
abilityCharge = 0; // Consume charge
clearMemoryLines();
spawnPiece();
}
💬 Let's Discuss:
- What is your high score in Paged Mode utilizing the Page Split ability?
- Did you notice how quickly a contiguous stack triggers a CUDA OOM compared to a paged system?
- How does VRAM Tetris change your perspective on memory allocation bottlenecks?
UnitBuilds-CC
/
PAGED-ATTENTION
A Tetris inspired game to teach how LLMs use VRAM
🧱 PagedAttention: VRAM Memory Allocation Tetris
An educational retro-cyberpunk Tetris-style simulator that maps classic block-packing gameplay directly to GPU memory allocation, external memory fragmentation, and virtual paging concepts.
👉 Play the Live Demo here! (Will be updated after deploy)
🎮 The Concept
In PagedAttention Tetris, you play as a GPU memory scheduler. Incoming requests of varying token sizes (represented by falling Tetris shapes) must be allocated in the GPU's memory registers. Gaps left behind represent External Memory Fragmentation. If memory becomes too cluttered and blocks stack to the top, you trigger a CUDA Out of Memory (OOM) crash.
Playable Memory Allocation Engines:
- 🏢 Contiguous Mode: Falling block sequences remain solid. If gaps are left underneath, they cannot be filled, causing fragmentation and system bloat.
- 🔋 Paged Mode (PagedAttention): Pressing
ShiftorPtriggers a Page Split. The active falling block shatters into individual 1x1 block pages that…
Disclaimer: AI was used throughout this project, it is just fitting that it would co-author with me, so special thanks to the Foundry for its tireless hours toiling away and Gemini for producing the cover image.
Top comments (7)
@xulingfeng The red flag catch you again?
🤣KFC is so good 🍗
Now that was mean 😭
My bad 😂 Next time I'll save you a screenshot of the order receipt so you can at least look at it.
... I'll get you back someday... Wont be today, might not be tomorrow, but I'll get you back for that 1... 😂😭
How you liking the tetris btw?
Dude this is genuinely cool — a Tetris game that actually teaches PagedAttention while you play? The Contiguous vs Paged mode switch is brilliant. 😂 Mad respect for building this. 🙌