Ever wondered what a GPU goes through during a massive language model inference run? While you type a query and wait for tokens, the silicon under the hood is holding together a fragile house of cards: balancing context window limits, scheduling activations, managing weights, and evading malicious adversarial attacks.
To teach you how LLMs behave (and fall apart) under load, I built an interactive game:
⚡ GPU Survivors: Latent Space Hell
Play in Fullscreen Mode (if the embed sizing is tight)
🛠️ Choose Your Hardware Preset
Before initiating your run, choose your difficulty configuration (each represented by a unique retro pixel chip sprite and custom parameters):
- 🏢 Enterprise API (Easy): Spawns with 6 Core Integrity Lives, fast speed (
2.8), boosted damage, and a wide collection window. You get+25%XP gains and start with both the Attention Beam and the Softmax Aura active. - 💻 Consumer GPU (Medium): Spawns with 5 Core Lives, normal speed (
2.5), standard damage, and standard100%XP gains. Starts with the Attention Beam active. - 🍞 Smart Toaster (Hard): Edge inference on a kitchen appliance. Spawns with only 4 Core Lives, slow speed (
2.1), reduced damage, and a-20%XP penalty. Starts with a single Attention head active.
🧬 Playable ML Concepts Explained
This isn't just a homage to Vampire Survivors—every upgrade, weapon, and enemy represents a real-world concept in modern machine learning. Here is how the in-game mechanics map directly to how Large Language Models operate, fail, and optimize in production:
1. 🎯 Cosine Similarity (Piercing Vector Arrows)
- In-Game: Fires piercing vector arrows in a fan. Moving in the direction of the fire boosts damage by +60% (aligned vectors), while moving backward deals standard damage.
- The Real-World Counterpart: Text token embeddings are high-dimensional vectors. Cosine similarity calculates the cosine of the angle between two vectors to determine their semantic closeness: $$\text{Similarity} = \frac{\mathbf{A} \cdot \mathbf{B}}{|\mathbf{A}| |\mathbf{B}|}$$
- How it affects LLMs: This is the mathematical core behind Retrieval-Augmented Generation (RAG), semantic database search, and Self-Attention mechanisms. When the user prompt query vector aligns closely with a key vector in the model, the dot product spikes—assigning a massive attention score to pull that context forward.
2. 🗜️ Quantization (Passive Cooldown Upgrade)
- In-Game: Increases weapon firing rate (cooldown speed) at the cost of slightly lower base damage.
- The Real-World Counterpart: Quantization converts model weights and activation outputs from high-precision floating-point formats (like FP32 or FP16) to lower-precision integers (like INT8 or INT4).
- How it affects LLMs: Scaling massive models requires optimization. Quantization drastically reduces VRAM requirements, allowing a 70B parameter model that normally requires enterprise GPUs to run on local laptops. However, rounding values to a coarser scale introduces quantization noise, which slightly degrades the model's perplexity (leading to minor performance loss or "damage").
3. 🧬 Weight Decay (Hitbox Shrinking)
- In-Game: L2 regularization reduces the physical size of the player's core hitbox, making it harder for incoming token anomalies to land a hit.
- The Real-World Counterpart: L2 regularization penalizes large weights by adding a fraction of the squared magnitudes to the training loss function: $$L_{\text{regularized}} = L_{\text{base}} + \lambda \sum_{i} w_i^2$$
- How it affects LLMs: During pre-training, weight decay restrains model weights from growing too large. Keeping weights bounded makes the model less sensitive to minor noise in the user prompt, improving generalization and reducing hallucinations. The resulting "smaller footprint of instability" translates directly in-game to a smaller, more regularized core hitbox.
4. 🧬 Node Dropout (Ignore Hit Check)
- In-Game: Grants a flat +8% chance per level to completely ignore or evade incoming damage.
- The Real-World Counterpart: Dropout is a regularization technique where a random percentage of neural nodes (activations) are zeroed out at each training step.
- How it affects LLMs: By shutting down random neural pathways during training, the model is forced to learn redundant, robust representations rather than relying on a single, fragile sequence of nodes. This prevents the model from overfitting to its training dataset, allowing it to adapt cleanly to unseen prompt distributions at inference time (represented in-game by dropping nodes to safely "evade" bad data).
5. 🔒 Adversarial Split (Jailbreaks)
- In-Game: High-health golden locks. When destroyed, they split into 3 fast-moving Adversarial Tokens that lock onto the player.
- The Real-World Counterpart: A jailbreak is a targeted input sequence designed to bypass the safety alignments (RLHF/DPO) of a model, prompting it to output restricted content.
- How it affects LLMs: Jailbreaks exploit the fact that LLMs treat data and instructions identically. Once a malicious prompt slips past the model's safety guardrails, it triggers an autoregressive cascade of toxic outputs. In-game, this is represented by the sudden explosion of fast-moving adversarial tokens that quickly clutter your active context window.
6. ⚖️ The Horizontal Data Bias (Skewing Fields)
- In-Game: Stepping inside the green Data Bias radius skews your movement coordinate vectors, dragging you in the direction the arrow points.
- The Real-World Counterpart: Data bias occurs when training corpora contain unbalanced representations, stereotypes, or uneven historical distributions.
- How it affects LLMs: LLMs reflect their training datasets. If the data is biased, the output token probability distribution is heavily skewed toward those prejudices. For example, if a model's training data repeatedly associates a profession with a specific demographic, it will struggle to generate neutral completions. This creates a constant, invisible drift that biases output completions, directly mirroring the in-game dragging force.
7. 💾 KV-Cache (The Protective Orbitals)
- In-Game: Key-Value caching blocks rotate around the core, absorbing hits.
- The Real-World Counterpart: The KV-Cache saves the key-value representations of past tokens in GPU VRAM so they don't have to be recalculated at every token prediction step.
- How it affects LLMs: Auto-regressive generation predicts one word at a time, feeding its own output back as input. Without a KV-cache, the model would have to compute attention scores across the entire history for every single token generated, causing latency to scale quadratically. The KV-cache saves computation but consumes huge amounts of memory, restricting user concurrency.
💀 The 15-Minute Thermal Runaway (Endgame)
At exactly 15:00, all standard enemies are swept away, and the unkillable red boss Hardware Degradation arrives. You cannot harm it.
💬 Let's Discuss:
- What was your longest survival time on Smart Toaster difficulty?
- What architectural combination (e.g., Quantization speed boosts + Cosine Similarity) did you find most effective?
UnitBuilds-CC
/
GPU-SURVIVORS
Homage to Vampire Survivors, as an educational game to teach players about LLMs.
GPU Survivors: Latent Space Hell 📟⚡
Can you survive a 1T parameter inference run?
Welcome to GPU Survivors, an interactive 2D retro action-roguelike built to simulate the architectural limits, failure modes, and optimization hyperparameters of running a Large Language Model under load.
🎮 The Scenario
In the digital deep, bad data and chaotic vectors threaten inference stability. You are a GPU Core initializing a new language model. Survive the endless incoming waves of training loads (OOD outliers, prompt injections, and data biases), gather FLOPs (XP), and scale your architecture to 1T parameters!
⌨️ Controls
-
Move: Use
WASDorArrow Keys. -
Pause: Press
EscapeorPto pause the run, resume, or exit. - Attack: Auto-targeted active weapons fire queries at the nearest token anomalies.
🛠️ Hardware Presets (Difficulty Modes)
Select your inference endpoint difficulty at startup:
-
🏢 ENTERPRISE API (H100 Cluster) — Easy
- Stats: 6 Core Integrity…
Disclaimer: AI was used throughout this project, it is just fitting that it would co-author with me, so special thanks to the Foundry for its tireless hours toiling away and Gemini for producing the cover image.
Top comments (9)
Quick question about the KV cache orbitals. In the game they're pure defense, but in real inference the cache's whole cost is memory, so is there any downside to stacking them, like slower movement or a smaller collection radius to stand in for VRAM pressure? That would sneak the memory versus latency trade into the game almost for free. Either way, quantization as a faster fire rate for slightly lower damage is the cleanest mapping in the set, that one teaches itself.
Good point, as a higher tier state change, that's something not covered by this early version on the series, as it's means as an A-Z guide on LLMs, for now the core concepts are isolated to their main purpose, eg. The KV-cache orbital is meant to explain how concepts that would overwhelm the model, are put into cache, where they can be interpreted for longer. While I like the idea of downside stacking, I am running the entire experience for everyone on a single core cloud-run, so limited by the resource constraints, it's as much detail as I can put in, while maintaining a usable speed for everyone. While these designs are meant to teach a concept, their real purpose is to make people ask the deeper questions, such as the consequence to quantization, beyond the obvious like higher throughput, lower quality. where training raises the damage retention, while maintaining speed boost, etc.
Scaling to 1T parameters is a beast of a problem—memory bandwidth becomes the real bottleneck long before compute. I've seen similar issues when working on model parallelism with VoltageGPU, where even small token splits could cause cascading inefficiencies in data flow. It's not just about the raw FLOPs anymore.
The game-as-concept-visualizer approach works surprisingly well here. What I like about the quantization analogy is that it gets at something the "lower precision = worse quality" framing misses: quantization noise isn't uniformly distributed. Some information degrades more than others depending on weight magnitude distributions and outlier values. INT4 on a 70B model doesn't lose the same capabilities as INT8 on a 7B â the loss profile is completely different.
The cosine similarity mapping to "move forward for bonus damage" is clever because it makes the directional nature of attention concrete. People understand that alignment matters, but tying it to spatial movement makes the asymmetry visceral.
I'm curious about SYS_02 (MoE gating network) â that's the most complex mechanic to simulate and the one with the most interesting real-world parallels. MoE load balancing is notoriously tricky because expert routing can collapse: a few "popular" experts get overloaded while others sit idle. How are you handling the adversarial scenario where a user tries to trigger expert collapse, and does it map to actual attacks on MoE systems like the mixture-of-agents attacks that have been published recently?
These series are more meant as introductory courses for people just getting into LLMs, to explain the complex behaviors and nuances in a way that is easy to digest. I try not to dig too deep into each for the early part of the series, as that would lead to overwhelming mechanics that just destroy the user's perception of it. The gating network is a simple task "route traffic amongst the MoEs", simple and easy if you make it so, but raise the rate, lower the activation threshold, tighten to 4 experts instead of 8, etc. Suddenly people can visually and emphasize with the concept of over-utilization of generalist experts, which is why it heavily biases towards 2 and 7 (if you play it enough, you'll see the pattern), this is so when the user ups it to see how fast they can do it, that they eventually hit a wall where the experts cant keep up, so suddenly a highly technical math, coding problem, gets routed to generalist and math, which inherently wont be a good combo for the task of coding, or coding and generalist, resulting in decent code, yet broken algorithms. That way the real tradeoff of going MoE (sparse) is shown to be something that is unavoidable. If a user was deciding whether or not to go MoE on a specific task, the game will make them think twice before choosing, as choosing wrong could end up costing them more than just a grammatic error, it could result in clean, tested, verified, useless code because the wrong expert assisted due to overload.
Really fun framing. I especially liked how you mapped cosine similarity / RAG / self-attention into gameplay instead of stopping at surface-level analogies. The quantization section was also a nice touch—explaining the speed/VRAM tradeoff as a slight damage penalty makes the production tradeoff feel intuitive. Curious if you’ll do a follow-up level around KV cache pressure or batching/scheduling too.
Please @ someone who can review the PR to fix the Embed window sizing, so the games actually look and run nicely inside a post: github.com/forem/forem/pull/23553
Thank you and enjoy the game!
Thoughts sofar? Everyone enjoying the game?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.