# **A Storm Didn’t Decide This Battle: How a Sergeant Fixed Our Broken Tactics Engine**
At 3 AM, the pager didn’t scream because of a storm. It screamed because a sergeant in our "deterministic" tactics engine had just walked through a mountain. Not a glitch, not a rendering error, a *ghost tile*. A leftover from a pathfinding cache that never got flushed. The logs showed the same seed, the same orders, but the outcome was different. The mountain wasn’t in the terrain grid. It was in the *memory*.
We’d sold this system as deterministic. We were wrong.
Determinism isn’t about the code you write. It’s about the code you *don’t* write, the edge cases, the race conditions, the silent failures. This is how we fixed it, under 8GB RAM, with zero downtime, and no excuses.
---
## **The Root Cause: Where the Blueprint Failed**
The architecture looked solid: immutable state, bounded queues, fixed-point math. But the devil was in the hardware constraints and the things we *assumed* would work.
### **1. The Ghost Tile Problem (Cache vs. Grid Desync)**
The terrain system used a 2D grid for collision detection, while pathfinding used a separate cache. When a unit moved, the cache updated, but the grid didn’t. The sergeant’s pathfinding used the cache; collision detection used the grid. Result? A unit that walked through walls.
**Failure Walkthrough:**
1. **Initial State:**
python
terrain_grid = ((1, 1, 1), (1, 1, 1), (1, 99, 1)) # 99 = mountain
path_cache = {(0, 0): 1, (0, 1): 1, (0, 2): 1, (1, 0): 1, (1, 1): 1, (1, 2): 99}
2. **Unit Moves to (1,2):**
- Pathfinding checks `path_cache[(1,2)]` → `99` (impassable).
- Collision detection checks `terrain_grid[1][2]` → `1` (walkable).
3. **Race Condition:**
- If the cache updates *after* collision detection, the unit moves into an invalid tile.
**Fix: Single Source of Truth (SSOT).** The terrain grid became the only source of truth. The pathfinding cache was eliminated. Movement costs were precomputed at startup.
python
Precomputed movement costs (hex grid)
MOVEMENT_COSTS = {
(0, 0): 1, (0, 1): 1, (1, 0): 1, (1, 1): 2, # Flat terrain
(2, 2): 99, # Mountain
}
def get_movement_cost(x: int, y: int) -> int:
# Default to impassable if tile not found
return MOVEMENT_COSTS.get((x, y), 99)
### **2. The Floating-Point Drift (Non-Deterministic Math)**
We’d sworn off floating-point math, but unit movement still used division for acceleration curves. Over 10,000 turns, rounding errors accumulated. Two identical games diverged by a single pixel after 5 minutes, enough to change battle outcomes.
**Failure Walkthrough:**
1. **Initial State:**
python
dx = 100 / 3 # Floating-point: 33.333333333333336
dx_fixed = (100 << 16) // 3 # Fixed-point: 33.33332824707031
2. **After 10,000 Turns:**
- Floating-point: `33.333333333333336 * 10000 = 333333.33333333336`
- Fixed-point: `33.33332824707031 * 10000 = 333333.2824707031`
- **Divergence:** `0.05086263024902344` pixels.
**Fix: Fixed-point math everywhere.** All division replaced with bit shifts.
python
def move_unit(unit: Unit, dx: int, dy: int) -> Unit:
# Scale back to integer coordinates
new_x = unit.x + (dx >> 16)
new_y = unit.y + (dy >> 16)
return Unit(id=unit.id, x=new_x, y=new_y, health=unit.health, state="moving")
### **3. The Backpressure Lie (Silent Order Drops)**
The order queue was bounded to 1,024 entries, but the rejection policy was naive. When full, new orders were silently dropped. A client sending 2,000 orders per second saw half vanish into the void.
**Failure Walkthrough:**
1. **Queue State:**
python
order_queue = deque(maxlen=1024)
2. **Client Sends 2,000 Orders:**
- First 1,024 orders accepted.
- Next 976 orders *silently dropped*.
3. **Result:**
- Client assumes all orders were processed.
- Game state diverges.
**Fix: Explicit backpressure.** The queue returns a status code for every push. Clients retry with exponential backoff.
python
class OrderQueue:
def init(self, max_size: int = 1024):
self._queue = deque(maxlen=max_size)
def push(self, order: dict) -> tuple[bool, str]:
if len(self._queue) >= self._queue.maxlen:
return False, "queue_full"
self._queue.append(order)
return True, "ok"
---
## **Hardware Constraints: 8GB RAM or Bust**
The engine had to run on 8GB RAM cloud instances. No swap, no OOM kills. We profiled memory usage and found three leaks.
### **1. The Immutable State Leak (Deep Copies)**
The `GameState` was immutable, but every turn created a deep copy. For 10,000 units, this meant 10,000 deep copies of a 1KB struct per turn.
**Failure Walkthrough:**
1. **Initial State:**
python
@dataclass(frozen=True)
class GameState:
units: FrozenSet[Unit] # Deep copy on every turn
2. **After 10,000 Turns:**
- Memory usage: `10,000 * 1KB = 10MB` per turn.
- Total: `10,000 * 10MB = 100GB` (OOM kill).
**Fix: Copy-on-write.** Only modified units are copied.
python
@dataclass(frozen=True)
class GameState:
units: FrozenSet[Unit] # Shared if unchanged
_modified_units: dict[int, Unit] = field(default_factory=dict)
def update_unit(self, unit: Unit) -> "GameState":
new_units = self.units - {unit} | {unit}
return replace(
self,
units=new_units,
_modified_units={**self._modified_units, unit.id: unit}
)
### **2. The Log Explosion (Unbounded Growth)**
The event log was append-only. After 10,000 turns, it consumed 1GB RAM.
**Failure Walkthrough:**
1. **Initial State:**
python
event_log = []
2. **After 10,000 Turns:**
- Log size: `10,000 * 100KB = 1GB`.
**Fix: Circular buffer.** Old events are overwritten.
python
class CircularLog:
def init(self, max_size: int = 10_000):
self._log = [None] * max_size
self._index = 0
def append(self, event: dict) -> None:
self._log[self._index] = event
self._index = (self._index + 1) % len(self._log)
### **3. The Pathfinding Cache Bloat (LRU Eviction)**
The A* cache grew without bound. After 1,000 turns, it consumed 500MB.
**Failure Walkthrough:**
1. **Initial State:**
python
path_cache = {}
2. **After 1,000 Turns:**
- Cache size: `1,000 * 500KB = 500MB`.
**Fix: LRU cache with fixed size.**
python
from functools import lru_cache
@lru_cache(maxsize=1_000)
def find_path(start: tuple[int, int], end: tuple[int, int]) -> list[tuple[int, int]]:
# A* pathfinding implementation
pass
---
## **Race Condition Resilience**
The engine had to survive network partitions and backpressure. We hardened it with:
### **1. Deterministic RNG (No System Random)**
We replaced `random` with a pure-Python PCG64 RNG, seeded at startup.
python
class PCG64:
def init(self, seed: int):
self.state = seed
self.inc = 1442695040888963407 # Arbitrary odd constant
def randint(self, low: int, high: int) -> int:
self.state = (self.state * 6364136223846793005 + self.inc) & 0xFFFFFFFFFFFFFFFF
xorshifted = ((self.state >> 18) ^ self.state) >> 27
rot = self.state >> 59
return low + ((xorshifted >> rot) | (xorshifted << ((-rot) & 31))) % (high - low + 1)
### **2. Zero-Downtime Workflows (State Boundaries)**
The engine was split into:
- **State Manager** (single source of truth).
- **Execution Engines** (stateless, scalable).
plaintext
┌───────────────────────────────────────────────────────────────┐
│ Distributed Engine │
├───────────────┬───────────────────┬───────────────────────────┤
│ State │ Execution │ Network │
│ Manager │ Engines │ Layer │
└───────────────┴───────────────────┴───────────────────────────┘
**State Manager:**
- Holds the canonical `GameState`.
- Serializes state to disk
Top comments (0)