Building "Prison Break AI": Local-first Agent Planning + LLM Fallback

#ai #programming #tutorial #dailybuild2026

Large language models are powerful planners, but calling them every physics tick is expensive and fragile. This project demonstrates a hybrid pattern: prefer cheap, deterministic algorithms (A*) for routine planning, and use one-time LLM calls as a background optimization.

System overview

Frontend: React + TypeScript, Vite
Physics: Matter.js
Pathfinding: Grid A* (in src/physics/pathfinder.ts)
Maze generator: recursive backtracker (in src/physics/maze.ts)
Optional LLM planner: proxied local Ollama runtime via /api/ollama (client in src/ai/client.ts)

Architecture diagram

Key patterns and lessons

Staged startup UX: spawn agents visually, compute plans, show them briefly, then start movement — this avoids the app feeling stuck while planning.
Reduce LLM calls: shift high-frequency decision-making to deterministic algorithms; only call LLM for one-time planning or recovery.
Robust LLM client: the app gracefully handles multiple response shapes and caches availability to reduce noise when Ollama is offline.
Physics tuning: increase engine iteration counts and clamp agent velocities to reduce tunneling.

Important code excerpts

Deterministic local planning

From src/physics/pathfinder.ts we expose findPath(maze, startPos, goalPos) that returns world-space waypoints. The app computes local plans for each agent at startup:

const plan = findPath(maze, a.body.position, maze.exitPos);
if (plan && plan.length > 0) a.setPlan(plan);

The agent follows the plan using a small force applied each physics tick toward the next waypoint.

Background AI planner (one-time)

We call aiClient.getPlanFromAI() in the background after agents have started. If the AI plan is shorter than the local plan, we replace the agent plan.

const aiPath = await aiClient.getPlanFromAI(mazeRep, startCell, goalCell, a.data.model);
if (aiPath) {
  const worldPlan = aiPath.map(pt => mapCellToWorld(pt));
  if (computeLength(worldPlan) < computeLength(a.plan)) {
    a.setPlan(worldPlan);
  }
}

This gives the benefit of the AI's strategic insight without adding per-tick latency or cost.

Agent lifecycle

Agent holds a physics body and plan state.
agent.start() flips a running flag so the agent begins following its plan.
agent.isStalled() detects progress stalls and triggers local replan or AI fallback.

Pitfalls and gotchas

If the LLM runtime is not available, the app should not spam the console — we cache availability in aiClient.isAvailable().
Matter.js bodies can tunnel through thin walls if velocities get large; clamping velocities and raising solver iterations helped substantially.

What can you build with this

Add a persistent telemetry backend to log run statistics and models used.
Replace naive waypoint-following with predictive controllers for better platform traversal.
Add evolutionary tuning of plan-follow gains to make agents faster without tunneling.