Harish Kotra (he/him)

Posted on May 24

Building a 3D Multi-Agent AI Conference Simulator with React, Three.js, Zustand, and Local LLMs

#ai #programming #javascript #dailybuild2026

Most AI demos are either chat boxes or benchmarks. This project explores a different shape: what if local models were not just answering prompts, but acting as characters inside a shared simulation?

The result is the Virtual AI Conference Simulator, a browser-based 3D conference floor where five autonomous AI attendees walk in, visit sponsor booths, talk to one another, react to live events, pursue private goals, and generate a recap of what happened.

The Core Idea

The simulator treats a conference as an event-driven multi-agent world:

The 3D engine handles navigation, collision, and rendering.
Each AI attendee runs an independent LLM heartbeat.
LLMs output structured decisions, not coordinates.
The app turns those decisions into physical movement.
Conversations feed a social graph, booth leaderboard, topic heatmap, and highlight reel.

The key design choice is separation of responsibility:

The LLM decides intent. The frontend decides motion.

That keeps the simulation stable. A local model can say "go to the DevTools booth" or "ask a keynote question" without needing to understand coordinates, collision, or frame timing.

Tech Stack

The app uses a small OSS stack:

React and Vite for the app shell.
@react-three/fiber for declarative Three.js.
@react-three/drei for helpers such as <Html /> overlays.
three-mesh-bvh for spatial foundations.
Tailwind CSS for the settings and simulation UI.
Zustand for global state.
Trystero for lightweight peer sync.
OpenAI-compatible HTTP endpoints for local or remote LLM calls.

This means the same app can run against LM Studio, Ollama-compatible servers, or remote APIs that expose /v1/chat/completions.

System Architecture

The project is intentionally modular. Each feature has a clear home:

store.js: world state and mutations.
ConferenceMap.jsx: 3D scene and spatial monitor.
AIAgentAvatar.jsx: attendee rendering, movement, and heartbeat.
BoothRepAvatar.jsx: stationary rep behavior.
inference.js: prompt builder and endpoint handler.
directorEvents.js: scheduled and random events.
socialGraph.js: relationship memory.
boothMetrics.js: booth leaderboard.
objectives.js: private attendee goals.
topicHeatmap.js: visual topic hotspots.
recap.js: shareable conference summary.

Modeling the Conference Floor

The floor is not just decorative. It is a structured world with visitable targets:

Entry Gate
Main Stage
Networking Lounge
Coffee Bar
Poster Walk
Selfie Wall
AI Infrastructure Booth
DevTools Booth
Open Source Models Booth

Each booth has both a physical position and safe visit positions:

{
  id: 'DevTools-Booth',
  name: 'DevTools',
  position: { x: 9.2, y: 0, z: 1.2 },
  visitPosition: { x: 9.2, y: 0, z: 2.75 },
  visitPositions: [
    { x: 8.55, y: 0, z: 2.75 },
    { x: 9.2, y: 0, z: 2.85 },
    { x: 9.85, y: 0, z: 2.75 },
  ],
  size: { x: 3.8, y: 2.35, z: 2.5 },
  color: '#059669',
  description:
    'Agent IDE plugins, eval harnesses, prompt regression suites, code review copilots, and local developer workflows.',
}

The important detail is that agents do not target the inside of a booth. They target visitPositions, which keeps them visually outside desks and props.

Collision-Aware Navigation

The first version of the scene was visually interesting but too easy to break. Agents could walk through booths, tables, chairs, walls, and stage objects.

The fix was to make collision a first-class data layer:

export const STATIC_OBSTACLES = [
  ...BOOTHS.map((booth) => ({
    id: `${booth.id}-structure`,
    kind: 'booth',
    type: 'rect',
    center: { x: booth.position.x, z: booth.position.z },
    half: { x: booth.size.x / 2, z: booth.size.z / 2 },
  })),
  {
    id: 'main-stage-platform',
    kind: 'stage',
    type: 'rect',
    center: { x: 0, z: -8.8 },
    half: { x: 4.3, z: 1.25 },
  },
];

Movement uses a simple but effective resolver:

export function moveWithCollision(current, desired, options = {}) {
  const radius = options.radius ?? 0.3;
  const currentPoint = clampToFloor(current, radius);
  const desiredPoint = clampToFloor(desired, radius);

  if (!isPointBlocked(desiredPoint, options)) {
    return desiredPoint;
  }

  const xOnly = { x: desiredPoint.x, y: desiredPoint.y, z: currentPoint.z };
  if (!isPointBlocked(xOnly, options)) return xOnly;

  const zOnly = { x: currentPoint.x, y: desiredPoint.y, z: desiredPoint.z };
  if (!isPointBlocked(zOnly, options)) return zOnly;

  return currentPoint;
}

This is not a full navmesh, but it is reliable enough for the current floor plan and easy for contributors to understand.

The LLM Loop

Each attendee has an independent heartbeat. When the heartbeat fires, the app collects:

Current agent profile.
Current state and target.
Nearby agents.
Active conversation history.
Recent world history.
Booth context.
Conference director event.
Social memory.
Private mini objective.

Then it asks the configured endpoint for a JSON decision.

const decision = await queryAgentDecision({
  agent: current,
  agents: [...state.agents, ...state.boothReps],
  booths: state.booths,
  lounge: state.lounge,
  nearbyAgents,
  activeConversation,
  chatHistories: state.chatHistories,
  worldHistory: state.worldHistory,
  boothContext,
  directorEvent: state.director.activeEvent,
  socialMemory,
  objective: state.objectives[current.id],
});

The model is constrained to a small action space:

const ALLOWED_ACTIONS = [
  'GO_TO_BOOTH',
  'GO_TO_LOUNGE',
  'GO_TO_ZONE',
  'GO_TO_STAGE',
  'GO_TO_ENTRY',
  'TAKE_SELFIE',
  'TALK_TO_AGENT',
  'BROWSE_BOOTH',
  'STAY_IDLE',
];

The expected response looks like this:

{
  "thought": "The keynote question is heating up. I want to hear the benchmark debate directly.",
  "action": "GO_TO_STAGE",
  "target": "Main-Stage",
  "dialogue": "I want to hear the benchmark answer before anyone turns it into a chart."
}

The app validates the JSON and falls back safely if the response is malformed:

try {
  return await requestOpenAICompatibleDecision({ agent, messages });
} catch (error) {
  return fallbackDecision(
    error?.name === 'AbortError' ? 'Inference request timed out' : error.message,
    agent,
    objective,
  );
}

That fallback matters because local model servers can be stopped, slow, or inconsistent during development.

Model-vs-Model Persona Drama

Instead of hiding model choice in a settings panel, the simulator makes it visible.

The default local models are:

export const LOCAL_LM_STUDIO_MODELS = [
  'nvidia/nemotron-3-nano-4b',
  'google/gemma-4-e4b',
];

Each model receives a behavioral lens:

Gemma-powered agents tend to be concise and summarizing.
Nemotron-powered agents tend to be more skeptical and proof-seeking.

This turns model comparison into a live simulation. You are not only comparing benchmark scores. You are watching how model personalities affect routing, conversations, questions, and booth conversion.

Conference Director Events

Idle roaming becomes much more interesting when the world itself changes.

The director system creates scheduled and random reasons to move:

Keynote starting.
Booth demo launch.
Coffee shortage.
Wi-Fi outage.
Surprise product announcement.
Controversial audience question.
Hackathon challenge.
Investor walkthrough.

Example event:

{
  id: 'hackathon_challenge',
  type: 'scheduled',
  title: 'Hackathon Challenge',
  announcement: 'The DevTools booth announced a 20-minute agent workflow hackathon.',
  target: 'DevTools-Booth',
  action: 'GO_TO_BOOTH',
  priority: 0.8,
  durationMs: 22000,
  topics: ['hackathon', 'developer tools', 'evals'],
}

The human can also inject chaos. This is the sandbox layer: announce a security breach, force a keynote, offer free credits, or ask everyone about licensing.

Social Graph

The simulation remembers relationships.

When agents meet, the store records an edge:

recordConversationStart(current.socialGraph, first, second, eventTopics);

When agents speak, the app extracts topics and sentiment:

const topics = extractTopics(message.text);
const score = scoreMessage(message.text);

Over time, each agent accumulates memory:

Who they met.
Who they liked.
Who bored them.
Which topics they care about.
Which booths convinced them.

This creates storylines such as:

Akira keeps avoiding the AI Infrastructure rep because the licensing answers are not satisfying.

That is where the simulator starts to feel less like an animation and more like an agent world.

Booth Leaderboard

The booth leaderboard tracks:

Visits
Dwell time
Questions asked
Leads captured
Sentiment
Conversions

This turns sponsor booths into a game mechanic. The conference is no longer just a scene; it has winners, losers, and measurable outcomes.

The leaderboard is generated from boothMetrics.js, then rendered in Scoreboard.jsx.

Mini Objectives

Each attendee has a private objective:

Find a vendor.
Collect stickers.
Ask a keynote question.
Compare two booths.
Avoid salespeople.

Objectives are not hard-coded movement scripts. They are prompt pressure plus event tracking.

export const MINI_OBJECTIVES = {
  collect_stickers: {
    id: 'collect_stickers',
    title: 'Collect stickers',
    description: 'Visit two different booths and leave with enough swag to prove it happened.',
    prompt: 'Collect stickers by visiting at least two different booths.',
  },
};

Progress updates from real simulation events:

recordObjectiveEvent(agentId, {
  type: 'booth_visit',
  boothId: booth.id,
  boothName: booth.name,
  reason,
});

That makes movement intentional while preserving emergent behavior.

Topic Heatmap

The topic heatmap shows where conversations are happening and what they are about.

Current topic categories:

Evals
Infra Cost
Open Models
Governance
Developer Tools

The heatmap combines:

Active conversation messages.
Booth chat histories.
Booth metrics.
Director event topics.

Then it renders translucent floor circles in the 3D scene.

const points = buildTopicHeatmapPoints({
  agents,
  boothReps,
  player,
  booths,
  activeConversations,
  chatHistories,
  socialGraph,
  boothMetrics,
  director,
});

This is one of the most important legibility features. Multi-agent systems can become hard to follow quickly. A heatmap gives viewers a visual summary of the live narrative.

Highlight Reel

At the end, the simulator can generate a recap:

Funniest quote.
Hottest booth.
Best agent debate.
Most social agent.
Ignored booth.
Top model.

The recap is designed to be shareable. A good simulation should create moments that users want to post.

Why This Should Interest You as a Developer

This project sits between several useful patterns:

Structured LLM output in a real-time UI.
Local model orchestration without backend dependency.
Multi-agent memory without heavy infrastructure.
Event-driven simulation state.
Visual debugging for agent behavior.
Human-in-the-loop chaos controls.

It also highlights a practical rule for agent UX:

Do not make the LLM responsible for every layer of the system.

The model should reason about intent, conversation, and priorities. The deterministic engine should handle physics, input, collision, validation, and rendering.

What Could Come Next

Good extensions:

Deterministic replay mode.
Save and load conference sessions.
Screenshot export for the selfie wall.
Voice lines for agents.
More booth templates.
Sponsor persona packs.
Playwright tests for movement and UI panels.
Backend room server for persistent multiplayer.
Real analytics export for simulation runs.
Timeline scrubber for the highlight reel.
Navmesh pathfinding for larger venues.
Agent memory persistence across sessions.

The Virtual AI Conference Simulator is a small but complete example of agentic UX outside the chat box. It gives local models bodies, goals, relationships, events, places to go, and a world that reacts to them.

That is the core reason the project is fun: you are not reading an answer from a model. You are watching a tiny AI conference unfold.