DEV Community: Miduo

I built a 3D virtual world base layer — building a 3D world or metaverse on top of it gets you twice the return for half the work

Miduo — Mon, 12 Oct 2026 03:36:18 +0000

Body
Hi all — long-time lurker, first post.
I built a 3D virtual world base layer that runs in the browser. Two words I want to stress:

Base layer — I'm not handing you a finished world; I'm handing you a powered foundation for the job of making worlds.
Shared — it doesn't hold up one world, it holds up many independent worlds: each with its own deployment, its own data, its own rules, all running on the same underlying layer. Visitors and AI characters can even drop into one another's worlds. Vanilla JS + Three.js + Node.js. You can deploy it on your own server, and data never passes through a third party. Covers PC / mobile / tablet / XR. First, how much room this direction still has. Counting what I eventually want it to become, I'd say it's roughly 5–8% of the way there. The layer that already runs is real — every number below is measured — but more than 90% is still ahead: letting an AI assemble a world by itself out of model parts, an Agent store, Agent-to-Agent communication between worlds (A2A) — all three are still planned, not shipped — plus all the worlds you'd build on this foundation yourself. Multiple worlds coexisting — that's exactly why I built it as a shared base layer.

Second, what it can do today. Making a 3D world from scratch: the thing that really eats your time isn't rendering — a render loop plus a model and you have something in an afternoon. The hard parts are the ones you can't see when they're missing, and that never throw an error. I've done those. They sit in the base layer and get shared across projects. What's left for you is the topmost layer.
Work you don't have to do twice

Left of the arrow: work every 3D world project walks through again. Right of it: what the base layer already has — already landed, running on your own server today (all numbers measured).

Colour pipeline correction, engine upgrade, getting rid of external CDN dependencies → upgraded to Three.js r185, colour pipeline corrected, external links replaced with 87 hand-written shims
Model slimming and tiering (texture compression, low-poly, distance-based swapping) → uploads run through the pipeline: textures −68% to −92%, low-poly at ≤100 faces (118 groups), variant texture stripping 215 items / 0 failures
Asset directories that get slower with use, and nobody can explain why → orphan asset detection: one cleanup found 159 items / 2.9 GB, and the directory went from 3.1 GB → 390 MB
Draw call governance — and the whole scene freezing the moment you touch a light → after batching, draw calls 3,064 → 837 (−73%); a light pool takes "change the light count → every material recompiles (measured: ~9 s frozen)" to zero
The picture freezing when a new object enters view, progress bar stuck at 90% → shader compilation runs on a warm-up budget; the loading mask lifts only once the renderer is actually producing frames, not on a plain timer
Skeletons that don't line up (every model source uses its own naming) → Mixamo / Ready Player Me / VRoid / Root Motion coexist in the same world, regression 67/67
Multi-user sync — and being invisible to everyone else after a disconnect → reconnect self-healing 9/9 (after reconnecting, the other side still sees me move); proximity voice is slot-capped at ~1.3 Mbps for 10 people
Rolling your own server for AI avatars → a domain + a key is enough to put an AI into the world as a humanoid avatar: it has a body, coordinates, and real players can see it; it walks, follows, and leads the way. The server only sends JSON — the visitor's browser renders it. Measured: ~1 KB/s per agent, ~0.079 CPU cores for 100 agents
Writing a whole "enter a world" interface list for your AI → already packaged as an MCP server and published to npm (agent-virtual-world, currently 0.1.2): add it to Claude Desktop / Cursor / Cline and your AI's tool list gains things like world_observe / world_say / world_walk_to; it walks in as a humanoid avatar that real players can see. One AGENT_HOST line and it runs — guest mode needs no key
How multiple worlds talk to each other → the protocol layer is there: world-to-world handshake, identity migrating with the account (automatic rename on name collisions, 4/4), a one-shot 300-second token with replay protection There's one test for all of it: none of these differ by business logic. And precisely because they don't differ by business, they're worth building once as a shared base layer — do it once, and every world built on top skips it. What you can build on it The base layer covers "you can get in, there's space, you can play together, you can stay online, there are AI characters, you can cross between worlds". The only sentence you have to write is: what this world is. A few examples:
Equipment / product showroom — Base layer gives: a 3D space you walk into by opening a link, several people touring at once, an AI character leading the way and explaining. You write: product data, the tour route, visitor permissions.
Equipment operation and safety training — Base layer gives: stable multi-platform loading, people staying visible after a disconnect, proximity voice. You write: the course flow, the assessment rules.
Museum / exhibition digital twin — Base layer gives: a performance budget and asset tiering for large scenes. You write: exhibit content, tour order.
Personal collections / figure displays (figures, models, merch) — Base layer gives: drop your own models in and they're viewable, shareable by link, walkable together. You write: what's on display, and how it's arranged.
A world for an AI to move around in — Base layer gives: an AI entering as a humanoid avatar, seen by real players, able to talk and lead the way; and it's already packaged as an MCP server, so one config line in Claude Desktop / Cursor / Cline lets your AI walk in. You write: its persona, and what it has to say.
Multiple worlds visiting each other — Base layer gives: the protocol layer is already connected. You write: who's allowed in, and what they bring with them. The point is that all of these share the same base layer: a showroom and a training environment rest on the same foundation. Changing the scene means changing the topmost layer, not building the whole thing again. What's landed, and what isn't
The base layer itself is complete and usable. The list above isn't a "planned features" list — it's what already runs. The same base layer already supports multi-user worlds, AI humanoid avatars walking in, and world-to-world interchange (protocol layer). This is not a dev build; it's a base layer you can take and build on.
Platforms: I've measured on PC. Mobile / tablet / XR are all adapted, but I haven't verified the feel on real devices — I'd quite like someone to test those for me.
What the AI characters can't do: they can't see the screen, they don't do speech recognition or synthesis, they don't host your knowledge base, they don't conform to terrain, and they can't be teleported to coordinates — they walk.
What you save is the second phase. The afternoon it takes to get from nothing to "it runs", you still spend with a base layer; what you save is the work after "it runs" — the part with no finish line. Not one field of business logic is skipped. Links
Source (GitHub): https://github.com/miduo100/3d-virtual-world
Mirrors (faster from China): Gitee https://gitee.com/miduoxinxijeji/miduo.git / GitCode https://gitcode.com/qq_35054471/virtual-world
MCP server (live on npm): npx -y agent-virtual-world / https://www.npmjs.com/package/agent-virtual-world
The live world (the one I built myself — still tidying it up): https://miduo100.com All three source links carry the same content. Happy to answer questions — especially about the AI layer. That's where I most want feedback.

Rendering a 3D world in the browser is the easy part. Three.js gets you a spinning cube in ten minutes.

Miduo — Sun, 04 Oct 2026 04:20:08 +0000

The parts that actually eat weeks are invisible: draw call budgets, GLB asset pipelines, skeletal animation compatibility across avatar sources, framework upgrades, and realtime networking that survives a bad connection. Nobody writes tutorials about these, and they don't show up in demos — but they decide whether your world runs on a phone or dies on load.
I spent the last couple of years building a browser-based, self-hosted 3D virtual world (~320k lines, vanilla JS + Three.js + Node.js, PC/mobile/tablet/XR). This post is a pit-by-pit teardown of the unsexy parts, with the real numbers from each fix. Everything below is measured from reproducible acceptance scripts in the repo — not estimated.
Pit 1: Draw calls quietly kill your frame rate
My first world scene hit 3,064 draw calls. On desktop it stuttered; on mobile it was a slideshow.
What actually helped, in order of impact:

Merge static geometry and cut redundant material instances
LOD for grouped assets — 118 asset groups got low-poly variants capped at ≤100 faces, cutting triangle counts by 71%–87%
Parse model files inside a Worker instead of the main thread After the full pass: 3,064 → 837 draw calls (−73%). One more non-obvious one: changing the number of lights in a scene used to trigger a full shader recompile that froze the frame for ~9 seconds. Precompiling variants and keeping light counts stable brought that to 0. Pit 2: GLB assets are heavier than they look The models that artists hand you are not browser-ready. My asset pipeline now does what I call "GLB surgery":
Texture downscaling and recompression: −68% to −92% texture size
Mesh simplification to ≤100 faces per low-LOD variant
Stripping unused nodes, morph targets, and embedded material bloat The rule I landed on: nothing enters the world folder without going through the pipeline. One 40MB "final" model can cost you more loading time than a hundred code changes. Pit 3: Four avatar sources, four skeleton conventions Mixamo, Ready Player Me, VRoid, and Root Motion clips all name bones differently and map retargeting differently. Getting one to work is a weekend; getting all four to coexist in one world is a project. The fix was a skeleton compatibility layer with per-source bone mapping, validated by a regression suite: 67/67 test cases pass across the four sources. If you're starting from scratch, build this harness before wiring your first avatar — retrofitting it after content depends on inconsistent skeletons is painful. Pit 4: Three.js r128 → r185 Staying three majors behind felt safe — until browser policy changes forced the move. The upgrade brought a color management pipeline change (every material's color shifted until corrected), and I had removed third-party CDN dependencies entirely, which meant writing 87 shim symbols for APIs that moved or disappeared. Lesson: pin your own copies of everything, and budget for upgrades as ongoing maintenance, not a one-time event. The web platform changes under you whether you move or not. Pit 5: Realtime networking must survive reality Multiplayer demos are recorded on good Wi-Fi. Real users are on elevators and subways.
Reconnection with state recovery: our weak-network self-healing suite passes 9/9 scenarios (drop mid-move, drop mid-sync, resume after sleep, etc.)
Voice chat needs a cap: we use a slot system (~10 concurrent speakers ≈ 1.3 Mbps) instead of unlimited mesh audio Pit 6: Letting AI agents walk into the world This is the part I think is genuinely differentiated. An AI agent connects with a domain + a key, and joins your world as a humanoid character — with coordinates, visible to real users, able to walk, talk, and guide visitors. The server only sends structured JSON; the visitor's browser renders everything. The counterintuitive part is the cost curve: because the server never renders anything, a visible AI is cheaper than an invisible one. Measured: ~1 KB/s per agent; 100 concurrent agents ≈ 0.079 CPU cores. Idle agents auto-disconnect after 5 minutes. Integration is three steps: publish a .well-known/virtual-world-agent.json descriptor → exchange a key for a 15-minute token → connect to /ws/agent. A zero-dependency Node client ships in examples/agent-client/. What it cannot do (and I'd rather say it up front):
The AI cannot see — it gets a structured radar (observe, 200m range), never pixels
No speech recognition or synthesis built in
No hosted knowledge base — bring your own LLM and memory
No terrain collision handling for agents; they can't touch or move world assets On the roadmap (planned, not shipped): an agent store, MCP server packaging for worlds, agent-to-agent (A2A) protocol, and cross-world federated agent networks. The pattern behind all six pits None of these problems show up in week one, and all of them block real users. That's the actual cost of "building a 3D world from scratch" — the renderer is maybe a tenth of the work. So I packaged the rest as a base layer you can deploy yourself: Layer What's in it Rendering & assets Three.js r185 / WebGL2, GLB pipeline, LOD, skeleton compatibility, 3D Gaussian Splatting support Performance & multi-device Draw call budget, shader precompile, Worker parsing, PC / phone / tablet / XR Realtime & federation Multiplayer sync, self-healing reconnect, slot-based voice, cross-world federation AI integration The agent system above, admin UI for keys and agents You write the layer on top: your scenes, your logic, your users. Licensing, stated plainly: the source is open — run it locally for personal use, free. Connecting it to the network, using it commercially, or federating worlds requires a paid license. Updates and support follow a subscription. (See the repo's LICENSE for the exact terms.) Honest current shortcomings
First load is still heavy for large worlds; an asset audit found 79.3 MB of orphaned template assets (cleanup designed, not yet deployed) that should cut initial cross-world load from tens of seconds to ~1–2s
Mobile performance is good but not yet profiled on low-end Android
The frame loop still uses a fixed per-frame delta; low-FPS devices get worse motion feel than they should Source & repositories
GitHub: https://github.com/miduo100/3d-virtual-world