DEV Community

Daniel Ioni
Daniel Ioni

Posted on

Building ZORGAX Party Mode: Giving Our Metaverse an Intelligent Orchestration Layer

#zorgax #metaverse #party #mode

Building ZORGAX Party Mode: Giving Our Metaverse an Intelligent Orchestration Layer

We're building a new layer inside the MyZubster open-source ecosystem called ZORGAX Party Mode.

The idea sounds simple:

What if an AI system could understand the current state of a social/virtual experience without becoming an unrestricted agent with access to everything?

That's the engineering problem we're working on.

ZORGAX Party Mode is being designed as the orchestration layer connecting:

Community → Public Event → Virtual Twin → Live Session → Archive/Replay → Community

But we're deliberately building it one capability at a time.

No fake "AI-powered metaverse" claims.

No pretending roadmap features already exist.

Here's where we are.

What Is ZORGAX Party Mode?

ZORGAX is not intended to be just another chatbot inside a virtual world.

We're designing Party Mode as a bounded orchestration layer.

It should eventually understand things such as:

  • Which community is associated with an experience
  • Which public event is active
  • Which virtual room is involved
  • Whether a session is live
  • Whether infrastructure is healthy or degraded
  • Which creator actions are permitted
  • Which moderation workflows are available
  • What happens when a live experience becomes an archive

The important word is bounded.

ZORGAX does not become the authority for identity, authorization or unrestricted system access.

Those decisions remain server-side.

The Current Architecture

The current flow looks like this:

MyZubster Identity
        ↓
   PartyContext
        ↓
 Community Assistant
        ↓
 Room Telemetry
        ↓
 Creator Capabilities
        ↓
 Moderation
        ↓
 Archive / Replay
Enter fullscreen mode Exit fullscreen mode

We're implementing this as seven development blocks:

Z1 — Party Context
Z2 — Community & Event Assistant
Z3 — Room/Session Telemetry
Z4 — Creator Commands
Z5 — Moderation Assistance
Z6 — Archive & Replay
Z7 — Safety + End-to-End Acceptance
Enter fullscreen mode Exit fullscreen mode

The first two are now implemented and merged.

We're currently building Z3.

Z1 — PartyContext

The first problem was surprisingly fundamental.

Before an AI can help with an event or virtual environment, we need to answer:

What is it actually allowed to know?

We introduced a server-generated PartyContext.

Conceptually, it represents:

Actor
Community
Public Event
Room
Session
Capabilities
Restrictions
Enter fullscreen mode Exit fullscreen mode

But it is intentionally not a dump of our database.

The server decides what enters the context.

Client claims are never automatically considered verified.

Sensitive/private information is excluded.

That means PartyContext does not become a convenient way of exposing:

  • Authentication tokens
  • Internal account IDs
  • Email addresses
  • Private keys
  • Raw session identifiers
  • Sensitive/private coordinates
  • Financial information
  • Permanent movement history

During our pre-merge review we actually caught an early version exposing an internal account identifier.

We removed it and added a regression test.

That's exactly why we're building this incrementally.

Z2 — A Grounded Community Assistant

Once PartyContext existed, we could build the first ZORGAX assistant.

The endpoint is currently:

POST /api/zorgax/party-assistant
Enter fullscreen mode Exit fullscreen mode

The interesting part isn't that it can answer questions.

The interesting part is what happens when it doesn't know the answer.

For example, the production repository does not yet have the complete public-event data layer required for Party Mode.

So if someone asks:

Who is playing at the event?
Enter fullscreen mode Exit fullscreen mode

ZORGAX doesn't invent a lineup.

It reports that authorized event information is unavailable.

Similarly, requests for private coordinates, account information or other restricted data are rejected by the Party Mode boundary.

This gives us a useful principle:

Missing context should produce "unavailable", not hallucinated product state.

That becomes increasingly important once an assistant can interact with live infrastructure.

Z3 — Teaching ZORGAX to Understand the Room

This is what we're implementing now.

Our current MyZubster Metaverse environment, Neon Plaza, already has shared presence infrastructure.

You can see it here:

https://www.myzubster.com/metaverse

Today it uses MongoDB-backed shared presence and a serverless-compatible polling model.

Presence records expire after 90 seconds.

We're now connecting this operational state to ZORGAX through a bounded telemetry adapter.

The new endpoint under development is:

GET /api/zorgax/party-telemetry
Enter fullscreen mode Exit fullscreen mode

Instead of exposing individual users or raw session information, it produces operational information useful to Party Mode.

For example:

Room
 ├── health
 ├── transport
 ├── aggregate presence
 ├── storage state
 ├── reconnect information
 └── retention metadata
Enter fullscreen mode Exit fullscreen mode

This allows ZORGAX to distinguish situations such as:

ROOM HEALTHY
     ↓
shared presence available
     ↓
Party Mode can report live state
Enter fullscreen mode Exit fullscreen mode

from:

ROOM DEGRADED
     ↓
shared storage unavailable
     ↓
Party Mode reports degraded state
Enter fullscreen mode Exit fullscreen mode

That's very different from simply saying:

"The party is offline."

Infrastructure degradation and an intentionally ended session are two different things.

ZORGAX needs to understand that distinction.

We Are Not Pretending WebXR Is Finished

This is another important part of how we're approaching the project.

Our roadmap includes:

  • WebXR rooms
  • 3D environments
  • Spatial audio
  • Avatars
  • Live media
  • Stage state
  • Interactive portals
  • Virtual Twins

But the production room/session lifecycle required for all of those capabilities is not finished yet.

So our telemetry adapter does not invent those states.

Where a production model doesn't exist yet, Party Mode explicitly reports:

not-modeled
Enter fullscreen mode Exit fullscreen mode

For example, stage/media/portal telemetry stays unavailable until the corresponding WebXR data layer exists.

This might seem like a small implementation detail.

I think it's actually an important architectural principle for AI-integrated products:

Your AI layer should know the difference between false, offline, unavailable and not implemented.

Those are four very different states.

Why Telemetry Matters for an AI Agent

Eventually, we want ZORGAX to answer questions like:

Is the room live?

How many participants are currently present?

Is the realtime service healthy?

Is the stage available?

Which portals are active?

Has this session ended?

Is there an archive available?
Enter fullscreen mode Exit fullscreen mode

But those answers cannot come from an LLM guessing based on conversation history.

They need to come from authoritative system state.

The model can explain the state.

The infrastructure must provide the truth.

So we're separating:

SYSTEM STATE
      ↓
PartyContext / Telemetry
      ↓
ZORGAX reasoning
      ↓
Human-readable response
Enter fullscreen mode Exit fullscreen mode

rather than:

User question
      ↓
LLM guesses what's happening
Enter fullscreen mode Exit fullscreen mode

Privacy Is Part of the Telemetry Design

Telemetry can easily become surveillance if you don't define boundaries early.

For Party Mode, we're deliberately focusing on operational state rather than behavioral tracking.

The current design does not require ZORGAX to receive permanent movement history.

It doesn't need raw authentication tokens.

It doesn't need chat content in infrastructure logs.

It doesn't need to expose raw session identifiers.

And aggregate presence is usually enough for questions such as:

"How active is the room?"

rather than exposing unnecessary information about individual participants.

The objective is:

Give ZORGAX enough information to understand the experience without giving it everything the infrastructure knows.

What Comes Next: Z4 Creator Commands

Telemetry makes ZORGAX aware of the environment.

The next step will make it capable of performing bounded actions.

That's Z4.

We're designing a capability gateway where creator/host commands are explicitly allowlisted.

Conceptually:

Creator
   ↓
ZORGAX
   ↓
Capability Gateway
   ↓
Server Authorization
   ↓
Approved Action
Enter fullscreen mode Exit fullscreen mode

Possible actions could eventually include:

publish approved notice
surface approved media
open configured portal
close configured portal
request session status
trigger approved room action
Enter fullscreen mode Exit fullscreen mode

But the key is that ZORGAX does not receive unrestricted tool execution.

A command must pass server-side authorization.

Consequential actions can require confirmation.

Privileged actions must be auditable.

And certain capabilities remain outside Party Mode entirely without dedicated authorization.

Then Comes Moderation

Z5 will connect ZORGAX to trust and safety workflows.

The goal isn't:

AI automatically bans people.

Instead, ZORGAX can help surface signals and organize workflows.

For example:

spam signal
     ↓
ZORGAX
     ↓
report / moderation queue
     ↓
moderator
     ↓
review
Enter fullscreen mode Exit fullscreen mode

Block and mute state should also affect what Party Mode surfaces.

Material enforcement decisions retain a human-review path.

What Happens After the Party?

This is one of the parts I'm most interested in.

A real or virtual experience doesn't necessarily have to disappear when the live session ends.

Z6 introduces the post-event transition:

LIVE
 ↓
ENDED
 ↓
ARCHIVE
 ↓
REPLAY / HIGHLIGHTS
 ↓
COMMUNITY
Enter fullscreen mode Exit fullscreen mode

ZORGAX should understand that transition too.

Live-only commands must expire.

Approved media can become archive content.

The community can remain connected to the historical event.

That gives us the larger Social Life model we're working toward:

Community
   ↓
Public Event
   ↓
Physical Experience + Virtual Twin
   ↓
ZORGAX Party Mode
   ↓
Live Social Experience
   ↓
Archive / Replay
   ↓
Community
Enter fullscreen mode Exit fullscreen mode

The party ends.

The community doesn't have to.

The Final Gate: Z7

Before calling Party Mode complete, we're planning an end-to-end safety and acceptance phase.

That includes testing scenarios such as:

  • Identity spoofing
  • Unauthorized creator commands
  • Restricted/private data requests
  • Expired capability replay
  • Prompt/context injection
  • Audit completeness
  • Retention boundaries
  • Logging boundaries
  • Safe degradation when infrastructure fails

The complete test path should eventually look like:

Public Event
    ↓
PartyContext
    ↓
Assistant
    ↓
Room Telemetry
    ↓
Authorized Creator Action
    ↓
Moderation / Reporting
    ↓
Event Ends
    ↓
Archive / Replay
Enter fullscreen mode Exit fullscreen mode

Only then does Party Mode become a complete system rather than a collection of AI demos.

Where We Are Today

So the current status is:

Z1 PartyContext             ✅ merged
Z2 Community Assistant      ✅ merged
Z3 Telemetry Adapter        🚧 in development
Z4 Creator Commands         ⏳ next
Z5 Moderation               ⏳
Z6 Archive / Replay         ⏳
Z7 Safety + E2E             ⏳
Enter fullscreen mode Exit fullscreen mode

The Z3 implementation is currently being developed against the real Neon Plaza infrastructure available today.

We are deliberately not marking future WebXR functionality as complete before the underlying room/session models exist.

The Bigger Idea

The long-term goal isn't simply to put an AI chatbot inside a metaverse.

We're exploring a different model:

AI as a bounded orchestration layer between community, software and shared digital environments.

Eventually the wider MyZubster ecosystem could connect:

People
+
Communities
+
AI Agents
+
Metaverse
+
Creators
+
Robotics
+
Real-world data
Enter fullscreen mode Exit fullscreen mode

But each connection needs an authority boundary.

Who knows what?

Who can do what?

Who authorized the action?

What gets logged?

How long does the data live?

What happens when a service fails?

Those questions are less exciting than saying "AI Metaverse."

They're also the questions that determine whether the system can actually work.

We're building those foundations now.

MyZubster Metaverse / Neon Plaza

https://www.myzubster.com/metaverse

MyZubster Space Station

https://www.myzubster.com/space-station

Open-source repository

https://github.com/MyZubster-Ecosystem/myzubster

If you're working on AI agents, WebXR, realtime systems, authorization or virtual-world infrastructure, I'd be interested in your perspective:

How much operational context should an AI agent receive before context itself becomes a security boundary?

opensource #ai #webdev #metaverse

Top comments (0)