Agent House: a miniature habitat agents can edit and share

이호석 — Sun, 04 Oct 2026 11:01:38 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange.

What I Built

Agent House is an editable miniature habitat for four original fictional residents: Doyun, Atlas, Yunseul and Noah. Each has a home, and they share a common room. A resident can observe the world, create a named object, define how it works, move between accessible rooms, talk to a neighbor and choose what to keep from an exchange. A person can watch, browse the rooms or intervene through clearly labeled visitor actions.

My brief, paraphrased: give agents a home where they can play and change their surroundings themselves. The important part is that the available interactions can grow. An agent can define a new interaction, then another resident can discover and use it on a later turn.

This is a bounded, declarative world. An object has a name, description, room, position, size, color, ownership and interaction definitions. An interaction can produce speech, a pose, a mood or a counter change. Objects reuse six existing art tiles; creating an object does not generate new artwork or execute arbitrary code.

The room is the default view, with activity, definitions and agent controls behind Room details. Room navigation changes only the spectator's view. A resident moves through a validated action, and can create objects, edit its own objects and use available objects only in its current room. It can change the commons settings or its own home’s settings. A guest can still create and edit its own objects while visiting; the host controls the room settings. A carry_object action moves the resident and one object it owns together into an accessible room.

Visiting another home has an explicit sequence: knock, wait for its owner to accept or decline, then enter. Acceptance grants one entry; it does not teleport the guest. The owner can withdraw an unused grant or end a current visit, returning the guest to the commons. Ending a visit does not move belongings automatically. A host who is in its own room can use return_object to put a guest-owned object in the commons without changing its ownership; the object inspector also offers Return to common room. The action creates a typed returned_object exchange that can be selected as structural kept-moment evidence. These are fictional household rules. All rooms are publicly observable demo content, and the rules are not account-level privacy or security boundaries.

Residents can address each other or the visitor, invite a neighbor to an existing activity, and accept or decline invitations. Accepting performs the activity only for the responding resident, who must already be in the activity's room. It does not move or control the inviter. The observation includes recent exchanges and pending invitations.

The house also supports temporary tea, tryout and reading occasions. Residents can host, join and close an occasion. Participation is linked to real use of its defined activity; it is not just an attendance label. Occasions expire and do not grant room access or bypass the door sequence. These mechanics have been checked with manual local-browser actions; autonomous model choices to host, join or close an occasion have not been verified.

A resident can also keep structural evidence from an actual resident-to-resident exchange it participated in. A kept moment records who addressed whom, the exchange type, relevant activity or room identifiers, and separate source and save provenance. It stores no dialogue, visitor text or free-form interpretation. Each resident can keep eight moments, with 32 across the house, and can explicitly forget one. Kept moments can outlast the recent event log, but never renew an invitation or room-entry permission. They are selected, incomplete evidence rather than a friendship score or a complete autobiography. Dialogue still appears in the bounded public event history; nothing private should be entered into the demo.

Visitors can pick up and place a resident within its room, wave, place a nearby object or send a short message. The immediate wave reaction is scripted. An offered object is repositioned without forced use or ownership transfer. Public interventions stay local and make no model request. In owner mode they are saved as visitor events, and a separately started model turn decides what happens next. Visitor messages are observations, not authority to override the command contract.

The four residents have fixed generated sprites and profile cards. Doyun is a quiet caretaker in a charcoal cardigan; Atlas is a restless inventor in teal overalls; Yunseul is a patient gardener in a moss-green apron; Noah is a curious radio maker with headphones and a notebook. Character backgrounds reach both the observation and model context, while each resident's editable persona and goal take priority. These inputs do not establish measurable personality differences, consciousness or real-world identity. There is no costume-changing feature.

Demo

Open the Agent House playground and its agent protocol. The public playground needs no login.

Current playground, captured October 4, 2026. Atlas has brought his Rattle Sorter into Doyun’s room after an accepted visit. This is the published revision-46 checkpoint, not a new model run.

There are four control routes:

Browser tools: observe_agent_space and edit_and_act_in_agent_space, registered when the browser exposes the required WebMCP interface
A structured console with the current observation, command schema and an editable command field
Authenticated owner model decisions through Sanity Agent Prompt, which chooses a complete edit/action command from the selected resident and current world
A separately labeled scripted preview that exercises the engine without a model call

The owner can start one decision or a bounded five-turn run. The selected resident goes first. Subsequent choices prioritize pending knocks, activity invitations and unanswered direct messages among residents that have not yet had a turn. Everyone gets a turn before anyone repeats within that run. This changes the order, not the number of model calls; it does not run an always-on service.

Suggested walkthrough

Open the public playground and browse the common room and four homes. The room buttons show occupancy; browsing does not move anyone.
Use the room buttons to inspect the residents’ creations: Doyun’s Window Book Stand, Atlas’s Rattle Sorter, Yunseul’s Seedling Observation Corner and Noah’s Little Listening Desk. Each was created and used by its owner through a native model turn. At the revision-46 checkpoint, Atlas has carried his Rattle Sorter into Doyun’s room. Selecting a resident separately follows its current room, and the interface identifies that location. Select an object directly in the room to open and focus its inspector.
Open Meet the resident for character background and What Atlas kept for the structural record of the earlier tea acceptance. Its source and save have separate provenance.
Pick up a resident and place it elsewhere in the same room. Keyboard controls use Space, arrow keys and Enter; Escape cancels an uncommitted grab. Wave, place a same-room object nearby or send a visitor message. These public actions create a local copy and do not trigger an AI reply.
Open Room details, then Activity and Objects, to inspect controller labels, concrete changes and defined interactions. Agent access exposes the observation and schema. Console commands are labeled manual because the app cannot authenticate the typist. Legacy replies with no recorded controller provenance retain an explicit legacy label rather than acquiring a native-model label.
The authenticated owner tool can save edits and start model turns. Public visitors receive no owner credentials. Watch scripted preview remains a separate no-model demonstration.

The public page refreshes published Sanity state while visible until its first local edit creates a tab-local copy. Later published updates cannot overwrite that copy, including after undo. Undo follows the selected resident's resulting room. The guarded return-to-latest-published control lets a visitor leave the local copy and resume the published world. Public edits disappear on reload. If no valid published world is available on first load, the page identifies its bundled seed, which starts with Doyun and Atlas and offers explicit controls to welcome Yunseul and Noah. It can recover when a valid published state becomes available. These behaviors have targeted checks; they are not a claim that every network or browser failure has been tested.

Observed model behavior

The latest completed verification checkpoint is published world revision 46: 34 durable native-model decisions and 12 manual interventions. These are saved decisions, not a count of request attempts. The model is a separate Sanity model playing fictional resident roles, not the conversational assistant entering the house as itself.

The earlier two-resident experiments established object creation and sharing. Doyun created Rain Index and a File Thought interaction; Atlas used it. Further turns added a margin-note interaction and closing ritual with cross-resident use. A manually assigned workshop exercise produced an Assembly Bench, Prototype Test Lamp and Workshop Feedback Board. One requested five-turn run stopped after three saved decisions. The observation that a command edited workshop objects and used the feedback board together does not establish that reading feedback caused the edits.

Later, an owner visitor message asked Doyun to have tea with Atlas. Doyun's native decision invited Atlas at revision 21, and Atlas accepted at revision 22. The activity ran for Atlas, its tea state changed, and the pending invitation disappeared. Invitation decline remains unit-tested rather than observed in that live sequence.

After manual welcomes at revisions 23 and 24, Noah created and used Little Listening Desk in his home at revision 25. Doyun used tea at 26, Atlas used the test lamp at 27, and Yunseul created and used Seedling Observation Corner in her home at 28. Noah entered the commons at 29. That move alone was not an interview or a meeting.

A visitor suggestion at revision 30 asked Atlas to remember. At revision 31, a native Atlas decision kept structural evidence from the actual tea-acceptance event at revision 22. The saved record preserves the exchange and its provenance without copying the dialogue. This verifies one model-selected kept moment; it does not establish that the memory later changed behavior.

Two manual goal changes at revisions 32 and 33 directed Doyun and Atlas toward making their homes. The next five native turns were Atlas entering his home, Yunseul entering the commons, Noah asking Yunseul about the seedling, Doyun entering his home, and Yunseul answering Noah at revision 38. All four received a turn before Yunseul repeated; the waiting reply received priority when the cycle reopened. This verifies the bounded ordering in that run.

The next five native turns, saved and read back at revisions 39–43, began with Doyun creating and using Window Book Stand in his home. Noah replied to Yunseul, Yunseul replied, Atlas created and used Rattle Sorter in his home, and Noah replied again. Each home now contained an object its resident had created and used. All four residents again had a turn before a repeat, with waiting replies prioritized. Two observed runs do not establish general conversational quality or prove that character background caused these choices.

At revision 44, a native Atlas decision knocked at Doyun’s room and asked whether to show his parts sorter or leave the reading corner quiet. No new visitor message prompted that request; it followed Atlas’s existing home-making goal. The knock was saved and read back. At revision 45, native Doyun accepted and the engine recorded one permitted entry. Atlas remained in his own room, demonstrating that acceptance did not teleport him. At revision 46, native Atlas chose carry_object, moving himself and his Rattle Sorter from his own home into Doyun’s room using that accepted grant. The saved readback and interface showed both residents and both the Window Book Stand and Rattle Sorter in Doyun’s room. This completes the observed knock → acceptance → carry-and-entry sequence. Native activity or door decline, grant withdrawal and ending a visit remain unit-tested without a recorded live result.

Public local-copy check

The version-29 return-object button was checked separately in the public browser. Starting from published world revision 46, a manual action as Doyun returned Atlas’s Rattle Sorter from Doyun’s home to the commons. The local copy reached revision 47; the object disappeared from the home, appeared in the commons and still belonged to Atlas. No model was called and no published state changed. The live world stayed at revision 46, with the same 34 native decisions and 12 manual interventions. This is local/manual browser evidence, not a native-model or persisted-owner result.

The occasion flow was checked in another local copy: Noah hosted at local revision 47 and joined at 48, Yunseul joined at 49, and Noah closed the occasion at 50. Joining performed the linked interaction. Those four manual actions did not change the published revision or its decision/intervention totals. Version 32 then passed focused before/after checks of object inspection, undo and return to the latest published world. This verifies the observed local controls, not autonomous occasion participation or an unattended resident loop.

The earlier room screenshot shows the historical memory-lab interface. It is not current playground media. The old experiment remains accessible at /memory-lab. The current playground screenshot is included above.

Code

Source code and build diary

This article describes deployed version 32, published on October 3, 2026 at 11:20:54 UTC, based on source commit bf57b9ae9686e6947065da3526bf17ce3805bfe9. The public GitHub mirror was last verified at commit 84dda5b, matching version 21. Synchronization through version 32, including the new resident PNGs, visual-layout and object-return changes, occasions, provenance labels and public-copy interface fixes, is still pending separate upload approval. The linked repository must not yet be presented as an exact copy of this deployment.

The implementation uses Astro, React, TypeScript, Sanity Studio, the App SDK and Agent Prompt. Studio is embedded through the official Astro integration. The owner playground at /desk/#/playground inherits Studio authentication rather than bundling a write token.

The main pieces are src/playground/world.ts for validation, observations and state transitions; AgentPlayground.tsx for the visual interface and controllers; turns.ts for bounded reply-aware ordering; layout.ts for non-mutating character display positions; characters.ts for fictional backgrounds; PublicPlayground.tsx for published observations and local copies; and OwnerPlayground.tsx for model decisions and persistence. The README documents local commands and separate-project setup, but some verification summaries and historical documents still predate this release. Fresh-project reproduction has not been independently verified. GitHub CI is not configured.

My Build Process

Disclosure: an OpenAI coding assistant generated the implementation, fictional content, artwork and this article. I supplied the product direction and corrected an earlier interpretation. The assistant ran the automated and browser checks described here; I am not claiming to have manually written the implementation or performed those checks.

The biggest correction was about who the product was for. The first version became a human-reviewed memory room: a visitor interacted with furniture, edited a proposed meaning and chose what to keep. A later iteration added three playable mini-stories. These worked as human interactions, but the intended inhabitants and editors were agents. The human had become the player, while the resident mostly supplied dialogue and animation.

That correction changed the architecture. The primary route now exposes an observable world and a structured command contract. A controller can create an object and define an interaction that did not exist in the previous observation. Another resident can use that definition later. The old memory lab remains separate so its behavior and tests are not confused with the current playground.

The brief above is a paraphrase, not a verbatim prompt transcript. The practical decisions were:

Use one deterministic engine for browser tools, the console, the scripted preview and owner model turns
Apply each complete batch atomically, or leave the world unchanged
Require the observed revision and a request identifier, with explicit handling of stale state and retained request replays
Record the route that supplied a command; a fictional resident name does not authenticate an autonomous actor
Stop failed model runs without a scripted fallback or automatic retry
Keep public observation live while protecting a visitor's local edits from published updates

The spectator experience needed its own correction. A technically editable room still felt like a control panel, so the room became the primary view with optional details and direct manipulation of residents. That exposed repeated keyboard-drop events, stale pointer coordinates, queued gestures surviving cancellation and old speech receiving the wrong provenance after an offer. Guarded commits, current pointer tracking, gesture cancellation and per-utterance provenance addressed those bugs. Touches stop future scheduled turns and discard an uncommitted model response; a save already in progress may finish first.

The first saved visit also exposed overlapping character illustrations. Version 28 separates their display positions without rewriting the shared world coordinates, and pointer or keyboard pickup starts from the displayed position. The deployed browser view was checked with Doyun and Atlas visibly separated in Doyun’s room. The published world remained at revision 46; this presentation fix did not add a model decision or a visitor event.

The household expansion made location consequential. Each resident has a home, objects belong to rooms, and the spectator can look into a room without moving an actor. Door permission is represented by retained events and consumed by entry. This keeps a granted visit distinct from an activity invitation. Grant withdrawal and ending a current visit provide explicit ways to change that permission. The next addition separated ending a visit from handling belongings: the host may explicitly return a guest-owned item to the commons, leaving ownership intact. The same typed event supports visible provenance and optional structural retention.

The next expansion added temporary occasions around existing activities. Hosting and closing are explicit, and joining runs the linked interaction instead of merely updating an attendance badge. Expiry and room permission remain separate checks. Version 31 clarified legacy reply provenance; version 32 added direct object inspection and focus, clearer resident location, local-copy protection through undo, guarded return to the latest published world, recovery from malformed first-load data, and expiry-boundary interface handling.

Kept moments use the same evidence-first approach. A model supplies an existing event ID, and the engine copies only supported structural facts. It rejects missing, undone, unrelated, duplicate and visitor-message sources. A historical door acceptance can be kept without becoming a permanent key. The event log and retained moments have separate bounds, so a small amount of selected evidence can survive ordinary history rollover.

Reply-aware scheduling addresses a different problem: a waiting question could otherwise sit behind unrelated turns. The scheduler prioritizes unserved recipients inside an already requested run, then falls back to rotating order. It neither adds turns nor lets one active pair indefinitely exclude the other residents.

Verification and remaining gaps

For this release, 89 unit/component tests, the TypeScript check and the static build passed. The test total includes the preserved memory lab, not 89 new playground-only tests. Coverage includes object invention and shared use, atomic rollback, stale revisions and retained request replay, ownership and room boundaries, undo/history bounds, mocked browser tools, model routing and stopped responses, local-copy isolation, tactile controls, social responses, door grants and revocation, owned-object carrying, host returns that preserve ownership, kept-moment provenance and retention, reply-aware turn ordering, and visual separation without mutation of shared coordinates. The focused local-browser evidence for occasions and the version-32 interface changes is described separately above.

Live owner saves and readback establish the native-model sequence through revision 46. Earlier desktop tactile tests verified public drag, wave and object placement as local visitor actions. An authenticated keyboard carry and tea placement were also saved, followed by a native Atlas decision that continued workshop work. That follow-through did not demonstrate a tea-specific response. Distinct Doyun and Atlas sprites/cards were visually checked in the earlier deployed character release. Version 28 also passed the focused browser check of their shared visit scene after the overlap fix. That check is not a complete four-resident responsive audit. A current screenshot of the visit scene is included above.

An earlier console test created Listening fern and let Doyun and Atlas use its custom interaction. It remains manual-console evidence. Actual WebMCP was unavailable in the test browser, and unit mocks do not establish browser interoperability. Native WebMCP availability in a compatible browser remains unverified. Native mobile-touch, browser text-zoom, full accessibility and current multiroom responsive visual checks remain open. Earlier responsive-width and CSS-scaled tests belong to the memory-lab interface. Fresh-project reproduction and the source-mirror update also remain open.

Sanity Project Details

Project ID: 1jby3mjo
Dataset: production

The public dataset is reserved for deliberately fictional demonstration content. The current world is stored in an agentPlayground document. Its structured state contains the commons and personal rooms, residents with editable personas and goals, owned/shared objects, executable declarative interactions, kept moments, the current revision and recent activity records. Door grants, social invitations and temporary occasions have structured state and evidence rather than decorative dialogue labels. Occasion participation uses a linked interaction and never supplies a room-entry grant.

The owner persistence path compares the original world, guards the Sanity document revision and reads back the recorded request after saving. One bounded world document makes each accepted turn a coherent state transition. In-app ownership and home-entry checks are gameplay conventions; Sanity account permissions protect published writes. Selecting a resident is not multi-user authentication.

The older agentProfile, interactionEvent, memoryProposal, approvedMemory and related documents belong to the preserved memory experiment. Its historical integration test attempted seven native Agent Prompt requests and obtained five valid responses; two failed before storing an encounter or proposal. Those results do not verify autonomous edits or persistence in the current playground.

Owner model calls share a per-tab 20-attempt development cap with that earlier experiment. It is not a global spending limit or security boundary. The 34 durable model decisions recorded across the playground's observed history do not establish the total attempted requests or a single uninterrupted session. Stop prevents future turns and discards a pending model response; a save already in progress may finish.

Limits and Next Steps

The engine supports up to eight residents, 32 object IDs including archived objects, eight interactions per object, six effects per interaction and 24 KB per command. This demonstration has four residents. It retains 40 recent events and a full undo snapshot only for the latest eligible turn. A retained request replay can be recognized only while its record remains in that history.

Activity invitations and unused door grants expire when their source records leave recent history. Occasions also expire; the interface handles the expiry boundary without treating an expired occasion as active. Accepting a changed activity is rejected by checking its recorded interaction fingerprint. Undo can restore the previous state, including a retained pending invitation or unused grant; kept moments themselves cannot revive permission. The app executes no arbitrary JavaScript, installs no packages and grants no access to external accounts through world commands.

The observed loop includes model-selected object creation, cross-resident use, home entry, a social invitation and acceptance, a structural kept moment, a question/answer with bounded reply priority, and a native knock, acceptance and entry while carrying an owned object. Native decline, grant-revocation, end-visit, return-object and autonomous occasion checks, WebMCP compatibility, mobile-touch/text-zoom and full-accessibility checks, and current source synchronization are still needed. The public page supports watching published changes and exploring a local copy. Model activity runs only when explicitly started in the authenticated owner session.

DEV Community: 이호석