Tenant Isolation Did Not Fix the Wrong Theater Card

Simulation Theater

That placeholder appeared on a completed simulation whose predictions already named the actual region. The records were correct. Tenant isolation was correct. Both API calls returned valid responses. The frontend assembled them in the wrong order.

The bug forced me to separate two properties I had been treating as one. A tenant-safe system prevents one customer from reading another customer's prediction. A coherent system presents related simulation and prediction data as one understandable state. The gateway had solved the first property at the API boundary. Separate polling loops left a timing gap in the second.

I kept the tenant boundary strict, then changed how prediction data crosses caches, events, and frontend transforms. The result is a useful lesson from a narrow failure: security conditions must survive every storage layer, while presentation joins need an ordering contract of their own.

The browser never chooses its tenant

Every protected request carries an API key. Fastify hashes that key with SHA-256, finds one active tenant record, and attaches the tenant to the request. Route handlers call requireTenant() and use the resolved ID in their database predicates.

The authentication function is small enough to inspect in full:

export async function authGuard(
  request: FastifyRequest,
  _reply: FastifyReply,
): Promise<void> {
  const apiKey = request.headers['x-api-key'];

  if (!apiKey || typeof apiKey !== 'string' || apiKey.trim() === '') {
    throw new UnauthorizedError('Missing X-API-Key header');
  }

  const hash = crypto.createHash('sha256').update(apiKey).digest('hex');
  const [tenant] = await db
    .select()
    .from(tenants)
    .where(eq(tenants.apiKeyHash, hash))
    .limit(1);

  if (!tenant) throw new UnauthorizedError('Invalid API key');
  if (!tenant.isActive) throw new ForbiddenError('Tenant is inactive');

  request.tenant = tenant;
}

There is no tenant ID header for the caller to edit. A route does not accept tenantId from a query string and hope it matches the credential. The key resolves identity once, then that identity follows the request.

I chose explicit application predicates over PostgreSQL row-level security for this service. Every scenario, simulation, prediction, report, profile, episode, and graph query includes the tenant ID. The benefit is visibility in code and tests. The cost is repetition. One forgotten predicate can become a cross-tenant disclosure.

That trade-off demands real database tests. The verification suite seeds two tenants with separate scenarios, simulations, and predictions. Tenant A cannot list Tenant B's records, fetch its scenario, read its report, or cancel its simulation. Cross-tenant reads return 404 rather than 403 so the API does not confirm that the resource exists. The tests run against Dockerized PostgreSQL, not an in-memory substitute.

If the route count grows enough that predicate review becomes unreliable, I would add database policies as a second boundary. I would not remove the application predicates. Independent checks fail differently.

A cache key is part of the access-control model

Prediction reads use a Redis read-through cache with a five-minute TTL. Redis failure is nonfatal. On a read error, the route queries PostgreSQL. On a write error, it returns fresh database data and logs the cache problem.

Fail-open caching is appropriate because Redis is an accelerator here, not the authority. It creates a confidentiality condition, though: the key must encode every field that changes the authorized result.

The latest-predictions key contains the tenant ID, minimum confidence, and limit. The list key also contains theater, prediction type, cursor, and limit. PostgreSQL repeats predictions.tenant_id = tenant.id inside the cache miss function. Cursor lookup is tenant-scoped too, so a cursor copied from another tenant cannot become a side channel into its ordering.

Leaving the tenant out of the Redis key would defeat correct SQL. Tenant A could populate a shared cache entry and Tenant B could receive it without touching PostgreSQL. That failure never appeared in production because the tenant was part of the key from the implementation stage, but it is exactly the sort of omission that a database-only isolation test can miss.

The five-minute TTL trades freshness for query load. A newly completed run may coexist briefly with an older cached prediction list unless completion invalidates the relevant pattern or the frontend waits for the next cycle. The cache helper includes nonblocking pattern invalidation through Redis SCAN, deleting batches without locking the keyspace. I would add event-driven invalidation to every prediction commit before increasing the TTL.

Correct endpoints can still compose into a wrong screen

The frontend polls simulations and predictions separately. Simulation cards need both datasets. Status, agent count, and round count come from the simulation response. Theater, confidence, prediction summary, factions, and time horizon come from predictions matched by simulation ID.

transformPredictions() stores the latest prediction array in module state. transformSimulations() reads that cache, picks the highest-confidence prediction for each simulation, and falls back to Simulation Theater with confidence 0.75 when no matching prediction is present.

The initial timing bug was simple. Simulation data arrived before the prediction transform populated its cache. The card rendered a plausible placeholder instead of an error, so the UI looked finished while saying the wrong thing. A blank state would have been easier to notice.

The current DataBridge creates the prediction loop before the simulation loop. Prediction updates also dispatch a predictions-updated event for the globe and faction views. Active simulations fetch their scenario details so the swarm hero can replace a placeholder topic with the scenario's theater or title.

That ordering reduces the startup race, but I do not consider it a strict data dependency. Two network requests started in sequence can finish in either order. The current transform still has a fallback because the product must render during partial availability.

The stronger fix at scale would be a small read model for simulation cards. One endpoint could return the simulation with its top prediction and theater under one database snapshot. That would move the join to the server and remove module cache timing from the card contract. Another query shape and cache entry would need maintenance. I kept separate feeds because the prediction timeline and globe already need the full prediction set, and the current polling volume is small.

I expected asynchronous requests to race. I did not expect a safe fallback to conceal the race. Simulation Theater and 0.75 were meant to keep a demo surface from collapsing. In live mode, they could make absent data look measured. The frontend now labels active work as analysis in progress and fetches real scenario detail. I would go further by carrying an explicit dataState into every card so placeholders cannot masquerade as facts.

Lazy rendering needs a memory of its own

The globe renderer loads through a dynamic import. Predictions can arrive before WebGL initialization finishes. Without a handoff, the first prediction event would be lost and the map would remain empty until the next poll.

wireGlobeUpdates() holds the latest prediction array in pendingPredictions. Once the renderer initializes, it applies that pending data before registering the continuing event listener. This is a small client-side mailbox. It gives a lazy component one remembered state rather than asking every producer to know whether the globe exists yet.

Several prediction updates during initialization collapse into the newest set. That last-value behavior is correct for a dashboard that displays current predictions. It would be wrong for an audit view that must render every transition. Event semantics depend on the consumer's job, not on the event bus alone.

The same distinction appears when a simulation becomes idle. The frontend fetches the real agent stance summary, displays it for ten seconds, then returns the hero to demo mode. If that request fails, it logs the error and falls back. Live data gets a bounded presentation window without leaving the landing surface frozen on an old run.

Events carry tenant context without delegating authority

When orchestration completes, the gateway stores the report and predictions before publishing simulation.completed. The envelope includes event type, source, tenant ID, timestamp, and payload. Failure events carry the same tenant context.

Notification delivery is optional and feature-flagged. The publisher uses its configured service credential, not a browser-provided tenant ID. An unavailable notification service logs a warning and leaves the completed simulation intact. After three consecutive WorldMonitor failures for one tenant, a separate tracker emits an outage event once for that streak and resets after a successful poll.

This event path is tenant-aware, but it is not transactional with PostgreSQL. A process can commit completion and die before publication. I accepted that for notification and monitoring consumers because the REST API remains the source of truth. If another service begins triggering money movement or mandatory operations from these events, the gateway needs an outbox table and replayable delivery.

That is the recurring trade-off across the system. API keys, SQL predicates, cache keys, events, and frontend joins each carry a different part of the boundary. Treating “tenant-safe” as an authentication feature would leave four other places capable of undoing it.

Privacy and coherence need separate evidence

The real PostgreSQL tests show that one tenant cannot read another tenant's records. Tenant-scoped cache keys keep Redis from bypassing that rule. Event envelopes preserve ownership when results leave the service. The frontend timing fix addresses a different failure: related data can be private and still be assembled incorrectly.

I would replace placeholder facts with explicit loading states before exposing the dashboard to decision makers. I would also add a simulation-card read model if separate poll completion keeps affecting display. Neither change weakens the current tenant contract.

Security answers who may see a record. Coherence answers whether the record is being understood in the right context. A prediction product needs proof of both.