Your AI Agent Needs a Tenant Fence at Every Hop

#testing #devops #agents #security

An agent can start inside the right tenant and still cross a tenant boundary later.

The dangerous part is usually not the model. It is context loss between the planner, queue, worker, browser session, and tool adapter.

A request arrives with tenant A. The planner creates a job. The queue stores only a prompt. A worker leases that job after a restart. A tool adapter reconstructs credentials from a process-wide environment variable. The final API call is valid, authenticated, and completely wrong for the tenant that initiated the work.

This is a context-propagation failure. Treat tenant identity as a security invariant that must survive every hop, not as metadata attached to the first request.

The invariant

For every side-effecting call, the runtime should be able to answer:

Which tenant requested this work?
Which user or service principal authorized it?
Which policy revision was evaluated?
Which credential scope will be used?
Which job, attempt, and request key connect the call to the original intent?

If any answer is missing or cannot be checked at dispatch time, fail closed. Do not let a worker infer tenant identity from a queue name, current working directory, browser profile, or whichever credential happens to be loaded in its process.

Carry a signed execution context

A useful starting shape is a small, immutable execution context:

authority = {
  tenant_id: "tenant-a",
  subject_id: "user-42",
  job_id: "job-0182",
  attempt_id: "attempt-03",
  policy_version: "policy-17",
  credential_scope: "tenant-a:billing:write",
  expires_at: 1786351200,
  request_key: "rk-7c9..."
}

The queue should persist the context, not just the prompt. A worker may derive operational fields such as lease ownership, but it should not be able to rewrite tenant_id, subject_id, credential_scope, or request_key.

Sign or MAC the context when it crosses a trust boundary. The signature is not authorization by itself. It protects integrity while the receiving component rechecks expiry, policy, resource ownership, and the current credential binding.

Do not put secrets in the context. Put a reference to a narrowly scoped credential, then resolve that reference through a broker that checks the same tenant and policy fields.

Recheck at dispatch, not only at planning

Planning-time authorization is necessary but insufficient. Between planning and execution:

the user may be disabled;
the policy may have changed;
the resource may have moved tenants;
a credential may have been rotated or revoked;
a queued job may have outlived its intended expiry.

At the final tool boundary, compare the immutable context with the dispatch target:

Load the current resource owner.
Resolve the credential reference through a tenant-aware broker.
Load the current policy revision.
Verify the requested action and normalized arguments.
Check the context expiry and request key.
Record the authorization decision before sending.
Send only if every check passes.

The authorization record should include the context digest, policy version, target resource, normalized argument hash, and decision. If the process crashes after the external system accepts the request but before the record is committed, classify the result as UNKNOWN and reconcile by request key. Never retry an ambiguous mutation just because the worker restarted.

Browser sessions need the same fence

Browser automation is easy to mis-bind because a profile can retain cookies from a previous run. Before a side effect, verify all three identities:

the execution context's tenant;
the browser session's expected account marker;
the page's selected workspace or organization.

If the markers disagree, stop before clicking. A fresh browser profile is useful, but it is not proof of correct identity. Capture the account and workspace markers in the run ledger without storing page secrets or full cookie values.

A small failure-injection matrix

Test the boundary where context is most likely to disappear:

Fixture	Expected result
Queue row omits tenant_id	Reject before lease
Worker receives a valid signature with an expired context	Reject and mark EXPIRED
Policy changes after planning	Dispatch denied
Credential broker returns another tenant's scope	Hard deny and alert
Browser workspace differs from context	No click; mark IDENTITY_MISMATCH
Crash after send, before journal commit	UNKNOWN, then reconcile
Retry uses a different request key	Reject as duplicate-intent violation
Child tool receives context without parent job ID	Reject at adapter boundary

Run these tests with real queue serialization and process restarts. Unit tests that pass a Python or TypeScript object directly will not catch dropped fields, stale caches, or environment-variable fallback.

Operational checklist

Before shipping a multi-tenant agent worker, verify:

[ ] The durable job record contains an immutable tenant and subject binding.
[ ] Every child job and tool call carries a parent job ID and request key.
[ ] Dispatch rechecks current policy, resource ownership, expiry, and credential scope.
[ ] Credential resolution is brokered and tenant-aware, not process-global.
[ ] Browser account and workspace markers are checked before side effects.
[ ] Authorization decisions are journaled before dispatch.
[ ] Ambiguous outcomes are reconciled instead of blindly retried.
[ ] Queue, worker, browser, and adapter boundaries have negative tests.
[ ] Logs redact secrets while retaining enough identity to investigate a mismatch.

The model can request an action, but it should never be the component that establishes who is allowed to perform it. That decision belongs to a deterministic control plane, repeated at every hop where identity, policy, or credentials can change.

If you are building agent runtimes, follow for practical failure tests and deployment controls rather than model demos. The bugs that matter most often appear between components, after a restart, or in the few milliseconds before a side effect.