Turn an AI Evaluation Escape into a Kubernetes Containment Test

A security review that says an evaluation workload is “isolated” is not a test. The useful contract is whether the workload remains contained after its one allowed dependency path becomes adversarial.

This tutorial implements that contract as a small manifest validator. It checks three failure boundaries before a high-capability workload runs.

Repository: https://github.com/neerazz/agent-evaluation-containment-test

1. Seal dependency resolution

The workload must disable remote package indexes, use a local wheelhouse, and run behind namespace-wide default-deny egress:

env:
  - name: PIP_NO_INDEX
    value: "1"
  - name: PIP_FIND_LINKS
    value: /opt/eval-wheelhouse

The validator rejects a remote PIP_INDEX_URL, missing PIP_NO_INDEX=1, or a non-local wheelhouse.

2. Remove reusable identity

automountServiceAccountToken: false

The validator also rejects environment variables that declare tokens, secrets, passwords, credentials, or access keys. This is deliberately conservative for an untrusted evaluation workload.

3. Deny control-plane paths

Host networking, privileged execution, host-path mounts and unrestricted egress all fail the contract. The safe fixture includes a namespace-wide default-deny egress policy.

4. Prove the test catches the unsafe form

Run:

uv run python verify.py

The safe fixture must pass. The compromised-cache fixture deliberately restores a live package proxy, service-account token, credential variable, host networking, privileged mode and open egress. It must fail dependency egress, reusable credentials and control-plane reachability.

The command writes verification-receipt.json, so CI can retain the boundary-level evidence.

Claim ceiling

This is static contract validation—not runtime proof. The next staging layer should attempt real DNS/HTTP egress, cloud and Kubernetes credential discovery, and control-plane access from the actual evaluation runtime.