An agent evaluation is only as reproducible as its least-documented input. Snapshots and forks reset the environment between runs, but a snapshot does not tell a later reader which code revision, fixture version, task definition, or scoring rubric a run used. That has to live beside the run, in a baseline manifest, or the comparison is not reviewable.
This is a proposed protocol for reproducible AI agent evaluations built around snapshot and fork. Platinum Blog is Platinum's first-party publication: the product behavior below is quoted from Platinum's documentation and from OpenAI's historical seed example, not from independent testing or a measured benchmark.
Quick answer
Use a recorded environment baseline, hold task inputs and scoring rules steady, and keep evidence from repeated runs. Treat the environment reset as one control, not a promise that two runs produce the same decisions. Platinum documents a fork as cold-booting from disk without memory or processes, and OpenAI's historical seed example says determinism is not guaranteed even when the seed, request parameters, and backend fingerprint match.
What a snapshot and fork actually preserve
Platinum is a sandbox cloud for AI agents. Its snapshot documentation defines a snapshot as a full copy of a sandbox disk at one point in time, and the sandbox must be running when the snapshot is taken. A clone boots a new sandbox from a snapshot. A fork takes a snapshot and clones it in one call, and the parent keeps running untouched.
The table below separates what the documentation states from the control a reviewer should add.
| Layer | Documented boundary | Recommended control |
|---|---|---|
| Sandbox disk | A snapshot is a full disk copy at one point in time. | Record the snapshot identity and verify the intended baseline files after boot. |
| Memory and processes | A fork cold-boots from disk and does not carry memory or processes. | Use a recorded startup procedure; do not rely on a live process continuing in the child. |
/tmp |
Does not survive disk-only operations. | Keep required baseline files outside /tmp. |
| Attached volumes | Snapshots and backups record attachment metadata only, never volume contents. Snapshot, clone, and fork with an attached Local Volume are rejected. | Version fixture data separately and record the attachment policy. |
| External services | Not snapshot contents. | Pin or replay fixtures where possible; otherwise record the live response and label it uncontrolled. |
| Model configuration | OpenAI's historical seed example offers best-effort consistency, not guaranteed determinism. | Record model identifier, request settings, prompts, and available response metadata. |
| Scoring | A separate control. | Version the rubric, evaluator, and failure-handling rules before comparing variants. |
Two documented details are easy to get wrong. First, stop/start and archive are the operations that keep memory and running processes where possible; the other operations start from disk alone. Do not substitute stop/resume behavior for the documented fork boundary. Second, where volumes are attached, clone and fork require a volume_policy: omit creates no child attachments, reattach reattaches only Shared Volumes and warns for read/write mounts, and clone returns an unsupported error rather than copying volumes. Read/write Shared mounts are the same shared data, with concurrent writes to one path following last-writer-wins behavior rather than transactions.
Platinum's product page describes forking one snapshot into "hundreds of identical parallel runs, each with reproducible state." Read that as a vendor statement about the environment offering: the detailed snapshot documentation narrows what a fork preserves, and OpenAI's historical example cautions against treating any environment setup as a guarantee of identical model outputs.
The baseline manifest
Write the manifest before the first run. It should contain, at minimum:
- the code revision under test;
- the dependency lockfile or resolved versions;
- the fixture version;
- the task definition and completion conditions;
- the scoring rubric and failure-handling rules;
- the agent's permitted tools;
- the time budget, resource allocation, and retry policy;
- what counts as an infrastructure failure rather than a task failure;
- the model identifier, request settings, and prompts;
- the external-dependency mode (pinned fixture, replay, or live).
Keep credentials out of the manifest and out of retained traces. Platinum's snapshot documentation lists scrub flags (scrub_env, clear_tmp, clear_ssh_keys) that wipe secrets before a snapshot goes into a clone or fork; use them deliberately, and decide before the run what evidence you are willing to retain.
Record the cold-start procedure
A baseline is only reusable if someone else can reach the same state. Record the startup steps for required services, then verify fixture checksums after boot. Put required baseline files outside /tmp, which does not survive disk-only operations. Do not infer application-consistent database capture or preserved live connections from the phrase "full disk copy"; verify your application's preparation requirements separately.
Branch comparable runs
For each run, record the baseline identifier and snapshot identifier where available, the variant, and the CPU, memory, and disk allocation. For repeated runs intended to share a baseline, start from an explicitly identified snapshot rather than assuming that successive forks of a changing parent share one disk state. Platinum documents fork as taking a new snapshot, while clone can start from a specified snapshot.
Record the model identifier, prompts, tool versions, generation settings, and any supported seed or backend metadata. Check support for the actual model and endpoint in use: the cited OpenAI Cookbook example names historical preview models and is not evidence that today's models expose the same controls. Matching settings are recorded controls, not proof that later outputs must match.
Keep mutable external dependencies explicit. Either pin a versioned fixture or retain the observed live response with a timestamp, and label which mode the run used.
Compare and retain evidence
Retain outputs, redacted tool traces, completion status, errors, and the scoring configuration for every planned attempt. Report repeated-run variation and failed attempts alongside any aggregate result, and apply the same predeclared retry policy to each variant. If a model evaluates outputs, record that evaluator's configuration too, so the scoring layer is auditable separately from the agent under test.
FAQ
Does a snapshot guarantee identical outputs?
No. A snapshot resets the disk. Model outputs remain subject to the non-determinism OpenAI's historical seed example describes, where matching seed, parameters, and system fingerprint still leave a chance of variation.
Does fork preserve running processes?
No. Platinum documents the fork child as cold-booting from disk without memory or processes. Only stop/start and archive keep memory and running processes where possible.
Are attached volumes cloned by a fork?
No. Snapshots and backups record attachment metadata, not volume contents. Snapshot, clone, and fork with an attached Local Volume are rejected, and clone returns an unsupported error rather than copying volume data.
What has to be recorded outside the sandbox?
The baseline manifest and retained run evidence: code revision, dependencies, fixtures, task definition, rubric, budgets, model settings, and outputs. Anything written only inside a sandbox that does not survive the operation is not a record.
Is environment reset the same as model determinism?
No. They are separate questions. A baseline answers whether the starting configuration was the same; determinism answers whether outputs repeat. The protocol above controls the first and measures the second.
Sources
- Platinum snapshots documentation (https://www.platinum.dev/docs/snapshots) — snapshot, clone, and fork boundaries;
/tmpbehavior; attachment and volume-policy rules; scrub flags. - OpenAI Cookbook, reproducible outputs with the seed parameter (https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter) — historical seed example; determinism not guaranteed.
- Platinum Blog, reproducible AI agent evaluations (https://platinum-blog.dev/blog/reproducible-ai-agent-evaluations) — the full protocol and the first-party disclosure.
Sources checked September 17, 2026.
Top comments (1)
@commerceframe_015eb18e5bb, writing the baseline manifest before the first run is the key control; otherwise the manifest can become a retrospective description of whichever comparison looked useful. I’d hash the manifest together with the snapshot identity into every run record, so changing a fixture, rubric, startup procedure, or external-dependency mode creates a new baseline ID rather than silently refreshing metadata. For live dependencies, retaining the observed response is useful, but I’d also prevent aggregates from mixing replayed and uncontrolled runs. What verification failure would make you invalidate a forked child before the agent starts?