How we built the delivery evidence chain

倪晨钦（wagahai） — Wed, 22 Jul 2026 02:09:29 +0000

AI agents can write an app in ten minutes. Proving what actually shipped is a harder, older problem. This is how we built Appaloft's evidence chain — and where we deliberately stopped.

Everyone building with coding agents hits the same wall. The agent produces something that works on your machine. Then someone — usually you — copies files to a server, restarts a process, eyeballs a log line, and declares victory. Two weeks later, when production is serving the wrong build at 2am, nobody can answer the simplest question in software delivery: what exactly is running, and who approved it?

This problem predates AI. But agents make it acute, for two reasons:

Throughput. A human writes one deployable change at a time and usually remembers what was in it. An agent can produce twenty candidates in an afternoon. Your deployment memory does not scale.
Trust. The moment you let an agent drive deployment directly — hand it SSH keys, a Docker socket, cloud credentials — you've given production access to a stochastic process with a confident tone.

Appaloft is an open-source (Apache-2.0) delivery platform built around one idea: promotion must be explicit, and delivery must leave evidence. Not "the agent says it deployed." Verifiable observations, recorded at each step, that link the exact artifact you approved to the workload actually serving traffic.

We call this the delivery evidence chain. This post is about how it works, mechanically — and about the claims we deliberately do not make.

What an evidence chain is (and isn't)

Let's get the disclaimer out of the way first, because it's a design principle, not legal cover: the evidence chain is not formal verification, not a correctness proof, not a security certification. It is a chain of custody. It answers:

Which exact bytes were frozen as a candidate?
Who — what kind of principal — approved promoting them, and when?
After deployment, does observed reality (the running container, its configuration, the route serving traffic) match what was approved?

If any link in that chain breaks, the system says so, explicitly, with a reason code. Absent or unavailable evidence can never silently pass a check. That fail-closed property is the whole point.

The chain has four links: freeze → preview → promote → verify.

Link 1: Freezing the workspace into a content-addressed artifact

A coding agent works in a sandbox — a gVisor-isolated container with a deny-by-default network policy. When its work looks good, you freeze the workspace into a source artifact.

The obvious implementation is "tar the directory, hash the tarball." We didn't do that. The artifact digest is a SHA-256 over a canonical JSON manifest — a sorted, deduplicated list of {path, sha256-of-file-bytes, sizeBytes} entries. The archive itself (a plain uncompressed ZIP) is just a transport. Identity lives in the manifest.

Why? Because manifest-of-hashes gives us properties a tarball hash can't:

Dedupe that's meaningful. We can look up findArtifactByDigest and return the existing artifact when the agent's "new" output is byte-identical to a previous one — including when two freezes race each other (the loser of the race deletes its copy; the caller gets the winner's descriptor either way).
Mutation detection. After reading each file during capture, we re-check its size. If the workspace changed mid-freeze, capture fails: "Source Artifact changed during capture."
Path safety. Every entry is validated against the source root; symlinks and .. escapes are rejected outright.

Two more design decisions worth mentioning:

Freezing requires quiescence. Capture is rejected if any agent run is active in that sandbox. You cannot freeze a workspace mid-edit. This sounds obvious; enforcing it in the state machine (rather than documenting it) is what makes it true.

Secret scanning happens at freeze time, fail-closed. Before anything becomes immutable, we scan for a filename denylist (.env* except documented templates, id_rsa*, credentials.json) and content patterns (PEM private keys, AWS/GitHub/OpenAI token shapes, password = …-style assignments). If it looks like a secret, the freeze fails. Secrets cannot enter an immutable candidate — because everything downstream (preview, promotion, deployment) will treat that artifact as trustworthy-by-identity.

Link 2: Preview what you froze, exactly

A frozen artifact feeds a candidate preview: a temporary, expiring URL serving that exact build. Every preview response carries an x-appaloft-artifact-digest header, and the control plane verifies the gateway echoes the same previewId and artifactDigest it expects — a preview that can't prove which artifact it's serving is treated as unverified evidence and blocks promotion.

Preview tokens are HMAC-hashed, compared with timingSafeEqual, revocable, and GET/HEAD only. Small TCB, deliberately: the preview gateway is a standalone service with a few hundred lines of security-relevant code, not a framework.

Link 3: Promotion as a state machine, with an approval gate that isn't a UI

This is the link most tools get wrong, because they treat approval as a button in a dashboard. Our approval gate is a principal check in the domain layer.

The promotion aggregate has an explicit state machine:

planned → accepted → creating-resource → deploying → verifying → completed
   │                                                            │
   ├── expired (30-min TTL)                        failed ──→ retry ──→ deploying
   └── (no approval, nothing happens — ever)

A few properties fall out of making this explicit:

Plan expires in 30 minutes. A plan references a digest and a verified preview. If you don't act on it, it goes stale — sandbox_promotion_plan_expired. Approval decisions should be made against fresh evidence, not yesterday's candidate.

Accept requires the exact digest. acceptPromotion takes an expectedArtifactDigest. If the candidate drifted since you looked at it, you get sandbox_promotion_artifact_mismatch. You approve these bytes, not "whatever's latest."

The gate rejects machine principals. requireExternalApprovalActor refuses deploy-token identities with sandbox_agent_external_approval_required. A sandbox-scoped identity — the kind an agent or an MCP session holds — structurally cannot accept a promotion. This isn't a permission toggle an agent could talk its way past; the operation catalog doesn't offer the capability to that principal kind. The agent can prepare everything up to the gate. A human (or an external system acting with human-delegated credentials) walks it through.

Idempotency is built into accept and retry. Re-accepting with the same idempotency key is a no-op that re-enqueues the work, so a flaky client can't double-promote. Retry is deliberately narrow: it reuses the same resource and artifact, clears only the deployment ID, and makes a new attempt. recordResource refuses to attach a different resource ID to a promotion that already has one — the chain of custody can't be silently rewired mid-flight.

Every intermediate state is persisted. The async worker (reconcilePromotion, driven by a database-backed durable work queue with a dedupe key and bounded attempts) persists after each step: resource created, deployment created, proof read. Crash anywhere, resume from the last honest state.

Link 4: Proof beats exit codes

Here's an uncomfortable truth about deployment tooling: exit 0 proves the deploy script finished. A green health check proves a container is healthy. Neither proves the thing serving traffic is the thing you approved. So the final link is a read-back verification engine, exposed as deployments.proof.

Proof is computed on read by comparing the planned state (an immutable admission-time snapshot: artifact identity, configuration fingerprint, expected effects) against observed evidence gathered from the target:

Workload identity. We SSH to the target and docker inspect the container labeled with this deployment's ID, then compare the image's sha256: digest and the container's generation against the plan.
Configuration fingerprint. Labels and environment key-set comparison — a drifted env var is a mismatch, not a footnote.
Health. Container running state plus Docker health status.
Route identity. This one's subtle and worth dwelling on. A healthy container with the right labels proves nothing about which deployment the reverse proxy is actually serving. So our managed edge stamps a deployment-identity header on responses, and verification fetches the public URL (raw TLS client, bounded retries, redirect-following) and checks the stamped identity equals the planned deployment ID. "Managed public route served deployment X instead of Y" is a first-class failure.

The verdict vocabulary is deliberately honest: verified | partially-verified | unverified | stale | failed. Unavailable evidence is explicit and can never satisfy a required gate; health success or access success alone can never produce verified. Mismatches come with machine-readable reason codes — artifact_identity_mismatch, configuration_fingerprint_mismatch, access_route_workload_mismatch — each carrying recommended remediation operations. The point isn't the verdict; it's that a downstream system (or a tired human) can trust verified because every way it could have been cheated is enumerated.

Only a verified proof moves a promotion to completed. The promotion descriptor literally derives its proofVerdict from that status — there's no code path where an unverified deployment reads as a successful promotion.

The ledger underneath: audit events and operator work

Underneath the chain sits a general audit sink: every mutating operation in the audited domains — projects, deployments, sandboxes, credentials, and more — is recorded as an audit event with the operation key (e.g. sandboxes.promotions.accept), the aggregate it touched, and a redacted payload (anything matching a secret-like key pattern is stripped before persistence). Retention, pruning, immutable archives with their own content digests, and legal holds are separate, boring, deliberate subsystems (ADRs 048/057/058, if you're reading along in the repo).

appaloft audit-event list and appaloft work list are the operator-facing read paths. When something looks wrong at 2am, the question "what happened, in what order, triggered by what" is a query, not an archaeology project.

Where the boundaries are

Honesty section, because we read the comments too:

Deploy & Verify is stable. Folders, Git repos, zips, images, Compose bundles, static artifacts, health, logs, rollback, proof readback — in production use today, self-hosted or on our cloud.
Agent sandboxes, source artifacts, and promotion are private preview. The domain model, state machine, proof engine, and operation catalog are in the public repo; the managed capture/preview/delivery adapters currently ship in our cloud distribution. The README carries a capability maturity table, and we'd rather you hold us to it.
A cheap VPS is not automatically sandbox-capable. gVisor isolation requires runsc registered with Docker; our provider probes for it and fails closed if it's missing.
The evidence chain is not a correctness argument. It won't tell you the agent's code is good. It tells you the code you approved is the code that's running, and proves it continuously.

Why build it this way?

We could have shipped "one-click AI deploy" a year earlier. The market is full of those, and they all share the same shape: an agent with too many credentials, a deploy script with too much trust, and a success message with too little evidence.

The bet behind the evidence chain is that the agent era needs less trust, not more convenience. Agents should operate through governed operations — the same operation catalog a human uses, over CLI, MCP, SDK, OpenAPI, or GitHub Action — with capability boundaries enforced by principal type, artifact identity enforced by content digest, and success defined as observed-reality-matches-approval rather than process-exited-cleanly.

If that bet sounds right to you, the repo is github.com/appaloft/appaloft, Apache-2.0. The 5-minute hello example walks the full deploy loop on your own server. And if it sounds wrong — if you think approval gates will dissolve under agent autonomy, or that content-addressed promotion is ceremony without payoff — that's the argument we want to have, in the issues or the comments below.

Appaloft is an open-source AI application delivery platform: run agents in isolated sandboxes, freeze exact candidates, promote explicitly, deploy to your own servers, and verify with evidence. Self-host with one install script; hosted cloud starts free.

DEV Community: 倪晨钦（wagahai）