DEV Community: SPOE

Every change our AI app builder makes gets a second model’s review. A timeout is never a pass.

SPOE — Sat, 10 Oct 2026 10:13:58 +0000

I'm building SPOE, an AI app builder. You describe an app, a model writes it into a real Next.js, Fastify and Postgres project, and you can export the whole thing. Last time I wrote about how every export is signed. This post is about the other half: nothing the builder writes reaches your project until a second model has read it.

Why a second model, and why a different family

A model reviewing its own output tends to like it. Two sizes of the same model family share training data, so they tend to share blind spots too. So the rule is that the reviewer must come from a different model family than the writer.

That rule lives in code, not in a doc. The server reads the family off each model's id (Qwen, GLM, Llama, DeepSeek, Mistral and so on). If it can't tell, the operator has to state it. If the writer and the reviewer turn out to be the same family, the server refuses to start:

if (g === r) throw new Error(
  `Reviewer and generator are both "${g}" family. The security review must use a different model family.`
);

If you pick your own model for building, the reviewer is chosen from a different family than that one too. Both run on Venice's private, zero-retention inference.

What the reviewer sees

It gets two things: the original request and the proposed diff. Its instructions open with "You are a security reviewer. You did not write this code."

It looks for, roughly in this order:

Critical: injection, auth that can be bypassed, secrets in code, arbitrary file access, one user reaching another user's data. Also any import of an SPOE package, because the export has to run without us.
High: missing validation at the edges, weak crypto or password storage, SSRF, path traversal, XSS.
Medium: missing rate limits on auth, error messages that leak, insecure defaults.

It answers in JSON at temperature 0: a verdict, and findings that each name a file, a line, the problem and the fix.

Don't trust the model's own summary

A model will happily list a critical finding and then write "verdict": "pass" underneath it. So the verdict is worked out again on our side, from the findings:

const verdict = findings.some((f) => sev(f) === 'critical') ? 'block'
  : parsed.verdict === 'pass' && findings.some((f) => ['high', 'medium'].includes(sev(f))) ? 'warn'
  : parsed.verdict;

A critical finding blocks the change, whatever the reviewer wrote. A blocked change can't be approved at all. You regenerate, and the findings go back to the writer word for word as a checklist.

A timeout is never a pass

This is the rule I care about most. When a review errors, times out or returns something that doesn't parse, the easy move is to let the change through. SPOE doesn't. A network or provider hiccup gets one retry. After that, the change comes back marked as not reviewed, with a note in plain words saying exactly that. Approving it takes an explicit "I know this is unreviewed", and that approval goes into the audit log.

Fast, without skipping the check

At first, every change waited for a human in a diff view. Safe, and slow. Now, with auto-apply on (the default), a change the review passes goes in by itself, as a version you can undo in one click.

Four kinds of change never apply themselves: a blocked one, an unreviewed one, one with a high or critical finding, and one whose files you edited by hand since it was written. Those wait in the diff view for you. A manual edit is never silently overwritten.

Where it falls short

A second model is not a security audit. It catches common, obvious mistakes. It will miss subtle logic bugs, and it can be wrong in both directions.
It reads the diff and the request, not the whole running system.
Calibration never ends. A reviewer that calls everything critical gets ignored, so its instructions spell out what critical means: a human must not merge this, full stop. A missing rate limit is not that.
Two families can still share blind spots. Different beats same. It is not a guarantee.

Try it, and tell me where it breaks

SPOE is at spoe.ai. You get 50 free credits to start, a build costs roughly 10 to 30, and your first payment of any size unlocks export for good.

I'd like to hear where you think this review would fail. What would you want it to catch that it doesn't?

We sign every export our AI app builder produces. Here's how it works

SPOE — Sat, 10 Oct 2026 01:49:26 +0000

I'm building SPOE, an AI app builder. Every tool in this space says "you own your code". I wanted to be able to prove it, so every export SPOE produces is signed. This is how that works, in enough detail that you can poke holes in it.

The problem

When an AI writes your app, two questions follow the code around. Is this exactly what I was shown? And does it quietly depend on the tool that made it? A ZIP answers neither.

What's in an export

Three files ride along with every project: PROVENANCE.json, PROVENANCE.sig and verify-provenance.mjs.

PROVENANCE.json lists every file with its SHA-256 and size, a hash over the whole tree, which models wrote and reviewed the changes (never the prompts), and the signer's public key:

{
  "spoe_provenance": "1",
  "project": { "slug": "tint-booker", "stack": { "frontend": "nextjs", "backend": "node", "database": "postgres" } },
  "version": { "number": 2 },
  "files": [{ "path": "web/app/page.tsx", "sha256": "9f2c…", "bytes": 1834 }],
  "tree_sha256": "41ad…",
  "generations": [
    { "role": "generator", "model": "…" },
    { "role": "reviewer", "model": "…" }
  ],
  "signer": { "alg": "ed25519", "key_id": "c85280961cc26721", "public_key_pem": "-----BEGIN PUBLIC KEY-----…" }
}

The signature in PROVENANCE.sig is Ed25519 over a canonical form of that JSON: keys sorted recursively, no whitespace. The tree hash is SHA-256 over the sorted lines path\0sha256\n, so the order files happen to be zipped in never matters. The key id is the first 16 hex characters of the SHA-256 of the public key's DER encoding.

Verifying it

The verifier is about 40 lines of plain Node 18+, with no packages and no network:

$ node verify-provenance.mjs
OK   signature valid (key_id c85280961cc26721)
OK   30 files match manifest
VERIFIED - exported 2026-10-09T01:24:13Z from project tint-booker v2

It checks that the key id matches the embedded key, that the signature is valid, that every listed file is on disk with the right hash, and the tree hash. It exits 1 if anything is off.

A valid signature only tells you the files match a key. To know who signed it, compare the key id with the keys spoe.ai publishes at spoe.ai/api/signing-keys. The page at spoe.ai/verify does all of this in your browser without uploading the ZIP, and also lists any file in the ZIP that the manifest doesn't know about. The offline script now does that same check too.

"Runs without us" is a test, not a promise

Before an export is signed, a lint refuses it if anything ties it back to SPOE: an import of an SPOE package, a dependency on one, code reading SPOE_* environment variables, a call to SPOE's API, or one of our internal folders. No warnings, no override.

And CI does the whole thing on every commit: export the scaffold, unzip it into an empty directory, verify the signature, docker compose up, hit the health check. If that fails, nothing ships.

The other half: a second model

Every change the builder makes is reviewed by a second model from a different model family before it lands. A critical finding blocks the change whatever the reviewer's own verdict says, and a review that times out or errors is never treated as a pass: it waits for a human.

What it doesn't prove

A signature proves integrity and origin, not quality. It can't tell you the code is free of bugs or safe to run; the review and your own eyes are for that. It also says nothing about the prompts, which are deliberately left out of the manifest.

Try it, and tell me what's wrong with it

SPOE is at spoe.ai. It's free to start (50 credits on sign-up and a daily top-up), runs on Venice's private models, and your first payment of any size unlocks export for good.

I'd especially like critique of the manifest format and the verifier. What would you need to see before you trusted an export from a tool like this?