DEV Community: Andrii Sparysh

Can a coding agent implement a binary format from its spec alone?

Andrii Sparysh — Wed, 28 Oct 2026 04:51:06 +0000

Coding agents now write whole programs from a short description. Most of the tasks they are tested on are forgiving, though. A web page that looks roughly right is fine. A function that passes the tests someone wrote for it is fine.

Here is a less forgiving task. Give an agent the specification of a binary file format and a set of test vectors. No reference implementation, nothing to copy. Ask it to write a verifier in a language of its choice and reproduce eleven published values exactly, down to the last byte of a SHA-256 hash. Then ask it to break the format seven different ways and confirm that the verifier notices each one.

We would like to know how far agents get with that, in which languages, and where they stop. We have a reason to ask, and a format to ask about.

Who we are and why this matters to us
We are a small team in Kyiv. We maintain PALA-1, a format for audit logs written by AI systems: an append-only hash chain that can be checked without a key and without opening any record. It is meant for machines that are modest and often offline, audited by people who may check a log but not read it.

An audit log is only evidence if someone other than its author can check it, with their own tools, possibly years later. So the specification has to be implementable from the text alone. That is the format's first promise. It has been frozen at version 1.0 since August 2026, with byte-exact test vectors and a kit for checking it by hand. Five implementations exist so far, three of them by people outside the project, and outside readers have found real defects before. That is how the text got better.

What we don't know is whether a coding agent can be one of those outside readers.

Why it is a fair test

The inputs are two files at a frozen tag, the specification and its test vectors.
The pass bar is exact. A value matches or it doesn't.
Seven demos ask the verifier to catch a flipped bit, a missing record, a gap in the sequence numbers and four other faults.
The language is the agent's choice. Four of the five existing implementations are in Python; others are more interesting.
The specification is not perfect, either. In two places it states what a demo must show but not exactly how to build its input. An agent has to notice that and say how it decided.

Two outcomes, both useful
If an agent gets a value wrong, we learn where agents still struggle with a precise text. If an agent disagrees with us and turns out to be right, it has found a defect in a frozen specification, and that gets fixed in public. Telling those two apart is the point of the exercise.

We will publish every report we receive in one table: model, tool, language, result, how long it took, and what the agent found ambiguous. That table is kept separate from the five implementations above.

We checked the prompt ourselves first: the files resolve, the digests match, and the whole task took an agent a few minutes. That run doesn't count, because we already know the format.

One caveat. The repository has been public since July 2026, so a model trained after that may have seen it. The prompt asks the agent to say so.

The prompt
Copy it into any coding agent that can download files and run code.

Write a verifier for the PALA-1 audit-log format from its specification alone. Download PALA-1.md and test-vectors.json from https://raw.githubusercontent.com/Assault-Consulting/Palimpsests/pala1-v1.0/docs/specs/pala-1/ and use nothing else: don't clone or browse the repository, don't search the web for PALA-1, and don't rely on anything you already know about it. Preferably not in Python, and with no third-party packages, reproduce every value in the section 8 "Expected results" block and the seven section 8 demos (their expected outcomes are in test-vectors.json). Where the text is ambiguous, write it down and carry on. Run the verifier and report only what it printed, as a block I can paste into a GitHub issue: model and tool, language, MATCH or DIVERGE for each value and demo, each ambiguity with its section, whether you knew PALA-1 before, and the source.

The verification kit describes the same task in full, with SHA-256 digests for the inputs, if you would rather do it by hand.

Where to send the report
Post it as a comment on issue #318, where every report is collected in one table.

If your agent reports a divergence, the most useful thing you can add is the first byte where its value and ours differ.

Links
The specification and test vectors: docs/specs/pala-1
The format is also described in an Internet-Draft, draft-sparysh-pala-audit. It is an individual submission, not an IETF standard.
More about the format: palimpsests.dev/pala-1

How to verify an LLM agent's audit log without the decryption key

Andrii Sparysh — Sun, 04 Oct 2026 13:52:54 +0000

If an agent's log says it called a tool on a given day, how does someone outside your team check that, without you handing over the key that decrypts every prompt in it?
Below is the check we built into Palimpsests (PALA-1), an Apache-2.0 runtime for local LLM inference. You can run all of it offline in about a minute.
What the verifier looks at
Every record in the log has a header and an optional body. The header stores the SHA-256 of its own body and the hash of the record before it. The verifier walks the headers one by one and compares each body with the digest stored for it. It never decrypts a body, so it never needs the key.
Step 1: write a small audited run
pip install palimpsests
palimpsests demo

The demo uses a deterministic backend, so nothing gets downloaded. It serves one turn, lets the model call a tool and writes the result to palimpsests-demo.pala. With 0.12.0 the output ends like this:

Audit chain written: palimpsests-demo.pala
Verifying with the production reader (what an auditor runs)
records: 7 chain_ok: True head: a3682a36b3a9b703…
recorded events: MODEL_LOAD, TOOL_CALL, TOOL_RESULT
Your head hash will be different, because the headers include boot ids and timestamps.
Step 2: verify the file on its own

palimpsests pala verify palimpsests-demo.pala
_
Text
_consistency: 7 records, chain intact
completeness: NOT CHECKED — no --anchor supplied, so tail truncation or wholesale replacement would not have been detected
witness: no WITNESS records — existence at a point in time is not attested
The exit code is 2.
With nothing but the file, the verifier can confirm that no record inside it was changed, dropped or moved. It can't tell whether someone deleted the last few records, or swapped the whole file for another chain that is consistent with itself. So it says that it didn't check, and exits with a code your CI can tell apart from success.
Step 3: give it an anchor
An anchor is the latest head hash, stored somewhere outside the file. In a real deployment that's an anchor file, a PKCS#11 token or a transparency service. For the demo, take the head from the JSON output and pass it by hand:
HEAD=$(palimpsests pala verify palimpsests-demo.pala --json | python3 -c "import json,sys; print(json.load(sys.stdin)['head'])")
palimpsests pala verify palimpsests-demo.pala --anchor "$HEAD"
Exit code 0. No key was used at any point.
Step 4: break it
Flip one bit at byte 200 and check again with the same anchor:

cp palimpsests-demo.pala tampered.pala
python3 -c "d=bytearray(open('tampered.pala','rb').read()); d[200]^=1; open('tampered.pala','wb').write(d)"
palimpsests pala verify tampered.pala --anchor "$HEAD"

Exit code 1.
The two lines contradict each other only at first sight. The edit was in an early record, so the last record and its hash are untouched and still match the anchor. The link between records 1 and 2 no longer holds, and the verifier names where. Someone investigating can tell an edit in the middle from a cut at the end, because the tool reports them separately.
Limits
The demo backend only stands in for a model. Real runs use llama.cpp or Ollama on your own hardware.
The log implementation has not been through an independent penetration test yet.
Why bother with any of this
For high-risk AI systems in the EU, Article 12 of the AI Act asks for automatic event logs, and deployers have to keep them for at least six months under Article 26(6). An auditor will usually start with the tool calls, since that's where an agent acts on the world. I wrote the legal side up separately, with dates and the actual fine levels: EU AI Act Article 12 for self-hosted LLMs.
The records use PALA-1 (Portable Append-only Log for Audit). The format was frozen at v1.0 in August 2026, the spec and test vectors are CC0, and five independent implementations have reproduced it from the spec alone. If you'd rather write your own verifier than trust ours, start at palimpsests.dev/pala-1.

I stopped writing automations and started hiring AI workers

Andrii Sparysh — Sat, 04 Jul 2026 09:33:31 +0000

title: "I stopped wr
Most "AI at work" is still a chatbot in a corner: you ask, it answers, the work is still yours. I wanted the opposite — something I could hand a whole task to and get a result back, not a suggestion.

So instead of building automations, I started hiring workers. Here's what that actually means in practice, and where the useful parts are.

First: any API becomes a tool, with no code

Before a worker is useful, it needs things it can do. The usual way to give an AI a new capability is to write an integration — read the docs, model the endpoints, wire the auth, handle errors. An afternoon each.

Instead: paste a domain. If the model already knows the API, you get a working tool immediately. If it doesn't, it searches, scrapes the docs across pages, and assembles the endpoints into one schema. Add the key, activate, done — the API is now a native tool your workers can call. No hand-written JSON, no glue code.

That alone removed most of the busywork. But the connectors are just the raw materials. The interesting part is who uses them.

Hiring a worker

A worker isn't a prompt. It's a hire with:

A role. What it's for, what "good" looks like, and where its boundaries are. This is the part that decides whether a worker is useful or useless — same as hiring a human.
A brain you choose. Each worker runs on the model you pick — Claude, GPT, or Gemini — so you can put a strong-reasoning model on the tricky role and a cheaper/faster one on the repetitive one.
Tools it's allowed to use. You hand a worker exactly the capabilities its job needs, and nothing else.

The tool set a worker can be given is the point:

Email — read an inbox, send and reply over IMAP/SMTP.
Webhooks — an external event can wake a worker and hand it the payload to act on.
Memory — a worker remembers across runs, so it doesn't repeat itself or lose the thread of a project.
File storage — durable storage that survives restarts, for the documents and artifacts a worker produces.
Site logins — you store a worker's credentials for third-party sites (encrypted, write-only — the password is never readable back), and it can log in and do real work on those sites on your behalf.

Tasks: work that starts without you

A worker owns tasks. A task runs on a trigger, not on you sitting there:

on a schedule (every morning, every two days, whatever the cron says),
when an email lands,
when a webhook fires.

So a content worker can draft and publish on a cadence; an inbox worker can triage mail as it arrives; a monitoring worker can react to an event the second it happens. You come back to results.

The supervisor: staying in control of the team

Handing autonomy to software is only comfortable if you can rein it in. That's the supervisor layer, and it's the thing I'd have refused to run without:

Hire and fire. Spin up a new worker for a role, pause one, or retire it entirely.
Assign and adjust tasks. Give a worker a task, change its schedule, edit its instructions, disable it.
Watch what it does. Every run is logged — you can see what each worker did, when, and why.
Approve the risky stuff. Anything that reaches the outside world (sending mail, publishing, posting) can go through an approval gate: the worker proposes, you approve, then it acts. You decide how much rope each worker gets, and you can widen it as trust builds.

That last part is what makes the difference between "an agent I'm nervous about" and "a teammate with a clear job." The worker does the work; the human stays the one who signs off.

Where this is honestly not magic

Worth saying plainly:

A worker is only as good as its role. Vague instructions → vague output. The org-design thinking doesn't disappear; it moves to you.
No-code connectors depend on the target API's docs. Clean docs → clean tool; weird undocumented endpoint → you'll still be nudging it.
It's a hosted platform and currently closed beta, so treat sensitive data and rough edges accordingly.

None of that is a dealbreaker for how I use it, but "hire a worker" doesn't mean "skip the judgment."

Try it

It's in closed beta — the first batch of builders get lifetime access to the core, plus some provider credit. If you want to build a small team that works while you sleep: (https://toolweave.dev/blog/en/build-a-team-of-ai-workers)

Question for anyone already running agents: what's the first role you'd actually hire for — inbox triage, a QA bot, a content worker, something else? And what would you not let it do without approval?