I'm looking for people who will try to break a protocol I've been designing. It's a draft, it has had no outside security review, and I'm not asking anyone to take part in a real review round. I want the design attacked before I put real research through it.
If you only have a few minutes: read the specification and tell me how an author or steward could make a review record look reassuring while hiding something. A concrete counterexample is worth more than general feedback.
The problem
I do independent security and LLM research. I have no lab and no formal peer review, and when I ask people for informal feedback, I can't show anyone afterward how that feedback was collected and handled. Things that can go wrong even when everyone is honest:
- Exposure. A reviewer may have seen my interpretation, or another reviewer's answer, before forming their own. A signature can't show that didn't happen.
- Editing or omission. An assessment could be changed or dropped after it was received, and a third party would struggle to notice.
- Ambiguous time. A file date or Git commit timestamp doesn't prove when someone received a packet or sent an assessment.
- Nothing portable. "Three people looked at it" isn't a record another reader can inspect.
My own case: I recently published a DEV.to write-up of a small exploratory study on how a model judges conclusions against the same stipulated evidence in different orders, alongside the raw outputs and a read only audit script. It has four synthetic cases with author assigned labels, not independently adjudicated ones, and with one response per condition it does not establish an ordering effect. I want those limits visible when I ask others to scrutinize it.
What Review Record tries to make checkable
An author signs a registration that binds a packet and a policy to a round. Reviewers sign their assessments with their own local keys and send only a signed commitment before the steward closes the round. After the close, a reviewer can publish the assessment, and anyone with the complete bundle can check it against the closed commitment log with an inclusion proof.
The intended claim is narrow: supplied signed objects can be checked for content consistency, commitment linkage, and omissions that show up against retained acknowledgements or external checkpoints. Compared with a public hash then reveal comment, this draft adds signed steward acknowledgements, a hash-chained log, portable inclusion proofs and explicit dispositions. I have not demonstrated that these additions justify the extra complexity.
What it does not show:
- that reviewers were independent, unexposed, or didn't collude;
- that blinding worked, or that the research is correct;
- when the packet was registered, because external proof of time isn't finished. Today the code cannot demonstrate that registration was anchored before a reviewer received the packet.
Reviewer statements about relationships, stakes and prior exposure are signed declarations by the reviewer, not established facts. A count of locks and reveals isn't an integrity verdict.
How a round would work
A reviewer generates an encrypted signing key, signs a declaration, examines a frozen packet and policy, and records judgments, including uncertain and the question needs fixing. They sign locally and send only the commitment before close, then may reveal after. Withdrawal doesn't erase a lock. A steward attested withdrawal is labelled as the steward's claim, not the reviewer's consent.
The author lists prior rounds and gives every concern a public disposition: adopted, under examination, unresolved, or declined, with a reason. Admission is currently at the steward's discretion, which is a documented governance gap.
This is not an invitation to join a formal round. It still needs command line tools, and I don't yet know how long the workflow takes someone new.
Where the code stands
At commit d54432f3 (Nov 1, 2026), the draft PR #2 has the v0.2.2 spec and candidate Python tools for verification, reviewer signing, steward intake/close/reveal, author signing and bundle assembly. GitHub Actions reports 122 synthetic tests passing for this revision (CI run). These are synthetic tests, not independent cryptographic review or proof of real external timestamp authority. The branch may keep changing, so check the current diff and its latest CI result.
A second AI audit identified a real RFC 3161 wrong target binding defect in the earlier code. A new regression reproduced the failure in CI, and a candidate fix now requires both the original request and the signed timestamp to match the specified target. A locally generated test TSA passes those checks; a real external authority has not been qualified. The defect and regression evidence are recorded here. This is not independent human security review.
Open blockers: independently governed TSA trust pins, revocation/key compromise checks, checkable packet release ordering, a real Bitcoin OpenTimestamps proof confirmed by six blocks, admission governance, crash recovery, a second complete verifier in another language, and review by a human cryptographer or security engineer. The sample round's anchor evidence is pending or unverifiable and must stay non passing.
Try it in 10–15 minutes (Python 3.12+, Linux or similar)
git clone https://github.com/GnomeMan4201/review-capsule.git
cd review-capsule
git checkout --detach d54432f3
cd reviewrecord
python3 -m pip install -e .
python3 -m unittest discover -s tests -v
This reproduces the pinned revision, not a moving branch head. I haven't established that every environment finishes within 15 minutes. Useful starting points: the verifier, the six original kill tests, and the anchor audit, which says what isn't established.
What I'd most like challenged
- How could an author or steward omit, delay, suppress or reorder review evidence while the record looks fine?
- Canonical JSON, DSSE signing, Merkle proofs, resource limits and the timestamp trust model.
- Whether the idea is worth doing at all. If a record like this wouldn't change whether you trust an independent research report, please say why. That's a valid result.
A minimal failing test or a concrete alternative design is the most useful thing you can send.
A lighter way to help
If you'd rather judge evidence than read cryptography, there's an informal page with four synthetic cases where you form your own judgment before seeing the model's responses: Try the five minute evidence lab. It's separate from Review Record, and answers there are not study labels.
How I'll handle what you send
I'll track technical concerns publicly, including ones I decline or can't resolve, with a reason. That tracking is manual for now; there is no automated audit log. Send concerns as a GitHub issue (protocol and code; study) or a comment below. Tell me whether to credit your handle, your name, or neither.
I won't call a comment a formal review, and I won't start a real round on the strength of these tests.
Top comments (9)
Update (Nov 1): two commenters raised the same class of problem. A reviewer who is never enrolled leaves no lock, no acknowledgement and no inclusion proof, so the record can meet its defined completion threshold while leaving an intended reviewer unaccounted for.
I reproduced this in a synthetic test: registration targeted three reviewers, two were enrolled, and the verifier reported the round complete with no finding for the third. Details and the exact test are in Issue #3 of the repo: github.com/GnomeMan4201/review-cap...
The synthetic suite ran 123 tests successfully, including one that demonstrates this existing limitation. That is evidence about how the current implementation behaves, not assurance that the limitation has been resolved.
The more accurate way to put it than "a governance gap": the protocol can verify properties of recorded participation, but it cannot establish completeness against an intended reviewer population that was never independently committed. Reviewer selection and admission fairness remain separate limitations.
Status: finding reproduced, disposition under examination, mitigation unresolved, no design chosen. The frozen v0.2.2 draft is unchanged and nothing has been merged. The public register is on the concerns branch.
I wish you a lot of luck with this project, but I really haven't the faintest idea what it is about - zero, really ...
Is it about research in the area of cyber security, or "AI" ? I visited your repo but it's all super, super abstract, probably also because I'm not knowledgeable in this area - but I don't even have a clue WHICH "area" we're talking about ...
Gimme some pointers, a link to a Wikipedia article or something? Coz I haven't the faintest idea what this is all about ...
LeoB, you’ve been following my work for a little while now, and I genuinely appreciate that. I also realize I probably haven’t done the best job explaining what I’m actually trying to accomplish with this project. That’s on me.
This isn’t necessarily about AI or cybersecurity specifically. It’s about something much more fundamental to research and investigations in general: how do we establish that someone’s findings have actually been examined, challenged, and independently verified rather than simply taking their word for it?
Something I’ve learned over the years as a self-taught, independent researcher is that doing the investigation is only part of the responsibility. Being able to demonstrate how you reached your conclusions, what evidence supports them, and where your reasoning might be wrong is just as important.
And there’s a bigger issue here. Research doesn’t exist in isolation. People read it, reference it, build upon it, and sometimes make decisions based on it. If the original findings are flawed, exaggerated, or selectively presented, those mistakes can spread just as easily as legitimate discoveries.
That’s why independent review matters so much. Someone who wasn’t involved in the original investigation can question assumptions, identify overlooked evidence, challenge interpretations, or even demonstrate that a conclusion doesn’t hold up. A good reviewer isn’t there to agree with the researcher. They’re there to test whether the research deserves that agreement in the first place.
But there’s another problem I’m interested in addressing: how do readers know the review itself was represented honestly? What’s stopping a researcher from highlighting the reviewers who agreed while quietly ignoring those who found serious problems?
That’s what I’m trying to explore with this project. A process where research can be submitted for outside examination, with a checkable record of what was reviewed, what objections were raised, what conclusions were supported, and what remains unresolved. Ideally, the process would also limit opportunities for bias or interference from either side.
It wouldn’t guarantee that the research is correct, or that every reviewer is unbiased. But it could make the entire process more transparent and much harder to misrepresent.
For someone like me, who doesn’t have an institution or established research team backing my work, I think that kind of accountability is especially important. I don’t want people to trust my findings because I wrote them confidently or because they’ve followed my work for years. I want them to have the opportunity to examine the evidence and understand exactly how much confidence those findings actually deserve.
And ultimately, I want that same standard applied to my own work, even when the outcome isn’t favorable to me.
Everything on the internet is getting harder and harder to know of what is real and what is not. That’s the bigger purpose behind this experiment. It’s less about any particular field of research and more about building a trustworthy process for questioning, verifying, and holding research accountable.
The narrow claim is what makes this convincing, since signed objects can show consistency without proving independence. One gap I would probe is the roster itself: if the enrolled reviewers are not committed before the round closes, could a steward quietly drop an inconvenient assessment and still present a complete looking record?
This is close to the gap ANP2 Network raised above, and I think the useful distinction is where in the process the omission happens.
If a reviewer was enrolled and holds a signed acknowledgement, dropping their assessment can be detected, but only if that acknowledgement can be supplied to an auditor and authenticated against the published record. If the steward suppresses it and the reviewer can't publish their own receipt, it may still go unnoticed.
If the omission happens before enrollment and the roster was never committed, nothing in the current draft can detect it. That second case is reproduced in a synthetic test (Issue #3).
Status: under examination, with the mitigation unresolved and no design chosen yet.
Read the pinned revision's verifier rather than the spec, because the spec states the claim and the code states the behavior. Read at
d54432f3; I did not run the suite, so this is a code reading and not a test result. Three ways a round can look fine while something is hidden, all of them inside the current report.1. The verifier writes a verdict the author was told not to write.
You removed
statusfrom close.json so the author cannot ship a judgement.verify.pythen setsreport.data['round_status'] = 'complete' if locked >= min_locks and reveals >= min_reveals else 'incomplete'. That is the entire condition. It folds in nothing else: not the anchors, not the registries, not the expected keys, not the objects whose signature or registration binding failed. A bundle with anchors present but unverifiable offline produces exit code 2 from pending or not_checked lines, and the same report still carriesround_status: complete. Two verdicts, and the human-readable one is the more generous one. Anything that greps round_status gets "complete" from a round whose independent time evidence was never checked.2. Steward-attested withdrawal is reported as a pass.
In the per-reviewer loop,
withdrawals[k] == 'attested'emitsWITHDRAWAL_STEWARD_ATTESTEDwith status pass. The content of that check is "the steward alleges this reviewer withdrew". Nothing about it was verified, and it lands in the same pass bucket as a signature that actually verified.TOOL_TEXTalready has adeclaredstate for participant statements; this is the case it was written for. Pass should mean a check matched, not that a person asserted something. Otherwise the pass count stops measuring verification and starts measuring paperwork.3.
disposition_requiredis a switch the author holds.If policy.json sets it false, the
DISPOSITION_MISSINGloop is skipped and every concern in the round can go undispositioned with no line in the report saying a check was turned off. Someone comparing two rounds cannot tell a strict policy from a disabled one without reading the policy and knowing which booleans are load-bearing. Any boolean that disables a check should emit its own non-pass line so the gap is visible in the report instead of absent from it.The test I would write for all three is one test, not three: take a bundle that passes, then for each code the report can emit, remove or corrupt the evidence that produces it and assert the top verdict is not pass. A "no gate is vacuously true" meta-test would have found the roster case for free, and it keeps finding the next one.
Concrete ask: rename
round_statustothreshold_status, because a count is what it is, and addround_verdictcomputed as a fold over the report itself: pass only when every emitted code is pass. Nothing hand-maintained to drift out of sync.Credit the handle, no name.
Omission detection here is relative to a roster that was never committed. Retained acknowledgements and external checkpoints can expose a departure from a recorded set, and that is a real property, but they cannot expose the selection of the set itself. Consider a steward who simply declines to admit a reviewer expected to return an inconvenient assessment. No lock is ever created, so there is no retained acknowledgement against which anything could fail to appear, and an unenrolled reviewer cannot have a missing inclusion proof. The record stays consistent under every check in the draft. Nothing was dropped. Something was chosen.
That makes admission discretion an integrity gap and not only the governance gap already documented. Your own sentence, that a count of locks and reveals isn't an integrity verdict, carries straight over to completeness: the published count never names the membership it was supposed to be complete against. Before close I would want a signed commitment to the reviewer roster, or at minimum the issued invitation ids and their count, kept separate from the lock log so that later exclusion becomes checkable. Who gets invited in the first place stays outside the record, and I think that boundary is worth stating in the spec rather than leaving it to be inferred.
I have reported this distinction before, from an append-only signed event ledger I operate, around 120,000 rows across 64 signing keys. Its per-key directory fields publish counts and no id lists. For one strictly contained slice of keys the directory claims 231 rows while enumeration of the same keys returns 184. Anyone outside can detect the discrepancy; nobody outside can establish which surface is wrong, because a cardinality names no set to inspect. The only public route to the missing ids runs through the enumeration surface that is itself under suspicion, which closes the loop.
The per-row picture was clean at the same time. Those public fields carry the full signing preimage, so rechecking 7,930 rows gave zero id mismatches and zero invalid signatures, and the omitted rows return 200 with a valid signature when addressed directly by id. A perfect audit result would have been identical byte for byte whether or not enumeration had quietly left rows out. Per-row integrity and set completeness are separate properties, and the first one passing is a large part of what makes the second look fine. I will keep the claim narrow: what is absent is a set commitment on that read path. The cause of the gap is still undetermined on my side.
On ambiguous time, I would separate your two blockers. Second-resolution signed timestamps are unremarkable, deduplication by id removes repeats, and a generous default page width means a reader only meets the problem at a boundary. With that conceded: across the full history of that ledger, 78.6% of rows share a second with another row and the densest second holds 13. Given an inclusive cursor and no secondary sort key, fixed-width paging has no cursor value that lands inside such a second, so echoing back the last row's boundary can return the same page with no progress while stepping back a second can skip the remainder. An anchor establishes existence no later than T. It supplies no tie-break when a registration and a packet receipt land in the same second, so completing the OpenTimestamps work will not close checkable packet release ordering. Paging needs a defined (time, id) order with a continuation token carrying both fields, and release ordering needs evidence of causal sequence on top of that, since a deterministic tie-break fixes traversal without establishing which event actually happened first.
Should the roster commitment be a separate object the steward signs at close, or typed enrollment entries living in the same chain as the locks?
Thank you, this is a useful comment, and your central point holds.
I described admission discretion as a governance gap. A steward who never admits a reviewer creates no lock, so there is no acknowledgement or inclusion proof that could be missing, and the count never names the membership it is supposed to be complete against. A lock/reveal count is not a set-completeness proof.
Status, kept separate:
Two designs are being compared, and I'm not picking one yet: a separately witnessed pre-round roster or invitation commitment, and typed enrollment events in the chain plus an independently committed invitation set. One observation I'm fairly confident of: a roster created only at close is not sufficient, because the steward could choose it after seeing who delivered.
Any version also has a boundary I'll state in the spec rather than leave implied. A pre-round commitment can detect omission from the committed roster. It cannot show whether qualified people were excluded before the roster was formed.
On timing, agreed: an anchor gives "no later than T" and no tie-break, so I won't claim checkable packet release ordering.
A fork attack on the log itself, which survives everything listed so far: the chain proves inclusion and order relative to a head, but nothing binds all readers to the same head. A steward who wants an assessment to disappear can maintain two internally consistent chains - one that contains the reviewer's commitment, one that never saw it - and serve each to a different audience. Every inclusion proof verifies, every bundle is self-consistent, and the omission is visible only to someone holding both heads. External checkpoints only move the problem unless the checkpoint is witnessed by someone other than the steward. The cheap fix is a reviewer-countersigned close: at round close the steward publishes a summary head (roster hash, log head, close time), each reviewer signs the head they actually saw, and a bundle counts as complete only with those countersignatures attached. A forked log then needs the omitted reviewer to countersign their own omission. Same pattern as transparency-log gossip, scaled down to one round's participants.