How to Make ASI Research Credible Before It Exists

There's a genre of AI writing I want to kill: the ambitious research proposal dressed up as results. Specific benchmark scores with no experiments. Budget breakdowns with no funding. "Submitted to Nature" with no submission. I've seen this pattern enough times — including in my own early drafts — that I wrote a framework for what the honest version looks like.

This post is that framework, compressed. The full cleaned document is linked at the end.

The core idea: validation-first

Most ASI writing goes: big claim → bigger claim → roadmap → funding ask. The validation-first version goes:

1. State the hypothesis — what would have to be true?
2. Define the measurement — how would you know?
3. Set the bar in advance — what counts as success or failure?
4. Commit to reproducibility — can someone else check?
5. Report everything — including the failures.

Nothing about this is original. It's just the scientific method, applied to a field that keeps forgetting it when the claims get exciting.

What the honest version removes

When I cleaned up my own draft, here's what had to go:

• Invented benchmark scores. My draft had a comparison table: GPT-4 at 78%, Gemini at 82%, my proposed system at >90%. Those numbers were made up. An asterisk saying "requires verification" doesn't un-make-up a number. The honest table has no scores — just qualitative comparisons, with every "proposed" cell labeled as hypothesis.
• Fabricated study results. The draft contained "preliminary results" for medical and educational studies — 94% vs 87% accuracy, p < 0.01, 35% faster learning. No studies were run. What survived: the study designs. A good experimental design with no results is honest. Results with no experiment are fiction.
• A $115M budget and named funding sources. Specific dollar amounts, "applied", "in discussion", "negotiations ongoing" — unverifiable and, frankly, invented. Removed entirely. Resource planning before you have a prototype is premature anyway.
• Submission claims. "Submitted to Nature Machine Intelligence." If you can't show the submission confirmation, don't claim the submission.
• Named institutional partners. Listing MIT, Stanford, DeepMind as validation partners with "agreements pending" implies relationships that don't exist. Validator profiles, not names.

What survived: the skeleton

What was worth keeping is the structure, not the specifics:

• Hypotheses paired with measurements. Not "our system is better" but "here is the exact experiment that would show it."
• A benchmark suite proposed in advance — MMLU, HumanEval, and friends — with no target scores attached. Targets without a working system are wishes.
• Reproducibility as a first-class requirement: open code, documented hyperparameters, containerized environments, independent verification.
• Pre-registration thinking: define success criteria before running the experiment, so you can't move the goalposts after.
• Safety and ethics as architecture, not an appendix: bias auditing, uncertainty communication, human oversight, fail-safes.

The uncomfortable rule

Here's the rule I'm holding myself to, and I'd suggest it to anyone writing ambitious AI proposals:

Every number in the document must answer: measured, cited, or removed.

• Measured: you ran the experiment, here's the protocol.
• Cited: someone else measured it, here's the source.
• Removed: everything else.

"Illustrative target" is not a fourth category. I tried that. It's just fabrication with better branding.

Why this matters

ASI is the highest-stakes claim in technology. Every fabricated number in an ASI proposal doesn't just mislead readers — it trains the field to accept theater as progress, and it gives skeptics legitimate ammunition to dismiss the entire endeavor.

The irony: the validation-first framework is more ambitious than the hype version, not less. It's easy to claim >90% on a benchmark. It's hard to design the experiment that would prove it, pre-register the analysis, open-source the code, and publish the negative results. The second thing is what actually moves the field.

---

Full framework document: "Toward Credible ASI Research: A Validation-First Framework" (REWRITE-CLEAN v1.0, CC BY 4.0) — a conceptual proposal containing no empirical claims. By Shivam Kumar, VisionQuantech.