A transparent protocol for evaluating an AI coding assistant on a real software task — with pre-registered acceptance criteria, controlled conditions, and reproducible evidence.
Byline: [Author] · [Publication date] · Experiment protocol; results pending
Status label: Worked example; no product run claimed.
Evaluate an AI coding assistant on a bounded, real task by pre-registering acceptance criteria, controlling conditions, running the task, checking the result against criteria, reviewing the patch, and rerunning from a clean state. Publish the evidence and limitations. One task shows only what happened in that recorded setup — not universal reliability.
Sections (with expert depth):
Why "it works" is not evidence.
Expert depth: Distinguishes demonstration from evidence; explains why "it compiled" ≠ "it's correct."
Pre-registering acceptance criteria (and why this matters for AI evaluation).
Expert depth: Explains why pre-registration prevents post-hoc rationalization; the AI can't redefine success after seeing the output.
Controlling conditions (model, tool, permissions, environment, time).
Expert depth: Explains why uncontrolled comparisons are meaningless; lists what to hold constant and what to disclose.
The run: what to capture and what to redact.
Expert depth: Explains why redaction is a design decision, not an afterthought.
Checking the result: behavior, not just compilation.
Expert depth: Explains why "tests pass" ≠ "behavior is correct"; covers edge cases.
Patch review: scope, correctness, regressions.
Expert depth: Explains why diff size is a signal; covers what to look for.
Clean rerun: the reproducibility test.
Expert depth: Explains why rerun is the hardest part and the most important.
What one task can and cannot show.
Expert depth: Explicit limits; no universal claims.
This method works independently of SKY. Check the current product page for what SKY actually supports
visit -https://codeforfree.in/.
Top comments (0)