I've been a developer for about ten years. I know how to find out whether code works: write a test, run it, read the output.
Finding out whether code helps people is a different job. I learned that by getting it wrong, repeatedly, over five weeks.
The project was an AI tutor for SQL with two versions, called arms. Arm A makes you try before it helps. Arm B gives the answer whenever you ask. Six people practised with one version or the other, then took a test a week later with no tutor. (Part 1 of this series has the full story and result.)
I kept a log of every assumption that broke. It ended up with 24 entries. Most of them are the same few mistakes in different clothes. Here are the six worth knowing about before you run an experiment of your own, whether it's a learning study or an A/B test on a feature.
1. The test measured a different skill than I thought
Before the study, I screened seven people with a written SQL test. No database, no Run button. Just write the query.
They solved 4 of 35 questions they attempted.
One colleague scored 0 out of 10. All four of his answers were conceptually right. He lost every mark to one habit: putting the sort direction in quotes. One run against a real database would have shown him the error.
Five days later, same people, I added a Run button. They solved 11 of 16.
Same people, one change. The first test wasn't measuring SQL skill. It was measuring "can you write perfect SQL in one shot with no feedback", which nobody does at work.
The rule: a test only predicts performance under the conditions it recreates. Before trusting a number, ask what conditions produced it. Then ask whether those are the conditions you care about.
2. I counted at the wrong grain
Four of my 24 logged mistakes were this one. I had real numbers, but counted at the wrong level.
The effect I cared about lives at one learner on one question. I called that a person-question. Three learners × 12 questions = 36 person-questions per arm.
But I kept slipping into "per person" or "per question" without noticing. Per person, I had 3 data points per arm, which tells you almost nothing. Per question, I mixed strong and weak learners together and hid who was struggling.
The worst version came early. I planned the study around "3 learners × 16 questions gives 48 moments where the tutor has to decide what help to give". But a question you solve on the first try needs no help. Neither does one you never open. So the real event isn't "a question asked". It's "a question where the first working attempt is wrong".
When I measured that rate, it was about 1 in 7 person-questions. The same 48 person-questions would give me about 7 help decisions, not 48. I had to cut the study from two topics down to one to have anything to measure.
The rule: for every number, say its unit out loud. "25 of 36" means nothing. "25 of 36 person-questions (3 learners × 12 questions)" can be checked.
3. "No help" is still a design
Arm B was my control: the version I compared the new idea against. It gave the answer on demand, and otherwise just showed the database's error message. I treated that as neutral. It wasn't.
One learner made 80 attempts in 47 minutes. None of them ran. His only real problem was writing keywords without a space: GROUPBY instead of GROUP BY.
SQLite's error message names the word after the broken one. So it pointed him at the wrong place. He rewrote his column names. He rewrote the HAVING line four different ways. In 63 tries on one question, he never touched GROUPBY.
Here's the trap. Suppose the bare error message is bad in a specific way, and the new version happens to fix that. Then part of the improvement you measure is just that fix, not the idea you were testing.
The rule: a control group is a design choice, not the absence of one. "No help" still has a user experience. Look at it as carefully as the new version.
This applies directly to A/B tests. If your control is the old checkout with a known confusing button, and your variant fixes the button, you might credit your new feature for the button fix.
4. Question order was a variable, and I didn't count it
The 12 test questions ran in skill order. Simple skills first, the hardest skill last.
Three of six learners stopped early, after question 8 or 9. Some ran out of time. Some gave up. On narrow screens, the question list cut off after question 9, so some may not have seen the rest.
Because of the order, every early stop landed on the hardest skill. So my biggest gap between the arms, 6 points on the hardest skill, was really just 6 blank answers.
That's the worst kind of hidden bias. It doesn't add random noise. It produces a clean-looking result in exactly the place you were going to read most carefully.
The rule: anything that changes along with the outcome is a variable, even page order. Randomise order per user, and log it.
5. Being in the group is not the same as using the feature
Arm A's whole difference is what happens when you press Help. If you never press it, both arms are the same page.
Three of my learners never pressed Help during practice. One of them was in Arm A. He never experienced the try-first tutor at all, and still scored 12 of 12 on the test.
My headline result counted everyone in the arm they were assigned to: 36 vs 25, an 11-question gap. Counting only the four people who used Help as designed, the gap was 24 vs 21. Three questions.
Both numbers are honest. They answer different questions. "What happens if you give people this tool?" versus "What happens when people actually use it?" You need both, and you need to say which is which.
The rule: log what each user actually received, not just which group they were in. Report both side by side.
In product terms: if half your variant group never saw the new feature because it's below the fold, your A/B test is mostly measuring the old page twice.
6. I didn't say the study was unsupervised
This is the one a judge caught at Demo Day.
He asked whether the practice and the exam were done live. They weren't. Learners got a link and did it in their own time. That wasn't on my slides, and he'd assumed it was live.
Unsupervised means you can't see what else people used. Google. ChatGPT. A friend.
And it's not equal across arms. Arm B learners never needed to leave the page; the answer was one click away. Arm A learners were held at a gate until they made an attempt. That's exactly when you'd open ChatGPT in another tab.
If they did, they skipped the struggle that Arm A exists to create. That weakens Arm A's tutor only, and pulls the result toward "no effect". The unsupervised exam does something similar: outside help lifts both arms and shrinks the gap.
So my setup most likely made the result look weaker, not stronger. That's a good thing to be able to say. I just didn't say it, so the audience filled the gap with their own default.
The rule: if a condition would change how someone reads your result, it goes on the slide. Live or remote. Supervised or not. Who dropped out. People assume the standard version of whatever you leave out.
The one habit that saved me
Before I saw a single answer from the test, I wrote down how I'd read the result. That included ten checks that could break it. Things like "neither arm above 80%", so there's room to see a difference, and "Arm B tries less than Arm A before asking for help", so the arms actually behave differently.
Two of those checks failed. Arm A hit 100%. Arm B tried before asking for help almost as often as Arm A.
Because I'd written the checks first, I couldn't explain them away. I had to change my claim. My hypothesis went from "trying first improves recall" to "trying first changes what you do after a wrong answer". That change is the most useful thing the project produced.
If you take one thing from this post: write down what would prove you wrong before you look at the data. It's the same discipline as writing a failing test before the fix. Ten years of TDD habits turned out to transfer after all.
Key takeaways
- Check the conditions. A test measures the skill its conditions recreate, not the one you named it after.
- Check the grain. Give every number its unit, and make sure it's the unit where the effect lives.
- Design your control. "No change" has its own UX, and that UX can create your result.
- Randomise anything ordered. Position is a variable.
- Log who actually used the feature. Being in a group and using the feature are different numbers. Report both.
- State your setup. Your audience will assume the default for anything you don't say.
All the logs, including the 24 broken assumptions, are in the repo:
Learning OS — 100x Cohort 7 final capstone
Vikrant Negi. Solo. Demo Day track. Submission 7 Oct 2026.
An SQL tutor that makes you try before it helps, and a 6-person pilot testing whether that changes what you can do a week later, with the tutor gone.
For where things stand today, read STATUS.md. That is the only file that claims to be current.
Try it
Live app: https://tranquil-starlight-f0129e.netlify.app/
- Withholding tutor: https://tranquil-starlight-f0129e.netlify.app/index.html?arm=A
- Answer-giving baseline: https://tranquil-starlight-f0129e.netlify.app/index.html?arm=B
What each one does:
-
?arm=Aopens the withholding tutor. Press Help before trying, and it refuses. Try once, and Help gives a hint. Two more tries, and it gives the answer. -
?arm=Bopens the baseline. Help gives the full answer at once.
Same page, same 16 practice questions, same grader. One config flag is the only difference.
The result
Arm A scored 36 of 36 person-questions (3 learners x 12 held-out questions). Arm…
Which of these have you hit in your own A/B tests? I'd bet #5 is the most common one in product teams. Please feel free to leave any feedback.
Top comments (0)