How to Score a Skills Test So the Number Actually Means Something

You have taken this test. Twenty multiple-choice questions, one point each, a percentage at the end. It is quick to build, easy to game, and it barely predicts whether someone can actually do the work.

If you are building any kind of skills test, quiz, grading logic, or automated assessment, that final number is the whole product. And "one point per right answer" quietly throws away most of the signal you could be capturing. We have spent an unreasonable amount of time on that one number while building a developer assessment engine, so here are the ideas that made our scores mean something. None of them are exotic, and all of them transfer to any scoring system you might build.

1. Multiple choice is a weak signal

A single-answer multiple-choice question with four options gives someone a 25% chance of being right by pure luck. Stack twenty of them and the noise floor is high. Worse, recognizing the right answer in a list is not the same skill as producing it, and neither is the same as knowing when to apply it.

So the first move is to use question formats that are harder to guess and closer to the actual work:

• Ordering: put the steps of a deployment, or a git workflow, in the correct sequence.
• Matching: pair concepts with definitions, or tools with use cases.
• Code completion: fill in the missing part of a real snippet.
• Fill in the blank / predict the output: type the answer, or say what a piece of code prints, checked against a set of accepted values.

These test whether someone can do the thing, not whether they have seen the thing. But the moment you leave multiple choice behind, simple right/wrong grading stops being good enough, which leads to the next idea.

2. Give partial credit: score a ratio, not a boolean

If a question asks someone to order six deployment steps and they get four in the right place, marking that a flat zero is bad measurement. It throws away real information about what they know.

The fix is small but changes everything: have each question return a score between 0 and 1, not a true/false. For an ordering question that is just "how many items landed in the right position":

// return a ratio 0.0 - 1.0, not just correct/incorrect
function scoreOrdering(array $correctOrder, array $answer): float {
    $hits = 0;
    foreach ($correctOrder as $i => $step) {
        if (($answer[$i] ?? null) === $step) $hits++;
    }
    return $hits / count($correctOrder);
}

Every question type gets its own version of this: matching returns the fraction of pairs correct, code completion the fraction of blanks correct, and so on.

One place partial credit needs teeth is "select all that apply." If you only reward correct picks, the smart move is to select every option and collect full marks. So subtract the wrong picks:

// selecting everything no longer guarantees a good score
$ratio = max(0, ($correctPicked - $wrongPicked) / $totalCorrect);

Now a scattershot answer cancels itself out, and a thoughtful partial answer still earns what it deserves.

3. Weight questions by what actually matters

Not every question is worth the same. "Debug this broken IAM policy" tells you far more than "which flag does this command take." So give each question a weight, and make the final score the weighted average of those per-question ratios:

finalScore = sum(weight * ratio) / sum(weight)

This has a nice side effect: a short, well-weighted test beats a long flat one for signal per minute of the taker's time, and respecting people's time is its own reward.

4. Derive correctness, do not store it

Here is the least obvious decision, and the one I would most encourage you to copy. When someone answers, store what they did (which options they picked, the text they typed). Do not store whether it was right. Compute correctness fresh, from the raw answer, every time you need it.

It feels like extra work. It pays off constantly:

• One source of truth. Fix a bug in your scoring, or refine how a question type is graded, and every past attempt reflects the fix the next time it is viewed. There is no is_correct column sitting in the database, frozen at whatever your scoring logic believed on the day it was saved.
• Answer keys change. Add a valid synonym you missed to a fill-in-the-blank question, and answers that were unfairly marked wrong become right automatically, retroactively, for everyone.
• It keeps the record honest. You are storing what the human actually did and deriving judgment separately. The judgment is code, and code can be improved. The raw response is a fact, and facts should not be overwritten by an opinion you held last Tuesday.

The cost is re-scoring on read instead of a cheap column lookup. For test-sized data that is nothing, and the "every historical result is always correct" guarantee is worth far more than the microseconds.

5. Turn the score into a map, not just a grade

A single percentage is a weak output. "72%" does not tell anyone what to do next. Tag every question with a topic, then group the results by topic to produce a strengths-and-weaknesses view: "Networking 100%, IAM 80%, Cost optimization 50%." Suddenly the result is actionable, both for the person who took the test and for anyone evaluating them.

The implementation tip that saves you pain later: compute that breakdown in one function and call it everywhere you show results. The moment the taker's summary screen and a reviewer's report calculate the breakdown two slightly different ways, they drift apart, and you get support tickets about numbers that do not match. Build it once, render it in both places, and they can never disagree.

The takeaways

If you are building assessment or grading logic, steal these three:

1. Return a ratio, not a boolean. Partial credit is better measurement and costs almost nothing. You can always threshold a ratio back to pass/fail later. You cannot recover granularity you never captured.
2. Derive correctness, do not store it. Store what the user did, compute the judgment every time. Your scoring rules and answer keys will change, and you want every historical record to move with them.
3. Score topics, not just totals. A grade says how someone did. A topic breakdown says what to do about it, and that is the part people actually use.

A score is a claim about ability. If you want the claim to hold up, the scoring has to be more thoughtful than one point per right box.

---

We are building TrueCert, where developers and other professionals take short practical assessments across DevOps, cloud, databases, office, project management and more, and get a verifiable, shareable credential with a topic-level breakdown. Everything above is what runs behind every attempt.