Printing the answer isn't solving the exercise: how I grade beginners' code

Every beginner finds the same trick sooner or later. The exercise shows an example:

Cara
Cara is at station 2

and the fastest way to a green check is this:

input()
print("Cara is at station 2")

If the grader only runs the program once and compares what it printed with the example, that passes. The student learns nothing, and worse, learns that the checkmark doesn't mean anything.

I'm building CyberStars, a free, open source site where you learn Python, Java, C and Linux in the browser. Every exercise is graded on the server, and I wanted a green check to mean "you wrote the thing the lesson teaches". This post is how the grader gets there, using one real Python exercise as the example.

The exercise

The lesson is about linear search. The mission:

The station roster is a list named crew. Write a function find_crew(crew, target) that uses linear search to return the index where target sits in the list, or -1 if it isn't there. Then read a name and print Cara is at station 2, or Zane is not on board.

The reference solution is written by me and stays on the server; students only see it if they ask for it.

1. Test on inputs the student never saw

The obvious fix for the printed answer is more test cases. This exercise has 50. The student sees 2 of them (Cara and Tommy), the other 48 are hidden: a name that isn't there, the first and the last name, a list with one person, names that start the same (Ann, Anna, Annabel), and so on.

When a hidden test fails, the student sees the input and what their program printed, but not the expected answer. Enough to debug, not enough to copy.

2. The reference solution is the oracle

Writing the expected output for 50 cases by hand is slow and easy to get wrong, and it drifts the moment someone edits the exercise. So the tests don't store expected outputs at all.

For every case, the grader runs both programs, the student's and the reference, with the same input, and compares what they print. The reference decides what's right. A test case is just an input:

{ "inject": { "crew": { "$list": ["Ada", "Ben", "Cyd", "Dot"] } }, "stdin": "Dot\n" }

Both programs run inside a Docker container with no network, but the comparison happens on the server, so the expected output never even enters the container the student's code runs in.

Running the reference costs time, so for the generated bulk cases its output is cached on the server. The hand-written cases always run it fresh.

3. Change the data, not just the input

Look at that test case again: besides stdin, it has inject. Different input alone isn't enough here, because a student can hardcode the list:

target = input()
if target == "Cara":
    print("Cara is at station 2")
elif target == "Tommy":
    print("Tommy is at station 0")
# ...one branch per name

So the mission names a variable, crew, and the grader swaps its value before running. It parses the student's code with Python's ast module, finds the assignment to crew and replaces the value with the case's list, in the student's program and in the reference alike. On one case, the student's program effectively starts with:

crew = ["Ada", "Ben", "Cyd", "Dot"]

instead of the five names they wrote. A program that really searches the list still works. One that memorised the answers for Tommy, Boris and Cara does not.

Only names the mission asks for are fixed. Everything else is the student's choice: this passes too, with other parameter names and enumerate:

def find_crew(names, wanted):
    for pos, who in enumerate(names):
        if who == wanted:
            return pos
    return -1

4. Right output, wrong way

Here is a solution that prints the right thing for every one of the 50 cases:

def find_crew(crew, target):
    return crew.index(target) if target in crew else -1

It's even good Python. But the lesson is about linear search, and this skips exactly the part the lesson teaches. So each exercise also carries structure rules, checked on the student's syntax tree:

"structure": {
  "requires": [
    { "kind": "variable", "name": "crew" },
    { "kind": "function", "name": "find_crew" },
    { "kind": "loop" }
  ]
}

No loop, no green check. The rule says which construct is required, not how to write it: for, while and enumerate all count.

5. Test the tests

The part I like most. How do you know 50 cases and a few rules are actually enough? You write the wrong answers yourself and check that they fail.

Next to each exercise sit a few extra programs. The ones named accept-* are other correct solutions that must pass, like the enumerate version above, or one with a while loop. The ones named reject-* are near misses that must fail. This one forgets the last element:

def find_crew(crew, target):
    for i in range(len(crew) - 1):
        if crew[i] == target:
            return i
    return -1

It passes both visible tests (Cara and Tommy aren't last). Only the hidden cases catch it: Cortez at the end of the list, and the list with just Solo in it. If someone ever trimmed the hidden cases down to the visible ones, this reject would start passing and the check would go red.

Every exercise in the course has variants like these: hardcoded output, the wrong loop, an off-by-one, a built-in that does the work. A script runs all of them through the real grader, the same one students use, and CI runs it on every change that can affect an exercise.

What it doesn't do

It isn't unbeatable. Someone determined enough can still find a way around it, the same way they could copy a solution from anywhere. The goal is smaller: the easiest way to get the green check should be to write the loop. The bet is that once cheating is more work than solving, most people just solve it.

And it costs some speed: every case runs two programs instead of one. The grader stops at the first failing case, which keeps a wrong answer quick to report.

Try it

The exercise from this post is Linear Search in Python on CyberStars. It runs in the browser and you don't need an account to try it. The code, the grader included, is on GitHub. Every lesson is also available in Romanian, which almost no free site offers.

If you build something similar, or you've found a way to beat this kind of grader, I'd like to hear about it in the comments.