I'm building a diagnostic tool for A-level maths. A student gives it a question they're stuck on, and it works out why they're stuck not by explaining anything, but by asking short questions downward until it finds the thing underneath that's actually broken.
The reason it has to work downward is that "I can't differentiate" is almost never the real answer. A student who can't differentiate √x very often can't rewrite √x as x^(1/2) in the first place. The power rule was never the problem. They were stuck one floor below, and no amount of re-teaching differentiation will touch it.
So the subject gets modelled as a graph, where each skill declares what it sits on top of:
SKILLS = {
"differentiate_root_function": ["rewrite_index_form", "power_rule"],
"power_rule": ["multiply_terms", "subtract_integers"],
"rewrite_index_form": ["fractional_indices", "negative_indices"],
"fractional_indices": ["index_laws", "fractions_basic"],
"negative_indices": ["index_laws", "reciprocals"],
"index_laws": ["multiply_terms"],
"reciprocals": ["fractions_basic"],
"multiply_terms": ["times_tables"],
"subtract_integers": ["number_line"],
"fractions_basic": ["times_tables"],
"times_tables": [],
"number_line": [],
}
That's a slice of it. The real graph is 58 nodes across five layers, from exam-level tasks down to arithmetic a Year 9 student would meet. An LLM generates a question for whichever skill we're testing; the graph decides which skill to test next. This post is about that second part, because that's where the product lived or died.
Validate the graph before you trust it
A skill graph is hand-authored, which means it's wrong. Two failures will bite you immediately: an edge pointing at a skill that doesn't exist, and a cycle A needs B, B needs C, C needs A. A cycle will hang your walk forever, and it's very easy to write one by accident when you're 58 nodes deep and convinced everything depends on everything.
def validate(graph):
for node, needs in graph.items():
for n in needs:
if n not in graph:
raise ValueError(f"{node} needs unknown skill {n!r}")
WHITE, GREY, BLACK = 0, 1, 2
colour = {n: WHITE for n in graph}
def visit(n, path):
if colour[n] == GREY:
raise ValueError(f"cycle: {' -> '.join(path + [n])}")
if colour[n] == BLACK:
return
colour[n] = GREY
for c in graph[n]:
visit(c, path + [n])
colour[n] = BLACK
for n in graph:
visit(n, [])
The three-colour trick is the standard cycle check. Grey means "currently on the stack" — if you arrive at a grey node you've come round in a circle. Black means "fully explored, nothing bad below." The path argument exists purely so the error message tells you which cycle, which matters a lot at 3am.
The version I wrote first
My first instinct was to be thorough: test the skill the student failed, then test everything directly underneath it, then everything underneath whatever failed there.
def walk_breadth_first(graph, entry, solid):
asked, gaps = 0, []
frontier = [entry]
while frontier:
nxt = []
for node in frontier:
asked += 1
if not solid(node):
gaps.append(node)
nxt.extend(graph[node])
frontier = list(dict.fromkeys(nxt))
return asked, gaps
solid(node) is the question: generate one, ask it, return whether they got it. This is a level-by-level sweep, and it is genuinely thorough. It's also unusable, and the reason took me embarrassingly long to see.
Every failure widens the frontier. Fail one skill and you've queued two more questions. Fail both of those and you've queued four. On my real graph, a student who was struggling broadly could be asked 42 questions before the walk terminated.
Nobody sits through that. But the number wasn't the worst part.
The inversion
Here's what the two strategies actually cost, measured over 30 seeded students across three entry points:
student type BFS mean BFS worst
knows most 1.3 5
mixed 3.0 14
fails everything 10.3 16
Read the last column downward. The stronger the student, the shorter their test. The weaker the student, the longer it goes on.
That is precisely backwards. The student who needs help most is the one being asked to sit through sixteen questions to get it, and they're failing nearly all of them on the way. The thoroughness wasn't a neutral cost paid for better data — it was being charged directly to the students the tool exists for.
Following one branch down instead
The fix came from noticing what I was actually looking for. I don't need a complete map of everything the student can't do. I need the break point — the lowest thing that's broken — because that's where teaching has to start.
So when a skill fails, don't fan out across its siblings. Pick one prerequisite and go straight down:
def walk_depth_first(graph, entry, solid, cap=15):
"""Follow ONE failing branch to the floor.
Siblings not taken are recorded as unchecked, not tested."""
asked, gaps, unchecked = 0, [], []
node = entry
while True:
if asked >= cap:
break
asked += 1
if solid(node):
break
gaps.append(node)
children = sorted(graph[node], key=lambda c: depth(graph, c))
if not children:
break
node, skipped = children[0], children[1:]
unchecked.extend(skipped)
return asked, gaps, unchecked
Two decisions inside that loop are worth pulling out.
sorted(..., key=depth) picks the most foundational sibling first. If a skill rests on both index laws and basic fractions, fractions is further down, so test that. If the deeper thing is broken, the shallower one was never going to hold anyway, and you found the true floor in one move instead of two.
unchecked is not a throwaway. Those are branches we deliberately didn't test, and the report has to say so — a "Not checked" section listing them, with a command to re-run down a different branch. A diagnostic that quietly skips things and presents itself as complete is worse than one that asks too many questions.
Same measurement, both strategies:
student type BFS mean BFS worst DFS mean DFS worst
knows most 1.3 5 1.1 3
mixed 3.0 14 1.6 4
fails everything 10.3 16 3.7 4
Worst case 16 down to 4. On the full 58-node graph, the same change took the worst case from 42 to 15, with a mean of three or four.
But look again at the bottom row rather than the headline. The student who fails everything used to be the expensive one; now they're one of the cheapest. That's not a tuning win; it's structural — if you fail everything, you hit the floor of the graph almost immediately, and the walk stops, because there's nothing underneath the floor to ask about.
The inversion didn't get optimised away. It got removed because the new traversal was answering a better question.
What I'd take from this
The LLM in this system generates every question a student sees. It is doing the part that looks like the hard part. And essentially none of my time went there.
The product was unusable because of a traversal choice, and it became usable because of a different traversal choice — textbook depth-first versus breadth-first, the sort of thing you'd meet in a second-year algorithms course. No prompt engineering involved.
I think that's going to keep being true for a while. The model is a component. Whether the thing around it is any good is still ordinary engineering, and it's still where the work is.
One last practical note: seed your simulated users and check the distribution, not the average. My means looked fine the whole time this was broken. The problem only showed up when I split the results by how much the student actually knew, and that's exactly the split where you'd least want to be wrong.
I'm a maths tutor, and I built this because I kept watching the same thing happen in lessons: a student stuck on differentiation who was really stuck on indices, and nobody had checked. The graph is my attempt to make that check automatic. It's an unvalidated hypothesis at the moment; I hand-authored 58 nodes from how I teach, and I fully expect real usage to tell me the order is wrong in places.
So I'd genuinely like to hear two things. If you've built a diagnostic or an adaptive assessment, how did you decide what to test next? And if you're a teacher reading this — where would you say my graph has the dependency backwards?
Top comments (0)