DEV Community: Oscar Rieken

Prompt engineering for teacher insights with Claude — structured JSON and graceful fallbacks

Oscar Rieken — Thu, 21 May 2026 04:08:11 +0000

What We Built

NumPath now generates a teacher insight on demand: click "Generate insight" on any student's panel and the system reads that student's KC mastery states plus their 10 most recent attempts, sends them to Claude, and returns a structured response: an actionable observation (text), a severity signal (type: warn/good/info), and a traceable evidence block pointing to the specific KC, p_mastery value, and mistake count that drove the insight. The evidence isn't generated by the LLM — it's assembled server-side from DB reads. Teachers get an interpretation layer backed by auditable data.

The backend piece is GenerateInsightUseCase — three DB queries, one LLM call, one JSON parse, one server-side evidence assembly. This post is about the schema design, the prompt engineering, and everything that can go wrong with the parse step.

The Design Decision

Structured JSON vs. free-form text

The original insight.txt prompt asked for a single sentence. Simple, but not structured enough for a UI that needs to display a summary and a suggested action as separate elements.

We rewrote the prompt — but here's the key design decision: the LLM only generates two fields. The evidence block is assembled server-side from DB reads before the call, so it's auditable regardless of what the model says.

You are a specialist math learning advisor for primary school teachers.
Given a student's Knowledge Component mastery data and recent attempt history,
generate a JSON response with exactly two fields:
- "text": one actionable sentence (max 30 words) citing a specific KC or mistake type
- "type": one of "warn", "good", or "info" based on urgency

Respond with only the JSON object. No explanation, no markdown, no code fences.

That last line — "no markdown, no code fences" — is the most important one. Without it, Claude wraps the JSON in a json block, and json.loads() raises JSONDecodeError on the backtick prefix. Every time.

Defensive fallback, never a 500

LLM output can't be trusted unconditionally. We introduced a module-level fallback constant and a pure parse function:

_FALLBACK_INSIGHT_TEXT = InsightResponse(
    text="Insight temporarily unavailable.",
    type="info",
    evidence=InsightEvidence(kc="", p_mastery=0.0, mistake_type=None, mistake_count=0, window=""),
)

def _parse_llm_fields(raw: str) -> tuple[str, str]:
    """Parse only the LLM-generated fields: text + type."""
    try:
        data = json.loads(raw)
        return data["text"], data["type"]
    except (json.JSONDecodeError, KeyError, TypeError):
        logger.warning("insight_parse_failed_using_fallback raw=%s", raw[:200])
        return None, None

The evidence block is never parsed from the LLM — it's built from the same DB data that was sent to the model. If parsing fails, the fallback fires. The endpoint always returns 200 with a clean InsightResponse. The warning log is the signal to watch for prompt drift after model updates.

StubProvider updated for the new schema

The StubProvider had to be updated to match — the old stub returned a plain string, which would now fail parsing:

class StubProvider:
    async def complete(self, system: str, user: str, max_tokens: int = 256) -> str:
        return (
            '{"text": "Student is consistently skipping the borrowing step — targeted regrouping practice recommended.", '
            '"type": "warn"}'
        )

The stub returns only text and type — the evidence block is assembled by the use case from DB data, not by the provider. All tests run offline with LLM_PROVIDER=stub — no API key needed, no network dependency, deterministic results. The provider abstraction (ADR-003) pays off here.

Why It Matters for the Research

The MacLellan NTI Framework (2018) requires that every AI insight be explainable and traceable — not just "this student needs more practice" but "this student has p_mastery=0.18 on SUB_BORROW with BORROW_SKIP classified in 9 of the last 11 attempts." The evidence field makes that traceability structural, not aspirational.

The Teacher-in-the-Loop principle is now fully implemented across three phases:

Phase 1: KC mastery bars — where is the student stuck?
Phase 2: Attempt history — what exactly did they do wrong?
Phase 3: LLM insight — what should the teacher do about it?

The distinction between Phase 2 and Phase 3 matters for the RCT. Phase 2 gives teachers raw evidence — they can verify it, disagree with it, or draw their own conclusions. Phase 3 adds an interpretation. The research question becomes: do teachers who see the LLM interpretation intervene differently from those who only see the raw data? We don't know yet. But we'll be able to measure it.

What We Learned

The hardest part wasn't the LLM call — it was making the output reliable enough to drive UI state. A streaming response or a multi-turn conversation would require a different interface entirely (ADR-003 explicitly deferred that). For a single completion call, the pattern is: instruct precisely, parse defensively, fall back gracefully.

The insight quality degrades predictably for students with few attempts — a student who's done 3 problems gets a much vaguer insight than one who's done 30. That's expected and honest, but it's worth surfacing to teachers: "This insight is based on 3 attempts" would make the confidence level explicit. That's a Phase 4 enhancement.

What's Next

The LLM provider abstraction supports multiple models — the next natural step is A/B testing Claude Sonnet vs. Haiku on insight quality vs. cost. That's not planned for MVP, but the infrastructure already supports it.

Key Takeaways

"No markdown, no code fences" is load-bearing prompt text — without it, Claude wraps JSON in backtick blocks and json.loads() fails; always explicitly instruct raw output
The fallback constant pattern beats exception propagation — a named _FALLBACK_INSIGHT at the module level makes the graceful degradation path explicit, testable, and readable
The StubProvider must match the current schema — when you change what the LLM is expected to return, update the stub immediately or you'll have tests that pass against outdated fixtures

Attempt history in the teacher dashboard — the scalar subquery pattern

Oscar Rieken — Thu, 21 May 2026 04:01:29 +0000

What We Built

In Phase 1 of the KC dashboard we gave teachers per-skill mastery states for each student — colour-coded bars showing Novice / Developing / Mastered. That answered "where is this student stuck?" Phase 2 answers "what exactly did they do wrong?"

We added a paginated attempt history table below the KC panel. For the selected student, teachers see every attempt in reverse-chronological order: the problem question, the student's answer, whether it was correct, the timestamp, which Knowledge Component was being practised, and — when the classifier fired — the mistake code (e.g. BORROW_SKIP, DIGIT_REVERSAL). The backend is a single endpoint, GET /teacher/students/{id}/attempts, teacher-only, with OFFSET pagination.

The Design Decision

The tricky part was joining MistakeEvent.mistake_code onto Attempt rows. Not every attempt produces a mistake event — the classifier only fires when an incorrect answer matches a known pattern. So the join is optional (LEFT JOIN territory), and the relationship is one-to-one in intent but not enforced that way in the schema.

We considered two approaches:

Option A — LEFT JOIN with GROUP BY:

SELECT a.*, me.mistake_code
FROM attempts a
LEFT JOIN mistake_events me ON me.attempt_id = a.id
ORDER BY a.created_at DESC
LIMIT 10 OFFSET 0

This works now. But if a future bug or edge case produces two MistakeEvent rows for one attempt, every teacher session silently doubles those rows. The bug would be invisible — no error, just wrong data.

Option B — Scalar correlated subquery:

mistake_code_subq = (
    select(MistakeEvent.mistake_code)
    .where(MistakeEvent.attempt_id == Attempt.id)
    .limit(1)
    .scalar_subquery()
)

stmt = (
    select(
        Attempt.id,
        Problem.content,
        Attempt.answer_given,
        Attempt.is_correct,
        Attempt.created_at,
        Skill.code.label("skill_code"),
        mistake_code_subq.label("mistake_code"),
    )
    .join(Problem, Attempt.problem_id == Problem.id)
    .join(Skill, Problem.skill_id == Skill.id)
    .where(Attempt.student_id == student_id)
    .order_by(Attempt.created_at.desc())
    .limit(page_size)
    .offset((page - 1) * page_size)
)

We chose Option B. The LIMIT 1 inside the subquery is a hard guarantee: regardless of how many MistakeEvent rows exist for an attempt, the result is always one row per attempt. The database enforces the invariant, not application logic.

The trade-off is a correlated subquery executing once per result row — potentially slower than a join on large datasets. At Phase 1 volumes (< 10k attempts per student), the composite index on (student_id, created_at) keeps p99 well under 150ms. Cursor-based pagination and a flattened join are on the Phase 3 backlog.

Why It Matters for the Research

The MacLellan framework's Teacher-in-the-Loop principle requires teachers to have information they can act on. Phase 1 gave teachers where students are stuck (KC mastery states). Phase 2 gives them evidence — the actual attempt record.

A teacher who sees "Emma: SUB_BORROW at 12% Novice" now has a follow-up panel: "Emma attempted Q.47 (345 − 178), wrote 177, BORROW_SKIP classified." That's a different kind of information — it's not a model output, it's an event record. Teachers can corroborate or question the classifier's judgement. That auditability matters for our RCT design: we're not studying whether the model is right, we're studying whether teachers who can see this data intervene differently.

Phase 3 will layer LLM-generated insights on top of this history. But the raw record had to come first — insights without evidence are just confident assertions.

What We Learned

OFFSET pagination has a well-known flaw: page 2 of an ordered result can drift if new rows are inserted between requests. For attempt history in a classroom tool this is acceptable — a teacher paging through a student's history isn't doing real-time analytics. We documented the cursor-based upgrade path in the architecture notes rather than over-engineering for a problem we don't have yet.

The access control pattern here is simpler than Phase 1's dual-role endpoint. This route is teacher-only, so require_teacher covers it cleanly — no route-level role check needed:

@router.get("/students/{student_id}/attempts", response_model=AttemptsPageResponse)
async def get_student_attempts(
    student_id: uuid.UUID,
    page: int = Query(default=1, ge=1),
    page_size: int = Query(default=10, ge=1, le=100),
    db: AsyncSession = Depends(get_db),
    _: dict = Depends(require_teacher),
) -> AttemptsPageResponse:
    ...

No require_authenticated + manual role check — the student version of this endpoint would be a separate route with its own access control. Keeping them apart is cleaner than a single endpoint that branches on role.

What's Next

Phase 3 adds a GenerateInsightUseCase — an LLM call that reads a student's attempt history and KC states, then produces a natural-language summary for the teacher. The attempt history table we built here is the input to that call.

Key Takeaways

Scalar correlated subquery > LEFT JOIN for nullable one-to-one — LIMIT 1 inside the subquery is a hard row-count guarantee that survives future schema changes; a LEFT JOIN can silently multiply rows if the cardinality assumption ever breaks
Raw evidence before generated insight — giving teachers the attempt record (what the student actually did) before generating LLM summaries (what the model thinks it means) keeps the teacher as the interpreter, not the model
OFFSET pagination is fine until it isn't — document the cursor-based upgrade path in architecture notes and move on; don't prematurely optimise for data volumes that don't exist yet

Why teachers need explainable AI, not just accurate AI — building the KC dashboard

Oscar Rieken — Thu, 21 May 2026 03:51:51 +0000

What We Built

NumPath's teacher dashboard previously showed one number per student: 7-day accuracy. A teacher looking at "Emma — 43%" has no idea whether Emma is struggling with borrowing, place value, number sense, or all three. The number is technically correct and completely unactionable.

In this post I'll walk through how we added a Knowledge Component (KC) mastery panel to the dashboard — colour-coded progress bars per skill that expand to show p_mastery %, mastery level label, and opportunity count. The backend piece is a single endpoint backed by a left-join use case. The research reason it matters is more interesting than the code.

The Design Decision

The core choice was: what data does a teacher actually need?

We had three options:

Accuracy-only (what we had): fast to compute, no additional queries, but unactionable
Raw BKT parameters: show p_mastery, p_learn, p_guess, p_slip — complete but overwhelming for a classroom teacher
KC mastery levels: translate p_mastery into a three-tier label (Novice / Developing / Mastered) with colour coding, keeping the raw number available on expand

We chose option 3. The mastery level thresholds are defined as named constants in get_kc_states.py:

_MASTERY_DEVELOPING = 0.40
_MASTERY_MASTERED   = 0.80

def _mastery_level(p_mastery: float) -> str:
    if p_mastery >= _MASTERY_MASTERED:
        return "Mastered"
    if p_mastery >= _MASTERY_DEVELOPING:
        return "Developing"
    return "Novice"

One deliberate UX choice: a student with no attempts at all still sees all 5 skills at 0% / Novice. There's no "no data yet" placeholder. The teacher sees the full KC grid from day one — an empty bar is information ("this student hasn't encountered this skill yet"), not an error.

The access control pattern is worth noting too. We added a require_authenticated dependency — any valid JWT — and enforced role logic in the route handler:

@router.get("/{student_id}/kc-states", response_model=KCStatesResponse)
async def get_kc_states(
    student_id: uuid.UUID,
    db: AsyncSession = Depends(get_db),
    auth: dict = Depends(require_authenticated),
) -> KCStatesResponse:
    role = auth.get("role")
    if role == "student" and auth.get("sub") != str(student_id):
        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Access denied")
    ...

Students see their own KC states. Teachers see any student's. The rule lives in one place — the route — rather than being split across two separate dependency functions.

Why It Matters for the Research

The MacLellan ITS framework's "Teacher-in-the-Loop" principle isn't just about giving teachers a screen. It's about giving them information they can act on. A 43% accuracy number tells a teacher "this student is struggling." A KC panel that shows SUB_BORROW at 12% (Novice, 8 attempts) while PLACE_VALUE is at 67% (Developing, 14 attempts) tells a teacher "this student needs targeted borrowing practice — and they've already tried eight times, so hints aren't landing."

That's the difference between a reporting tool and a teaching tool. The RCT we're designing in Phase 4 will measure whether teachers who have KC-level visibility actually intervene differently than those who see accuracy alone. This dashboard is the instrument we're studying, not just a convenience feature.

What We Learned

The left-join strategy — two separate queries plus a dict lookup — turned out to be cleaner than an ORM outerjoin(). SQLAlchemy async outerjoin() with nullable columns requires explicit handling of None values in ways that are easy to get wrong. Two queries and a dict.get() with a default is more readable and easier to test with mocks:

kc_by_skill_id = {record.skill_id: record for record in kc_records}

summaries = [
    KCStateSummary(
        skill_code=skill.code,
        p_mastery=round(kc_by_skill_id[skill.id].p_mastery, 3)
        if skill.id in kc_by_skill_id else 0.0,
        ...
    )
    for skill in all_skills
]

Nine unit tests covering the use case ran in 0.03s with no live database. That's the payoff for keeping the domain logic in a use case rather than inline in the route.

What's Next

Phase 2 of the KC dashboard adds recent attempt history to the student detail panel — the specific problems a student got wrong, with their classified mistake codes, so a teacher can see patterns as they form.

Key Takeaways

Accuracy is output, KC mastery is signal — a single accuracy number is not enough for a teacher to act; per-KC mastery state is the minimum viable explainability for an ITS
Empty is informative, not broken — showing all KCs at 0% for a new student tells a teacher "this skill hasn't been practised yet"; hiding it implies the data is missing
Two queries + dict > one complex join — for small, static reference data (5 skills), two simple queries and a dict lookup are more readable, testable, and maintainable than an ORM outer join with nullable column handling

Closing the feedback loop: how mistake classification drives adaptive problem selection in NumPath

Oscar Rieken — Thu, 21 May 2026 03:50:28 +0000

What We Built

NumPath is an AI math tutor for children with dyscalculia. At its core is an adaptive engine that picks the next problem for each student based on their Bayesian Knowledge Tracing (BKT) mastery estimate. In this post I'll walk through a problem we had — and solved — in the rule-based phase: classified mistakes were being logged but completely ignored by the selection engine.

The fix was a 60-line change across two files. The research implication is significant.

The Problem: a diagnostic signal going nowhere

Our MistakeClassifier already tagged every wrong answer with a structured code — BORROW_SKIP when a student adds instead of subtracts with borrowing, DIGIT_REVERSAL when they write 51 for 15, MAGNITUDE_MISJUDGE when they pick the smaller number as larger. These MistakeEvent records were hitting the database on every incorrect attempt.

But GetNextProblemUseCase — the code that decides what problem a student gets next — never read them. The engine was selecting problems purely on BKT p_mastery. A student could hit BORROW_SKIP three sessions in a row and still receive problems at the same difficulty, on the same skill, with zero response to the pattern.

This violates what MacLellan et al. call the "Error as Diagnostic Signal" principle: mistakes should trigger targeted remediation, not generic retry.

The Design Decision

The core question was: when should a mistake pattern trigger a response, and what should that response be?

We settled on three rules, each encoded as a named constant:

MISTAKE_WINDOW = 3        # look back this many MistakeEvents
# threshold = ceil(MISTAKE_WINDOW / 2) = 2 — dominant code must appear ≥ 2× in window

MISTAKE_KC_MAP = {
    "DIGIT_REVERSAL":        "PLACE_VALUE",
    "BORROW_SKIP":           "SUB_BORROW",
    "MAGNITUDE_MISJUDGE":    "PLACE_VALUE",
    "PLACE_VALUE_CONFUSION": "PLACE_VALUE",
    "OPERATION_CONFUSION":   "OPERATION_SIGN",
}

When _detect_mistake_signal() fires, two things happen:

Skill override — the engine targets the KC linked to that mistake code, even if another KC has lower p_mastery.
Difficulty drop — one DIFFICULTY_STEP (0.2) down, floored at ENTRY_DIFFICULTY (0.3) to prevent over-scaffolding students who are already at entry level.

What we explicitly rejected: resetting difficulty to zero (too harsh for students who've been making progress), and weighting by mistake severity (too complex for Phase 1 with no real data to calibrate against).

The reason field on every NextProblemResponse now names the triggering pattern:

"Remediation: BORROW_SKIP detected 2× on SUB_BORROW (p_mastery=0.41)"

This is the explainability requirement. A teacher looking at this in the dashboard can understand exactly why the system chose what it did.

Why It Matters for the Research

The central claim of NumPath's RCT will be that adaptive, mistake-aware tutoring produces better outcomes than static worksheets for dyscalculic learners. Before this change, we had a system that adapted difficulty based on streaks but ignored the type of error a student was making. That's not meaningfully different from a worksheet that repeats problems when you get them wrong.

Closing this loop — mistake code → KC target → difficulty adjustment → reason field — is what makes the system an Intelligent Tutoring System rather than a difficulty slider. Every MistakeEvent record is now a longitudinal data point that shapes the student's next experience, and that chain of causality is fully traceable.

What We Learned

The implementation was straightforward. The harder question was the threshold: why 2 of 3, not 3 of 3? Three-of-three is too strict — a student who makes BORROW_SKIP, then DIGIT_REVERSAL, then BORROW_SKIP again has a clear pattern but the strict threshold misses it. Two-of-three catches the pattern earlier at the cost of occasional false positives. We don't yet have real student data to validate this choice — it's a hypothesis. We've logged it as a research note for Phase 4.

The one thing I'd do differently: add the MistakeEvent index to the model on day one. It was missing and only caught during the performance review pass. A composite index on (student_id, created_at) is obvious in hindsight for any table you're going to query with ORDER BY created_at DESC LIMIT N.

What's Next

Next up: wiring the KC states into the teacher dashboard so educators can see p_mastery per student, not just 7-day accuracy — the final piece of the MacLellan "Teacher-in-the-Loop" principle.

Key Takeaways

Logging errors is not the same as learning from them — a diagnostic signal only matters if it changes what happens next; wiring MistakeEvent into select_next_problem() is a 60-line change with a meaningful research impact
The reason field is not a nice-to-have — every adaptive decision must be explainable to a teacher; string-formatted rationale on each NextProblemResponse is the minimum viable explainability
Named constants beat magic numbers — MISTAKE_WINDOW, FRUSTRATION_WINDOW, MASTERY_WINDOW sit side by side; when we have real data to calibrate thresholds, we change one line each

Why I'm building an AI math tutor for dyscalculia — and grounding it in 30 years of ITS research

Oscar Rieken — Thu, 21 May 2026 03:28:14 +0000

The problem I kept seeing

When I started this research I expected dyscalculia to be a niche condition — something a handful of children had, easily addressed with extra practice. The numbers don't support that. Roughly 5–7% of school-age children have dyscalculia: a specific learning difficulty with number sense that is neurological in origin, persistent across development, and largely invisible in standard classroom assessments.

It's not "bad at maths." A child with dyscalculia might have strong reading comprehension, solid spatial reasoning, and consistently fail to grasp that 9 comes before 10. The difficulty is specific, categorical, and resistant to the kind of general maths instruction classrooms provide. Standard adaptive apps — the ones with stars and progress bars — don't help much because they adapt difficulty without adapting to the mechanism of the difficulty. A child who confuses 51 for 15 (digit reversal) needs something different from a child who skips borrowing. Treating both as "got it wrong, try again" misses the point.

This is the gap NumPath is designed to study. Not to solve — to study, rigorously, in a randomised controlled trial.

Why ITS research, not a product

Intelligent Tutoring Systems have a 30-year research literature, and the core insights from it are not widely implemented in consumer software. That's partly because the research is paywalled, partly because it's written for academics rather than engineers, and partly because implementing it properly requires more architecture than a typical edtech MVP.

NumPath is built on four distinct research contributions. I want to be precise about attribution here because I got it wrong early on:

1. The Knowledge Component (KC) model

Anderson & Corbett (1992); Koedinger & Aleven (2007)

A KC is the smallest unit of knowledge required to complete one step in a problem. Not "subtraction" — that's too coarse. Not "subtract 178 from 345 by borrowing from the tens column, carrying 1, and writing the result in the ones column" — that's too specific. A KC is something like SUB_BORROW: the procedure of regrouping when subtracting multi-digit numbers.

In NumPath every skill in the database is a KC:

SKILLS = [
    {"code": "SUB_BORROW",    "name": "Subtraction with borrowing"},
    {"code": "PLACE_VALUE",   "name": "Place value understanding"},
    {"code": "NUMBER_LINE",   "name": "Number line navigation"},
    {"code": "NUMBER_SENSE",  "name": "Basic number magnitude and ordering"},
    {"code": "OPERATION_SIGN","name": "Reading and applying operation signs"},
]

Each student has an independent mastery estimate for each KC. Progress on SUB_BORROW doesn't transfer to PLACE_VALUE. A student doesn't "advance past subtraction" — they master individual KCs, one at a time.

2. Bayesian Knowledge Tracing (BKT)

Corbett & Anderson (1995)

BKT is the probabilistic model that estimates whether a student has mastered a KC, updated on every attempt. Four parameters per student per KC:

p_mastery — P(student has learned this KC)
p_learn — P(learning occurs on this attempt, given not yet learned)
p_guess — P(correct answer given KC not learned)
p_slip — P(incorrect answer given KC is learned)

Update equations after observing an answer:

# After a correct answer:
posterior = p * (1 - slip) / (p * (1 - slip) + (1 - p) * guess)
# After an incorrect answer:
posterior = p * slip / (p * slip + (1 - p) * (1 - guess))
# Learning update (applied after either):
p_new = posterior + (1 - posterior) * p_learn

This is not MacLellan's work — it predates his research by ~20 years. I got the attribution wrong in early drafts and I've since corrected it throughout the codebase. BKT is Corbett & Anderson.

3. The Apprentice Learner (AL) Architecture

MacLellan, Harpstead, Patel & Koedinger — EDM 2016 (Exemplary Paper Award)

BKT predicts whether a student knows something. The AL Architecture models how they acquire it. MacLellan distinguishes three learning mechanisms:

How-learning — generalising the procedure from examples
Where-learning — learning which contexts a skill applies to
When-learning — learning the conditions that trigger the skill

This is the conceptual basis for NumPath's mistake classifier. BORROW_SKIP (student applies subtraction but skips the regrouping step) is a how-learning failure — the student hasn't generalised the full procedure. OPERATION_CONFUSION (student adds when the problem requires subtraction) is a when-learning failure — the student fires the wrong skill given the sign.

Classifying mistakes this way means the tutor can respond differently: how-learning failures need procedural scaffolding; when-learning failures need context discrimination practice.

4. The Natural Training Interaction (NTI) Framework

MacLellan, Harpstead & Koedinger — AAAI 2018 Spring Symposium

This is MacLellan's most directly applicable contribution to NumPath's teacher dashboard. The NTI Framework treats teachers as first-class participants in the adaptive loop, not passive viewers of a report. Concretely, it means:

Every AI-generated insight must cite specific evidence — a KC code, a p_mastery value, a mistake count. No generic "this student needs more practice."
Teachers must be able to confirm or override AI judgments (their feedback improves the system over time)
The system must support natural conversation between AI output and teacher judgment

In NumPath this shows up in the insight response shape:

{
  "text": "Aiden skips borrowing in 9 of 11 recent subtraction attempts.",
  "type": "warn",
  "evidence": {
    "kc": "SUB_BORROW",
    "p_mastery": 0.18,
    "mistake_type": "BORROW_SKIP",
    "mistake_count": 9,
    "window": "last 11 attempts"
  }
}

The evidence block isn't generated by the LLM — it's assembled from DB reads before the prompt is sent. Explainability is structural, not hoped for.

What NumPath actually is

A FastAPI + Vue 3 web application, deployed with Docker Compose. Students practice math problems; the BKT model updates on every attempt; the adaptive engine selects the next problem based on KC mastery states and recent mistake patterns. Teachers see a dashboard showing per-student KC progress, attempt history, and LLM-generated insights.

The four-phase roadmap:

Phase	Focus	Status
1	MVP: BKT + adaptive engine + teacher dashboard	In progress
2	Mistake classifier v2 + DKT model	Planned
3	RCT instrumentation	Planned
4	Randomised controlled trial	Planned

The RCT will compare learning outcomes for students using NumPath against a control group using standard worksheet practice. The teacher dashboard is an instrument in that experiment, not just a feature.

What this series covers

Four posts are already written, each covering a specific engineering decision:

Closing the feedback loop — how we wired mistake classification into adaptive problem selection
Why teachers need explainable AI — building the KC mastery dashboard (Phase 1)
Attempt history and the scalar subquery pattern — paginated attempt history and a safe LEFT JOIN alternative
Prompt engineering for teacher insights — structured JSON output and graceful fallbacks with Claude

All posts are first-person engineering notes. The code is real, the trade-offs are real, and I'll be honest when something didn't work as expected.

What's Next

Phase 1 is nearly feature-complete. The next milestone is running the full test suite on live data with a small pilot group before the RCT design begins.

Key Takeaways

Dyscalculia is specific, not general — "more practice" is the wrong intervention; KC-level mastery tracking is the minimum viable response
The ITS research literature has four distinct pillars, not one — KC model (1992), BKT (1995), AL Architecture (2016), NTI Framework (2018) — each does a different job and attribution matters
Explainability must be structural — an insight that cites a specific KC and mistake count is auditable; one that says "this student is struggling" is not