Building a Learner Model: Bayesian Knowledge Tracing in Production

If you are building a learning platform and want it to do anything more interesting than serving videos in order, you need a learner model. This post is about implementing one without the academic detour.

The job: maintain, per learner per concept, a probability that they have mastered it. Update on evidence. Decay over time.

Step 1 — The knowledge graph

Before any modelling, you need a structure to model over.

concept(
  id, name, domain,
  description
)

concept_prerequisite(
  concept_id, prerequisite_concept_id,
  strength          -- 0..1, how hard the dependency is
)

item_concept(
  item_id, concept_id,
  weight            -- how much this item evidences this concept
)

Two practical notes.

Granularity is the hard call. Too coarse and the model cannot discriminate; too fine and you never gather enough evidence per node. A workable heuristic is that a concept should be something a learner could plausibly master in fifteen to forty minutes, and something you could write three to five distinct assessment items for.

This requires domain experts and cannot be rushed. Budget weeks, not days, and expect the graph to change as evidence accumulates. It is the most valuable artefact in the system and the one you cannot generate your way out of.

Step 2 — BKT, concretely

Classical Bayesian knowledge tracing carries four parameters per concept:

• p_init — probability of mastery before any evidence
• p_learn — probability of transitioning to mastery per opportunity
• p_guess — probability of correct answer without mastery
• p_slip — probability of incorrect answer despite mastery

The update on observing a correct response:

posterior = (prior * (1 - p_slip)) /
            (prior * (1 - p_slip) + (1 - prior) * p_guess)

On an incorrect response:

posterior = (prior * p_slip) /
            (prior * p_slip + (1 - prior) * (1 - p_guess))

Then apply the learning transition:

p_mastery = posterior + (1 - posterior) * p_learn

Parameter fitting is expectation-maximisation over historical response data. Before you have data, seed from reasonable defaults (p_guess = 1/number of options for multiple choice, p_slip around 0.1) and refit once you have a few thousand responses per concept.

Step 3 — Propagation

Evidence about one concept says something about its neighbours. A learner demonstrating mastery of a concept provides weak positive evidence for its prerequisites.

Keep this simple and damped:

for prereq in prerequisites(concept):
    delta = (posterior - prior) * strength * DAMPING
    p_mastery[prereq] = clamp(p_mastery[prereq] + delta, 0, 1)

with DAMPING around 0.3 and propagation depth limited to one or two hops. Undamped propagation over a dense graph produces mastery estimates that drift upward across the entire domain on very little evidence, which is both wrong and embarrassing when a learner notices.

Step 4 — Decay, which everyone skips

Without a forgetting term, your model will assert mastery of something a learner touched once nine months ago.

An exponential forgetting curve applied lazily at read time is sufficient and avoids a background job over every learner-concept pair:

def current_mastery(record, now):
    elapsed = now - record.last_evidence_at
    half_life = base_half_life * (1 + record.reinforcement_count * 0.5)
    decayed = record.p_mastery * 0.5 ** (elapsed / half_life)
    return max(decayed, record.p_floor)

Two details matter. Half-life extends with each successful reinforcement — this is the spacing effect and it is the mechanism behind spaced repetition scheduling. And a floor prevents mastery decaying to zero, since a learner who genuinely learned something retains more than nothing indefinitely.

Step 5 — Event sourcing, non-negotiable

This is the architectural decision that people regret skipping.

Store every interaction as an immutable event. The learner model is a projection over that stream, not the source of truth.

learning_event(
  id, learner_id, occurred_at,
  event_type,        -- item_response | tutor_exchange | content_view
  concept_ids[],
  payload jsonb
)

The reason is straightforward: your model will improve. You will refit parameters, change the propagation rule, add decay, switch from BKT to a sequence model. If current state is your only record, every improvement applies to future learners only, and your existing cohort keeps the old estimates forever.

With an event log, a model change is a recomputation. Replay the stream, rebuild the projection, and every learner benefits retroactively.

This also happens to make xAPI emission nearly free, since the actor-verb-object structure maps directly onto the event shape, and enterprise buyers increasingly require the ability to pipe learning events into their own warehouse.

Step 6 — Reading from it

Two query patterns dominate.

Next activity selection. Find concepts where the learner has satisfied prerequisites (all prereqs above a mastery threshold) but the concept itself sits below mastery, and prefer items whose difficulty puts predicted success probability in the 0.6–0.8 band.

Review scheduling. Find concepts whose decayed mastery has dropped below a review threshold, ordered by how far below. This is your spaced repetition queue and it falls out of the decay function for free.

A note on interleaving: mixing concepts rather than blocking them produces better long-term retention and learners consistently rate it as less effective. If you tune sequencing on satisfaction metrics, you will drift toward blocked practice and worse outcomes. Decide this deliberately.

What this costs

For a single domain: two to three weeks with domain experts for the graph, two to three weeks for the model implementation and parameter fitting harness, and another two for the sequencing layer. Call it two months with a small team, assuming the event infrastructure exists.

That is the component that makes outcome claims defensible, which is the difference between selling per seat and selling against a business result.

Full architecture guide including tutor guardrails, assessment design, standards and cost breakdown: LMS Development in 2026: Architecting a Learning Platform Around AI. Related: our SaaS development work.

Frequently Asked Questions

What is Bayesian knowledge tracing?

A model that maintains a probability of concept mastery per learner, updated on each observed response using four parameters: prior mastery, learning rate, guess probability and slip probability. It is the standard workhorse for learner modelling and is straightforward to implement.

How granular should concepts in a knowledge graph be?

A workable heuristic is that a concept should be masterable in fifteen to forty minutes and should support three to five distinct assessment items. Too coarse and the model cannot discriminate; too fine and you never accumulate enough evidence per node.

Why does the model need a forgetting term?

Without decay, the system asserts mastery of concepts a learner encountered once months ago and never revisited, making progress reporting dishonest. An exponential curve applied lazily at read time is sufficient, with half-life extending on each successful reinforcement.

Why is event sourcing necessary for a learner model?

Because the model will improve — refit parameters, new propagation rules, different algorithms. If current state is the only record, improvements apply to future learners only. With an event log, a model change is a recomputation that benefits everyone retroactively.

How do you avoid mastery estimates drifting upward across a domain?

Damp the propagation to prerequisites (around 0.3) and limit depth to one or two hops. Undamped propagation over a dense graph inflates estimates across the whole domain on very little evidence.

How long does building a learner model take?

For a single domain, roughly two months with a small team: two to three weeks with domain experts on the knowledge graph, two to three weeks on the model and parameter fitting, and two on the sequencing layer — assuming event infrastructure already exists.