DEV Community: Oscar Rieken

Making LLM outputs auditable: the provider abstraction pattern

Oscar Rieken — Mon, 01 Jun 2026 02:28:08 +0000

The problem with calling an LLM directly

NumPath's teacher dashboard generates per-student insights — one-sentence observations like "Emma skips borrowing in 9 of 11 recent subtraction attempts" with a suggested action. The obvious implementation is to import the Anthropic SDK, call messages.create(), and return the result.

That works until you need to test it. Or run it offline. Or swap providers. Or audit where the insight came from.

This post covers how NumPath abstracts the LLM behind a protocol interface, tests with a deterministic stub, and structures the insight pipeline so the evidence is assembled from database reads — not generated by the model.

The Protocol: 6 lines

The entire LLM abstraction is a Python Protocol:

from typing import Protocol, runtime_checkable

@runtime_checkable
class LLMProvider(Protocol):
    async def complete(self, system: str, user: str, max_tokens: int = 256) -> str: ...

No base class. No ABC. No framework. Any object with an async def complete(self, system, user, max_tokens) method satisfies this interface — that's structural typing via Protocol. The @runtime_checkable decorator lets you write isinstance(provider, LLMProvider) if you need a runtime check, though in practice the type checker catches mismatches at lint time.

The signature is deliberately narrow: one system prompt, one user message, one token limit. No conversation history, no tool use, no streaming. NumPath's insight generator makes a single completion call per request. If multi-turn conversation becomes necessary in Phase 3, the protocol gains a new method — existing implementations aren't broken.

Two implementations

ClaudeProvider — the production implementation:

class ClaudeProvider:
    def __init__(self) -> None:
        self._client = anthropic.AsyncAnthropic(api_key=settings.ANTHROPIC_API_KEY)

    async def complete(self, system: str, user: str, max_tokens: int = 256) -> str:
        message = await self._client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=max_tokens,
            system=system,
            messages=[{"role": "user", "content": user}],
        )
        return message.content[0].text

StubProvider — deterministic, zero dependencies, zero API calls:

class StubProvider:
    """Deterministic LLM stub for tests and local dev without API keys."""

    async def complete(self, system: str, user: str, max_tokens: int = 256) -> str:
        return (
            '{"summary": "Student is building foundational numeracy skills '
            'with consistent effort.", "suggested_action": "Try place value '
            'exercises with physical manipulatives to reinforce digit positioning."}'
        )

The stub returns a fixed JSON string that matches the expected response schema. Tests assert against this exact output. If someone changes the response schema, the stub breaks, the tests break, and the problem is caught before deployment.

Wiring: one environment variable

def get_llm_provider() -> LLMProvider:
    if settings.LLM_PROVIDER == "claude":
        return ClaudeProvider()
    return StubProvider()

LLM_PROVIDER defaults to "stub". Running uv run pytest requires zero environment variables — no API key, no network. Production sets LLM_PROVIDER=claude and provides ANTHROPIC_API_KEY. The config uses Literal["claude", "stub"] so a typo like "Claude" fails at startup.

The use case receives the provider through its constructor, not through a global:

class GenerateInsightUseCase:
    def __init__(self, db: AsyncSession, llm: LLMProvider) -> None:
        self._db = db
        self._llm = llm

The router wires it:

@router.get("/students/{student_id}/insight", response_model=InsightResponse)
async def get_student_insight(
    student_id: uuid.UUID,
    db: AsyncSession = Depends(get_db),
    _: dict = Depends(require_teacher),
) -> InsightResponse:
    llm = get_llm_provider()
    use_case = GenerateInsightUseCase(db, llm)
    return await use_case.execute(student_id)

Evidence is not generated — it's assembled

This is the design decision that matters most for a research project. When a teacher sees an insight, they need to trust it — and "trust" in an educational context means "I can check this against the data."

The insight prompt receives two blocks of structured data, both assembled from database queries:

KC states:
- SUB_BORROW: Novice (p_mastery=0.18, 8 attempts)
- PLACE_VALUE: Developing (p_mastery=0.45, 3 attempts)
- NUMBER_LINE: Novice (p_mastery=0.15, 1 attempt)

Recent attempts (last 10, most recent first):
1. Skill: SUB_BORROW | Correct: No | Mistake: BORROW_SKIP | Q: "52 − 27 = ?"
2. Skill: SUB_BORROW | Correct: No | Mistake: BORROW_SKIP | Q: "31 − 14 = ?"
3. Skill: PLACE_VALUE | Correct: Yes | Mistake: none | Q: "Which is larger: 47 or 74?"

The LLM generates two fields: summary (what's happening) and suggested_action (what to do). It does not generate the evidence — the KC codes, mastery percentages, mistake counts, and attempt records are all server-side data. The LLM synthesises a narrative from that data, but the data itself is verifiable.

The prompt enforces this structurally:

You are a specialist math learning advisor for primary school teachers.
Given their Knowledge Component mastery states and recent attempt history,
generate a JSON response with exactly two fields:
- "summary": one sentence (max 20 words) describing the student's current learning state
- "suggested_action": one concrete teaching action (max 20 words) the teacher can take today

Respond with only the JSON object. No explanation, no markdown, no code fences.

Strict JSON. Word limits. No room for hallucinated statistics or invented KC codes.

Graceful fallback

LLMs produce unpredictable output. The response parser handles malformed JSON without crashing:

_FALLBACK_INSIGHT = InsightResponse(
    summary="Insight temporarily unavailable.",
    suggested_action="Review the student's recent attempts for patterns.",
)

def _parse_insight(raw: str) -> InsightResponse:
    try:
        data = json.loads(raw)
        return InsightResponse(
            summary=data["summary"],
            suggested_action=data["suggested_action"],
        )
    except (json.JSONDecodeError, KeyError, TypeError):
        logger.warning("insight_parse_failed_using_fallback raw=%s", raw[:200])
        return _FALLBACK_INSIGHT

The fallback is a valid InsightResponse — the teacher sees a neutral message, not a 500 error. The warning log captures the first 200 characters of the raw response for debugging without logging the entire LLM output.

Why not LangChain?

This was an explicit decision, documented in ADR-003. LangChain adds 50+ transitive dependencies and significant abstraction cost for what NumPath actually needs: one completion call with a system prompt and a user message. The protocol-based approach is 6 lines of interface, 8 lines of stub, 9 lines of production implementation. The total abstraction surface is smaller than LangChain's ChatModel base class alone.

If NumPath needed retrieval-augmented generation, multi-step chains, or agent loops, LangChain would earn its weight. For two structured completion calls (insight generation and hint narration), it would be accidental complexity.

The fitness function

ADR-003 specifies a concrete test: uv run pytest must pass using StubProvider with no environment variables set. This means every LLM-dependent code path has a test that runs offline. If someone adds a new LLM feature and writes a test that requires ANTHROPIC_API_KEY, CI fails — not because the test is wrong, but because it violates the architectural constraint that the test suite runs without external dependencies.

What's next

The current provider interface handles single-turn completions. Phase 3 may need multi-turn conversation for interactive teacher coaching. When that happens, the protocol gains a second method — complete() stays unchanged, and a new converse() method handles the multi-turn case. Existing implementations get a NotImplementedError default until they're updated. The key is that the interface extends forward without breaking backward.

Key Takeaways

Protocol-based abstraction costs 6 lines and buys full test isolation — StubProvider returns deterministic output; no API key, no network, no flaky tests; the type checker enforces the contract at lint time
Evidence must be assembled from data, not generated by the model — the LLM writes the narrative but doesn't produce the numbers; KC codes, mastery percentages, and mistake counts come from database queries and are independently verifiable
Graceful fallback is a first-class design requirement — a teacher sees "insight temporarily unavailable" and a neutral suggestion, never a stack trace; the warning log captures the raw output for debugging without exposing it to the user

60 hand-crafted math problems: what I learned writing seed data for an adaptive tutor

Oscar Rieken — Mon, 01 Jun 2026 02:27:18 +0000

Why hand-author anything?

The obvious approach for seeding an adaptive math tutor is to generate problems programmatically. Pick two random numbers, subtract them, done. I tried this first and it failed for a specific reason: generated problems don't have meaningful hints.

A hint like "Try subtracting the ones column first" is generic. A hint like "2 ones minus 9 is impossible without borrowing — take a ten from the 3 tens" is diagnostic. It names the exact step where a dyscalculic student is likely to get stuck, and it names the operation they need to perform. That second kind of hint requires a human who understands the problem.

NumPath's Phase 1 seeds 100 problems across 5 Knowledge Components, each with two progressive hints, a calibrated difficulty score, and structured metadata that the MistakeClassifier uses to diagnose errors. Every one is hand-authored.

The content schema

Each problem is a JSONB column in Postgres. The schema is intentionally flat — no nested objects, no polymorphism:

{
    "type": "subtraction",
    "question": "32 − 9 = ?",
    "answer": "23",
    "difficulty": 0.25,
    "operands": [32, 9],
    "hints": [
        "2 ones − 9 is impossible without borrowing. Take a ten from the 3 tens.",
        "Now you have 12 ones. 12 − 9 = 3. You have 2 tens left. Answer: 23.",
    ],
}

Three fields deserve explanation:

operands / choices — these aren't shown to the student. They exist for the MistakeClassifier. When a student answers "41" instead of "23" on a subtraction problem, the classifier checks whether the answer matches subtracting the digits in the wrong direction (3 - 2 = 1, 9 - 0 = 9 → 91... no). It checks whether the answer omits borrowing (32 - 9 without regrouping gives 33... no). It checks for digit reversal (23 → 32... close, but the student wrote 41). Each check operates on the operands, not the question string.

difficulty — a float from 0.1 to 0.9, calibrated by hand. This is the initial difficulty estimate. The adaptive engine uses it to match students to problems at their current level. I'll explain the calibration logic below.

hints — always exactly two, always progressive. The first hint names the obstacle. The second hint walks through the solution. Students reveal hints one at a time, voluntarily. Hints are never forced — forcing hints on students who don't want them creates learned helplessness, which is the opposite of what we're trying to study.

The five skill areas

Each skill has 20 problems covering a difficulty gradient from 0.1 to 0.9:

Skill Code	Domain	Example at 0.1	Example at 0.9
`SUB_BORROW`	Subtraction	11 − 4 = ?	1003 − 567 = ?
`PLACE_VALUE`	Number sense	Which is larger: 3 or 8?	What does the 6 represent in 3,641?
`NUMBER_LINE`	Number sense	What number comes after 3?	What is halfway between 250 and 350?
`NUMBER_SENSE`	Number sense	Which is more: 2 or 5?	Order from smallest: 892, 829, 928, 289
`OPERATION_SIGN`	Arithmetic	2 + 3 = ?	15 − 7 + 3 = ?

The difficulty gradient is not linear. The jump from 0.1 to 0.3 (single-digit to simple two-digit) is smaller than the jump from 0.7 to 0.9 (two-digit with borrowing across zeros to three-digit with cascading borrows). This mirrors what the dyscalculia research literature reports: difficulty is not proportional to number size. It's proportional to the number of cognitive steps, particularly steps that require regrouping or holding intermediate results in working memory.

Hint design: what I got wrong

My first draft of hints was procedural — they described what to do:

"Borrow from the tens column. Subtract. Write the answer."

This is useless for a student with dyscalculia. The difficulty isn't knowing what borrowing is — it's executing the procedure without losing track of which column they're in. The second draft of every hint follows two rules:

Name the specific obstacle. Not "this is tricky" — rather "2 ones minus 9 is impossible without borrowing."
Walk through the state change. Not "borrow and subtract" — rather "Take a ten from the 3 tens. Now you have 12 ones. 12 − 9 = 3. You have 2 tens left."

The second rule matters because dyscalculic students often lose the intermediate state — they borrow correctly but then forget what changed. The hint reconstructs the full number after regrouping so the student can see where they are.

This pattern held across all five skill areas. Place value hints name the specific column ("the tens digit is the second from the right"). Number line hints name the direction and distance ("7 is to the right of 4 — count 3 steps forward"). Operation sign hints name the symbols and their meaning ("the − sign means subtract — take the second number away from the first").

Difficulty calibration

Difficulty scores are not arbitrary. They follow a rubric I developed after the first round of testing:

Score range	Criteria
0.1 – 0.2	Single-digit or simple two-digit; one cognitive step
0.25 – 0.4	Two-digit; requires one borrowing or comparison step
0.45 – 0.6	Two-digit with borrowing across columns, or three-digit without borrowing
0.65 – 0.8	Three-digit with borrowing; or problems requiring intermediate computation
0.85 – 0.9	Three-digit with cascading borrows (e.g., borrowing from hundreds when tens is 0)

The adaptive engine uses a DIFFICULTY_BAND of 0.15 around the target difficulty when selecting problems. So a student at target difficulty 0.5 sees problems between 0.35 and 0.65. This means each difficulty tier overlaps with its neighbors — a student improving from 0.4 to 0.6 transitions gradually rather than hitting a cliff.

The seed script

The seed is idempotent — safe to run on every deployment:

async def seed_problems(session, skill_id_map: dict[str, str]) -> int:
    for skill_code, problems in PROBLEMS.items():
        skill_id = skill_id_map.get(skill_code)
        for p in problems:
            content = {k: v for k, v in p.items() if k != "difficulty"}
            stmt = (
                pg_insert(Problem)
                .values(
                    skill_id=skill_id,
                    content=content,
                    difficulty=p["difficulty"],
                    problem_type=p["type"],
                )
                .on_conflict_do_nothing()
            )
            await session.execute(stmt)

on_conflict_do_nothing() means re-running the seed doesn't duplicate problems. The difficulty field is stored both inside the JSONB content and as a top-level column on the Problem model — the column is indexed for the adaptive engine's range queries, while the JSONB copy preserves the original specification.

The full seed runs inside a single transaction: skills first (because problems have a foreign key to skills), then problems, then test accounts. If any step fails, nothing is committed.

What I'd do differently

Two things:

More problems per skill. Twenty problems with a 0.15 difficulty band means some bands have only 2–3 candidates. When the adaptive engine excludes recently-seen problems, it can run out of fresh options at a specific difficulty level. The fallback chain handles this gracefully (widen the band, then allow repeats), but 30 problems per skill would eliminate most fallback cases.

Machine-assisted hint generation. The hints are the bottleneck — each one took 2–3 minutes to write well. For Phase 2, I plan to generate candidate hints with Claude and then manually review them. The human is still in the loop, but the first draft comes faster.

Key Takeaways

Generated problems are easy; generated hints are not — an adaptive tutor's value is in the scaffolding, not the arithmetic; hand-authoring hints that name the specific obstacle and walk through the state change is what makes the system useful for dyscalculia
Difficulty is not proportional to number size — it's proportional to cognitive steps, particularly regrouping and intermediate state; a three-digit problem with no borrowing (350 − 120) is easier than a two-digit problem with cascading borrows (100 − 67)
Idempotent seeds inside a transaction are non-negotiable — on_conflict_do_nothing() plus a single transaction means the seed runs safely on every deployment, fresh clone, and CI pipeline

Clean Architecture in a FastAPI + Vue 3 monorepo

Oscar Rieken — Mon, 01 Jun 2026 02:26:44 +0000

Why architecture matters in a research project

Most research prototypes are throwaway code. NumPath is not. It needs to survive four phases over 30 weeks, accumulate real student data for a randomised controlled trial, and remain testable without live infrastructure at every step. That means the architecture has to enforce rules that hold up under pressure — not just conventions someone remembers to follow.

This post walks through how NumPath uses Clean Architecture to keep a FastAPI backend, a Vue 3 frontend, and a Python ML module in a single repository without coupling them together.

The monorepo layout

The project lives in a single repo with a clear namespace boundary:

phd-research/
├── numpath/
│   ├── backend/      # Python 3.12 + FastAPI + SQLAlchemy
│   ├── frontend/     # Vue 3 + Tailwind CSS + Pinia
│   └── ml/           # BKT, DKT, adaptive engine
├── docs/
│   ├── adrs/         # Architecture Decision Records
│   ├── architecture/ # System design, feature specs
│   └── posts/        # This blog series
└── DOMAIN_DICTIONARY.md

The alternative was three separate repos. For a solo PhD project where a data model change touches the migration, the API schema, the ML engine, and the Vue component in the same commit, separate repos mean coordinated PRs across three remotes. That's overhead with no benefit when one person owns all three layers.

The escape hatch is clean: if NumPath ever needs to become a standalone repo, the numpath/ directory lifts out intact.

Dependency direction: the one rule that matters

Clean Architecture has many principles, but only one that I enforce mechanically: inner layers never import from outer layers.

In NumPath's backend, the layers are:

Domain (models)  →  Use Cases  →  Adapters (routers, DB, LLM)  →  Frameworks (FastAPI, SQLAlchemy)

A use case like GetNextProblemUseCase receives a database session — but it does not import FastAPI, does not know about HTTP, and does not call Depends():

class GetNextProblemUseCase:
    def __init__(self, db: AsyncSession) -> None:
        self._db = db

    async def execute(self, student_id: uuid.UUID) -> NextProblemResponse:
        kc_states = await self._build_kc_states(student_id)
        recent_attempts = await self._fetch_recent_attempts(student_id)
        recent_mistakes = await self._fetch_recent_mistakes(student_id)

        selection: ProblemSelection = select_next_problem(
            kc_states=kc_states,
            recent_correctness=[row.is_correct for row in recent_attempts],
            current_difficulty=recent_attempts[0].difficulty if recent_attempts else 0.3,
            recent_mistakes=recent_mistakes,
        )

        problem = await self._select_problem(selection, ...)
        # ... return NextProblemResponse

The router — the adapter layer — is the only file that knows about FastAPI:

@router.get("/next-problem/{student_id}", response_model=NextProblemResponse)
async def get_next_problem(
    student_id: uuid.UUID,
    db: AsyncSession = Depends(get_db),
    _: dict = Depends(require_student),
) -> NextProblemResponse:
    use_case = GetNextProblemUseCase(db)
    return await use_case.execute(student_id)

The router does three things: parse the request, inject dependencies, and delegate to the use case. No business logic. If I replaced FastAPI with Litestar tomorrow, I'd rewrite the routers and touch nothing else.

Configuration as a boundary

Settings are another place where framework details leak into domain code if you're not careful. NumPath uses Pydantic's BaseSettings with a .env file:

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", extra="ignore")

    DATABASE_URL: str = "postgresql+asyncpg://numpath:numpath@localhost:5432/numpath"
    LLM_PROVIDER: Literal["claude", "stub"] = "stub"
    ANTHROPIC_API_KEY: str = ""
    ENVIRONMENT: Literal["development", "production", "test"] = "development"
    CORS_ORIGINS: list[str] = ["http://localhost:5173"]

Every secret has a default that works locally. LLM_PROVIDER defaults to "stub" so that tests and local dev never require an API key. The Literal type annotation means a typo in the .env file fails at startup, not at runtime when a teacher clicks "Generate insight."

The ML module as a pure function boundary

The ml/ directory is a separate Python package (numpath-ml) with its own pyproject.toml. The backend depends on it, but the dependency is narrow: two functions and a dataclass.

from numpath_ml.adaptive_engine import select_next_problem, ProblemSelection
from numpath_ml.bkt import KCState

select_next_problem() takes dictionaries and lists — no SQLAlchemy models, no async, no database. It returns a ProblemSelection with a skill_code, target_difficulty, and reason string. The use case translates between database rows and these pure data structures.

This boundary exists because the ML code changes on a different cadence than the web application. When I replace the rule-based engine with Deep Knowledge Tracing in Phase 2, the use case stays the same — only the function it calls changes.

The frontend: same principle, different language

The Vue 3 frontend mirrors the same layering:

API client — a thin Axios wrapper that handles auth tokens and 401 redirects:

export const apiClient = axios.create({
  baseURL: '/api/v1',
})

apiClient.interceptors.request.use((config) => {
  const token = localStorage.getItem('token')
  if (token) config.headers.Authorization = `Bearer ${token}`
  return config
})

Stores — Pinia stores manage state. The auth store handles login/logout and persists the JWT to localStorage. Views consume stores, not the API client directly.

Views — PracticeView.vue, TeacherView.vue, LoginView.vue. Each view composes API calls and store access. No view imports another view.

Router — role-based guards redirect students and teachers to their respective views:

router.beforeEach((to) => {
  const auth = useAuthStore()
  if (to.meta.requiresAuth && !auth.isAuthenticated) return '/login'
  if (to.meta.role && auth.role !== to.meta.role) {
    return auth.role === 'teacher' ? '/teacher' : '/practice'
  }
})

Docker Compose as the integration layer

The four services — Postgres, Redis, backend, frontend — are composed with health checks so the backend waits for the database:

backend:
  environment:
    DATABASE_URL: postgresql+asyncpg://numpath:numpath@postgres:5432/numpath
  depends_on:
    postgres:
      condition: service_healthy
    redis:
      condition: service_healthy
  volumes:
    - ./backend:/app/backend
    - ./ml:/app/ml
  command: uv run uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload

Volume-mounting backend/ and ml/ means hot reload works inside Docker — change a use case, save, and the server restarts. The port mapping (5433:5432 for Postgres) avoids collisions with a local Postgres install.

What this buys you

Three concrete benefits I've already seen:

Test isolation — use cases are testable with a real async database session and no HTTP server. The test creates a GetNextProblemUseCase(db) directly. No FastAPI test client needed for business logic tests.
LLM swappability — GenerateInsightUseCase receives an LLMProvider protocol. In tests it gets StubProvider. In production it gets ClaudeProvider. The use case doesn't know which one it has.
Safe ML replacement — when BKT gives way to DKT, only numpath_ml changes. The use case calls the same select_next_problem() function with the same signature. The router doesn't change. The frontend doesn't change.

What I'd do differently

I wouldn't change the layering, but I'd add one thing from day one: a fitness function that statically checks import direction. Right now the rule is "use cases don't import routers" — but it's enforced by code review (i.e., me reviewing my own code). A linter rule or CI check that fails on from backend.routers inside use_cases/ would catch drift automatically.

Key Takeaways

Dependency direction is the only architectural rule worth enforcing mechanically — inner layers never import outer layers; everything else is convention that erodes under deadline pressure
A monorepo is the right default for a solo research project — coordinated PRs across three repos is overhead without benefit when one person owns all layers and changes cut across them
Pure function boundaries between modules pay for themselves — the ML module exports two functions and a dataclass; the web layer translates between database rows and those pure structures, making the ML code replaceable without touching the application

From Bayesian to deep knowledge tracing — upgrading NumPath's student model with a PyTorch LSTM

Oscar Rieken — Mon, 01 Jun 2026 02:26:00 +0000

BKT told us how well a student knows subtraction-with-borrowing. It had no idea that a student who reverses digits on subtraction problems probably also reverses them on place value problems — because BKT treats every Knowledge Component as an island.

Deep Knowledge Tracing (DKT) fixes that. Instead of four independent scalar parameters per KC, it maintains a shared LSTM hidden vector across all KCs and learns the dependencies from data. This is Phase 3 of NumPath: swapping out the Markov model for a neural sequence model.

Here's what we built, the design decision that almost made us reach for a transformer, and the student simulator we had to build first to test it without any real students.

What We Built

Two components that feed each other:

Student simulator — five named personas that generate realistic attempt sequences for testing. Each persona has a per-KC accuracy curve and weighted mistake preferences drawn from the dyscalculia ITS literature:

Persona	SUB_BORROW accuracy	Characteristic errors
`ConfidentLearner`	0.80	Rare, careless (OFF_BY_TEN)
`StrugglingSUB`	0.35	Frequent BORROW_SKIP, slow timing
`PlaceValueGap`	0.60	DIGIT_REVERSAL across skill areas
`FrustrationLoop`	0.30	Fast random guessing
`FastMaster`	0.90	Near-zero mistakes

DKT model — a single-layer LSTM that takes a sequence of (skill, correctness) interactions and predicts P(correct on skill k) at each subsequent step.

model = DKTModel(n_skills=3, hidden_size=64)
state = model.initial_state()

# Student answers SUB_BORROW correctly
state = model.step(state, skill_idx=0, is_correct=True)

# Query mastery on any KC
p_mastery = model.predict(state, skill_idx=0)  # → float in (0, 1)

The Design Decision

Why not stay with BKT?

BKT's four parameters — p_mastery, p_learn, p_guess, p_slip — are per-KC and independent. A student who has DIGIT_REVERSAL on subtraction problems and DIGIT_REVERSAL on place value problems is modelled as having two unrelated problems. BKT cannot learn that these are the same underlying representational gap.

DKT's hidden state is shared. After the student makes a digit-reversal error on subtraction, the LSTM adjusts its hidden vector in a way that also shifts the place value prediction. It learns the cross-KC structure from data.

Why not a transformer?

The sequence lengths we're working with are short — 10 to 30 attempts per session. Transformers need longer sequences to exploit their attention mechanism meaningfully. An LSTM is a better fit: it handles variable-length sequences natively, trains faster on small datasets, and produces interpretable per-step hidden states we can inspect.

More importantly: the Piech et al. (2015) DKT paper established LSTMs as the baseline for knowledge tracing. Improving on the baseline is Phase 4 work; Phase 3 is implementing it correctly.

The encoding

The input encoding follows Piech et al. exactly. At each step t, the input is a one-hot vector of size 2 × n_skills:

x[k]             = 1  if skill k was answered CORRECTLY
x[k + n_skills]  = 1  if skill k was answered INCORRECTLY

For three skills (SUB_BORROW=0, PLACE_VALUE=1, NUMBER_LINE=2), a correct subtraction answer encodes as:

[1, 0, 0,  0, 0, 0]
  ↑ correct half    ↑ incorrect half

An incorrect subtraction answer:

[0, 0, 0,  1, 0, 0]

The LSTM sees this 6-dimensional input and updates its hidden state. The output layer projects the hidden state back to 3 dimensions — one P(correct) per KC.

The training objective

The model learns to predict the NEXT response from the current history. At step t, given the encoded interaction x_t, the LSTM outputs:

ŷ_t[k] = σ(W × h_t + b)[k]  =  P(student answers skill k correctly at t+1)

The loss at each step uses only the skill that was actually asked next:

# At step t, the next question has skill_idx q and correctness r
target = torch.tensor([float(r)])
pred   = logits[0, t, q].unsqueeze(0)
loss_t = BCE(pred, target)

Training is one sequence at a time with Adam and gradient clipping. Small dataset — no need for batching yet.

Why the simulator came first

We can't train DKT on real data until the pilot delivers ≥150 attempt records. But we can validate the architecture right now with the student simulator.

The final integration test runs both pipelines end to end:

Generate 30 sequences from StrugglingSUB (35% accuracy on SUB_BORROW)
Generate 30 sequences from FastMaster (90% accuracy on SUB_BORROW)
Train two separate DKT models on each persona's sequences
Simulate 6 practice steps with each model
Assert FastMaster's model predicts higher mastery than StrugglingSUB's

# From test_dkt.py
fast_mastery = mastery_after_steps(result_fast.model,
    [True, True, True, True, True, False])    # 5/6 correct

struggling_mastery = mastery_after_steps(result_struggling.model,
    [True, False, False, True, False, False]) # 2/6 correct

assert fast_mastery > struggling_mastery      # ✓ passes

This gives us confidence the model learns the right signal before we hand it real children's data.

Why It Matters for the Research

BKT's independence assumption is a known limitation in the ITS literature. It was acceptable for Phase 1 and 2 because we didn't have cross-KC interaction data. Now that the mistake classifier is generating BORROW_SKIP and DIGIT_REVERSAL events consistently, we have a sequence model that can learn from them.

The specific research claim that DKT enables: a student's error pattern on one KC predicts their likely error pattern on a related KC. If DKT learns this and BKT doesn't, that's measurable evidence that the LSTM captures structure that the Markov model misses — and a direct contribution to the Phase 4 RCT analysis.

The upgrade path is explicit:

Pilot delivers ≥150 attempts
train_dkt(sequences_from_db) on the full dataset
Evaluate against BKT's predictions using held-out sessions
Replace update_bkt in SubmitAttemptUseCase when DKT's per-KC accuracy exceeds BKT's

The ADR for this transition is on the backlog.

What We Learned

The student simulator is the missing test fixture for ITS research. Standard software testing assumes you can construct any input you need. In adaptive tutoring, your input is a real child's learning trajectory. The simulator bridges that gap — it's not a replacement for real data, but it lets you test that the model responds in the right direction before you commit to an ethical review and a cohort of participants.

BKT and DKT coexist cleanly at the domain layer. KCState stays unchanged. DKTState is a separate dataclass with a different shape. The backend currently uses KCState; swapping in DKTState is an interface change at SubmitAttemptUseCase and GetNextProblemUseCase — two files, no schema migration.

Gradient clipping mattered more than I expected. Early training runs without clip_grad_norm_ diverged on the frustration-loop persona (all-incorrect sequences). Clipping at max_norm=1.0 stabilised training across all five personas.

What's Next

Backend wiring: load the trained DKT model at startup, store hidden state vectors in Redis per student, and swap the two use cases. That's the integration step that puts DKT into the live adaptive loop.

Key Takeaways

DKT's shared LSTM hidden state captures cross-KC dependencies that BKT's independent scalar parameters cannot — a student with DIGIT_REVERSAL on subtraction is more likely to have it on place value, and DKT learns this from data
Build the student simulator before the model: testing an adaptive learning architecture requires synthetic student trajectories, and the simulator lets you validate directional correctness before any ethics review or pilot recruitment
LSTM beats transformer for short sequences (10–30 steps): attention needs length to work; LSTMs handle variable-length sequences natively and train faster on the small datasets typical of ITS research

Building a mistake taxonomy for dyscalculia — 8 error patterns, rule-based, no ML required

Oscar Rieken — Mon, 01 Jun 2026 02:14:58 +0000

"Wrong" isn't a diagnosis.

When a student answers 32 − 9 = 37, they didn't randomly guess. They subtracted in the wrong direction in the ones column — a specific, named error called a borrow-skip. A tutor that just marks it incorrect and moves on has wasted the most informative signal in the attempt: why the student got it wrong.

NumPath's Phase 2 mistake classifier turns wrong answers into structured MistakeEvent records. Here's how we built it, what we got wrong the first time, and why rule-based classifiers beat a neural network for this job at this stage.

What We Built

Eight rule-based classifiers covering all three of NumPath's Phase 1 skill areas:

Code	Skill	Pattern
`DIGIT_REVERSAL`	SUB_BORROW / NUMBER_LINE	2-digit answer with digits transposed
`WRONG_OPERATION`	SUB_BORROW	Student added instead of subtracted
`BORROW_SKIP`	SUB_BORROW	Ones subtracted in reverse — no borrow taken
`OFF_BY_TEN`	SUB_BORROW	Result ±10 from correct (borrow applied to wrong column)
`PLACE_VALUE_CONFUSION`	PLACE_VALUE	Compared units digits only, ignored tens
`MAGNITUDE_MISJUDGE`	PLACE_VALUE	Chose the smaller number as larger
`NUMBER_LINE_DIRECTION`	NUMBER_LINE	Said "left" when answer is "right"
`OFF_BY_ONE`	NUMBER_LINE	Numeric answer ±1 from correct (miscounted steps)

Each classifier is a pure Python predicate — no external dependencies, no DB imports, testable in isolation. The main function runs them in priority order and returns the first match.

The Design Decision

The first question was: classify with rules or train a model?

The case for rules: we don't have labelled training data yet. Phase 1 just shipped. We have zero MistakeEvent records. Training a classifier on nothing produces nothing.

The case for ML: rules are brittle. A student might make a novel error we didn't anticipate, and rule-based code silently returns None.

We went with rules for Phase 2 because the error patterns for dyscalculia are well-documented in the ITS literature — specifically in the work of VanLehn (1982) on subtraction bugs and the later SIERRA system. "Borrow-skip" and "digit reversal" aren't our taxonomy; they're 40-year-old findings from cognitive science. A rule that detects them is more reliable than a model trained on 150 attempts.

The ML path opens in Phase 3 once the mistake_events table has enough volume. The rule-based classifier generates the labelled training data that Phase 3 will learn from.

The BORROW_SKIP bug

The Phase 1 classifier had a BORROW_SKIP function. It was wrong.

# Phase 1 — incorrect
def _is_borrow_skip(problem_content: dict, given: str) -> bool:
    a, b = operands
    no_borrow_result = str(a + b)    # ← adds a + b, not the borrow-skip result
    return given == no_borrow_result

This detected addition (32 − 9 → 41) and called it BORROW_SKIP. But addition is a completely different error — confusing +/− signs, not misapplying the borrowing algorithm. The mistake was labelled wrong in every event record.

The real borrow-skip pattern: when ones(a) < ones(b), the student skips borrowing and instead subtracts in the wrong direction in the ones column.

For 32 − 9:

Correct: borrow a ten → 12 − 9 = 3 ones, 2 tens → 23
Borrow-skip: ones = 9 − 2 = 7, tens = 3 (unchanged) → 37

# Phase 2 — correct
def _is_borrow_skip(a: int, b: int, given: str) -> bool:
    ones_a, ones_b = a % 10, b % 10
    if ones_a >= ones_b:
        return False  # no borrow needed — pattern doesn't apply
    borrow_skip_result = (a // 10 - b // 10) * 10 + (ones_b - ones_a)
    return given == str(borrow_skip_result)

Verified: 32 − 9 → 37 ✓, 43 − 18 → 35 ✓, 31 − 14 → 23 ✓

The old code was shipping the wrong signal for every borrow-skip attempt. This is exactly why MistakeEvent records are useless until the classifier is correct — the adaptive engine was routing "borrow-skip" students to the wrong remediation path.

The priority ordering problem

Multiple patterns can fire for the same wrong answer. For 43 − 16 = 27, the student wrote 72. That's a DIGIT_REVERSAL (27 reversed). But priority ordering becomes meaningful when patterns genuinely overlap.

The classifier runs a hierarchy:

subtraction problems:
  1. DIGIT_REVERSAL    ← most specific free-form error
  2. WRONG_OPERATION   ← added instead of subtracted
  3. BORROW_SKIP       ← skipped borrowing algorithm
  4. OFF_BY_TEN        ← borrow applied to wrong column

place_value problems (multiple-choice — no free-form digit writing):
  1. PLACE_VALUE_CONFUSION  ← compared units digits only (more specific)
  2. MAGNITUDE_MISJUDGE     ← picked the smaller number (less specific)

number_line problems:
  1. NUMBER_LINE_DIRECTION  ← wrong direction word
  2. DIGIT_REVERSAL         ← transposed digits in numeric answer
  3. OFF_BY_ONE             ← miscounted steps

Place value problems are multiple-choice, so DIGIT_REVERSAL doesn't apply there — the student picks from a given set, they don't write digits freely. Scoping by problem type prevents false positives.

Why It Matters for the Research

Every MistakeEvent record becomes a training signal twice:

Now: the adaptive engine reads the last MISTAKE_WINDOW (3) events. Two BORROW_SKIP codes in a row triggers remediation mode — the engine drops difficulty and targets SUB_BORROW problems specifically. Correct classification = correct remediation.

Later: Phase 3 will train a logistic regression (and eventually a transformer) on the mistake events table. The rule-based classifier generates the initial labelled dataset. If the rules are wrong — as BORROW_SKIP was — the ML model learns the wrong pattern from poisoned labels.

For a dyscalculia intervention study, this matters more than it would in a general tutoring system. Dyscalculia-specific errors like borrow-skip and digit reversal appear in the ITS literature as distinct cognitive profiles. Getting them right means the model can eventually distinguish students who have a procedural gap (BORROW_SKIP) from students who have a representational gap (PLACE_VALUE_CONFUSION) — a distinction that should affect the instructional intervention.

What We Learned

Rule-based classifiers need domain literature, not just intuition. The original BORROW_SKIP implementation was plausible — "student added instead of subtracting" — but wrong. VanLehn's subtraction bug taxonomy makes the actual pattern explicit. Reading the paper would have saved months of mislabelled data.

Priority ordering is a design document. The order in which classifiers run encodes assumptions about what matters more. We chose "most specific fires first" — but that could be wrong. Maybe WRONG_OPERATION (a conceptual error) should always beat DIGIT_REVERSAL (a transcription error) regardless of specificity, because they imply different interventions. We don't have the data to answer that yet.

50 tests is the right investment for a classifier that labels training data. A wrong label propagates forward through every model that trains on it. Testing every predicate in isolation, including priority ordering and edge cases, is not over-engineering — it's protecting the integrity of the entire data pipeline.

What's Next

The mistake_events table is now correctly populated with each session. Once the pilot delivers ≥150 records, Phase 3 can fit a logistic regression on the labelled events — using the rule-based codes as ground truth — and eventually replace the rules with a model that generalises to error patterns we haven't seen yet.

Key Takeaways

Rule-based mistake classifiers are the right first step when training data doesn't exist yet — they generate the labelled dataset that trains the eventual ML model
The real borrow-skip pattern (subtract ones in reverse: 32−9=37) is different from wrong-operation (add instead of subtract: 32+9=41) — getting this wrong poisons every downstream model that trains on the events table
Classifier priority ordering is a design decision that encodes instructional theory; document it explicitly and treat it as something to validate with data

Building a FastAPI + Vue 3 research platform: the 4 bugs that almost broke Phase 1

Oscar Rieken — Mon, 01 Jun 2026 02:11:33 +0000

Phase 1 of NumPath is done. Seven of eight Definition of Done items are checked — the eighth requires real children completing pilot sessions, which no amount of code will substitute for. The stack runs cleanly in Docker Compose, 56 unit tests pass, and a student can log in, answer ten problems, and see their knowledge state update in real time.

What the commit history doesn't show is the afternoon I spent fighting four bugs that don't appear in any FastAPI or Vue tutorial. This post is that afternoon.

What We Built

NumPath is an adaptive math tutor for children with dyscalculia. Phase 1 ships the minimum research instrument: a student practice loop, a rule-based adaptive engine, and a read-only teacher dashboard. No ML yet — just clean infrastructure and a data collection pipeline capable of generating the 150+ attempt records that Phase 2 needs to train the BKT model.

The stack: FastAPI 0.110 + SQLAlchemy 2 + Alembic + asyncpg on the backend; Vue 3 + Tailwind + Pinia on the frontend; PostgreSQL 16 + Redis 7 in Docker Compose.

Bug 1: passlib AttributeError on bcrypt ≥4.0

The symptom was immediate on first login attempt:

AttributeError: module 'bcrypt' has no attribute '__about__'

passlib has a version check that reads bcrypt.__about__.__version__. bcrypt 4.0 removed the __about__ module. The libraries have been incompatible for two years and passlib is effectively unmaintained.

The fix: delete passlib entirely. Replace it with three lines of direct bcrypt calls:

# backend/auth/password.py
import bcrypt

def hash_password(plain: str) -> str:
    return bcrypt.hashpw(plain.encode(), bcrypt.gensalt()).decode()

def verify_password(plain: str, hashed: str) -> bool:
    return bcrypt.checkpw(plain.encode(), hashed.encode())

pyproject.toml: swap "passlib[bcrypt]>=1.7.4" for "bcrypt>=4.0.0". Done. Don't reach for passlib on new Python projects — the dependency is dead.

Bug 2: pnpm 10 security policies blocking Docker builds

The frontend Dockerfile used node:20-slim and installed the latest pnpm via corepack. When pnpm 10 shipped, the build started failing with:

ERR_PNPM_PREPARE_PKG_FAILURE  Error when preparing the package
 Blocked by policy: electron-to-chromium@1.5.134 is not allowed
 because it was released 0 days ago (policy: minimumReleaseAge=3 days)

pnpm 10 introduced release-age security policies that refuse to install packages published within the last N days. A reasonable feature in production — a CI-breaking surprise when your lock file pins a package that was published yesterday.

Two separate policies hit us: minimumReleaseAge and ignored-builds (which blocks esbuild and vue-demi unless explicitly allowed). The package.json "pnpm" field that's supposed to configure these policies is silently ignored in pnpm 10 — it logs a warning and reads nothing.

The fix: pin to pnpm 9:

FROM node:22-slim
RUN corepack enable && corepack prepare pnpm@9.15.9 --activate

pnpm 9 has no release-age policies. The upgrade to pnpm 10 can wait until the project has a proper CI environment to absorb the breaking change.

Bug 3: FastAPI container connecting to localhost instead of postgres

The backend started cleanly. Every database call returned:

asyncpg.exceptions.ConnectionRefusedError: connection refused (host 127.0.0.1, port 5432)

The DATABASE_URL in .env was postgresql+asyncpg://numpath:numpath@localhost:5432/numpath. Inside a Docker Compose network, localhost is the container's own loopback — not the postgres service. The postgres container is reachable by its service name.

The fix: override the env var at the service level in docker-compose.yml:

backend:
  env_file: ../.env
  environment:
    DATABASE_URL: postgresql+asyncpg://numpath:numpath@postgres:5432/numpath
    REDIS_URL: redis://redis:6379/0

The environment block wins over env_file, so local development (which uses localhost) keeps working. Containers talk to each other by service name.

Bug 4: SQLAlchemy column defaults not applied at construction time

This one cost the most time. POST /attempts returned a 500:

TypeError: unsupported operand type(s) for -: 'int' and 'NoneType'

The BKT update equation was subtracting from p_learn, which was None. The KCStateRecord model had:

class KCStateRecord(Base):
    p_learn: Mapped[float] = mapped_column(Float, default=0.3)
    p_guess: Mapped[float] = mapped_column(Float, default=0.2)
    p_slip:  Mapped[float] = mapped_column(Float, default=0.1)

The bug: SQLAlchemy's default= is a server-side or flush-time default. When you construct KCStateRecord() in Python and haven't flushed to the database yet, those columns are None on the Python object. The domain code ran immediately after construction, before any flush.

The fix: set defaults explicitly in the constructor, then flush and refresh before returning:

record = KCStateRecord(
    student_id=student_id,
    skill_id=skill_id,
    p_mastery=0.1,
    p_learn=0.3,
    p_guess=0.2,
    p_slip=0.1,
    opportunity_count=0,
)
self._db.add(record)
await self._db.flush()         # write to DB so defaults are applied
await self._db.refresh(record) # re-read the DB-populated values

The rule: if you use a newly constructed SQLAlchemy model object before any flush, assume every default= column is None. Either set defaults in the constructor or flush first.

What the BKT update looks like in practice

With those bugs cleared, the full attempt flow works end to end. A correct answer on a SUB_BORROW problem with a fresh KCState shows:

before: p_mastery=0.100, opportunity_count=0
after:  p_mastery=0.533, opportunity_count=1

That 0.1 → 0.533 jump is the Bayesian update working — prior p_mastery combines with p_learn, corrected for p_guess and p_slip. The math is covered in detail in Bayesian Knowledge Tracing in 37 lines of Python.

Why It Matters for the Research

Phase 1's job was never to be elegant — it was to be instrumented. Every attempt record written to the attempts table is a training signal for Phase 2's BKT parameter estimation. We need ≥150 records (5 students × 3 sessions × 10+ problems) before Phase 2 can begin.

The bugs above are why research-grade software is harder than it looks. Each one silently corrupts data in a different way: password hashing fails outright (detectable), Docker networking fails silently on every write (detectable but subtle), SQLAlchemy defaults produce None BKT parameters (corrupts ML inputs, hard to detect in test data).

The fix for all of them is the same: run the full stack. Not unit tests. Not import my_function; print(my_function()). Start the containers, log in as a real user, and watch what happens.

What We Learned

The honest retrospective:

Seed data is harder than it looks. Writing 60 hand-crafted math problems at three difficulty levels takes longer than writing the adaptive engine. Every problem needs a machine-checkable answer, a hint, and a calibrated difficulty score.

Docker Compose env_file + environment is the right pattern. env_file carries the defaults; environment carries container-specific overrides. The pattern is obvious in hindsight and invisible until you need it.

The flush() + refresh() pattern is load-bearing for async SQLAlchemy. Any code that creates an ORM object and immediately passes it to domain logic needs an explicit flush. The async path doesn't auto-flush the way the synchronous ORM used to.

What's Next

Phase 2: BKT parameter estimation from real student data, and a mistake classifier that categorises subtraction errors beyond "wrong." The attempts table is waiting.

Key Takeaways

passlib is dead — use bcrypt directly; it's three functions and no transitive dependency risk
Docker Compose containers reach each other by service name, not localhost; override DATABASE_URL in the environment block rather than the env_file
SQLAlchemy default= columns are None on a freshly constructed Python object until after a flush() + refresh() — always set constructor defaults explicitly when domain code runs immediately after creation

Bayesian Knowledge Tracing in 37 lines of Python — how NumPath models what a student knows

Oscar Rieken — Wed, 27 May 2026 05:23:15 +0000

What We Built

NumPath maintains a KCState for every student × Knowledge Component pair. After every attempt, update_bkt() revises the probability that the student has mastered that KC. That probability — p_mastery — is what the adaptive engine reads to pick the next problem and what the teacher dashboard displays as a progress bar.

The entire model is 37 lines. Here it is unabridged.

from dataclasses import dataclass

MASTERY_THRESHOLD = 0.80

@dataclass(frozen=True)
class KCState:
    p_mastery: float
    p_learn: float
    p_guess: float
    p_slip: float
    opportunity_count: int = 0

    @property
    def is_mastered(self) -> bool:
        return self.p_mastery >= MASTERY_THRESHOLD


def update_bkt(state: KCState, is_correct: bool) -> KCState:
    """Standard Bayesian Knowledge Tracing update (Corbett & Anderson, 1995)."""
    p, L, G, S = state.p_mastery, state.p_learn, state.p_guess, state.p_slip

    if is_correct:
        posterior = (p * (1 - S)) / (p * (1 - S) + (1 - p) * G)
    else:
        posterior = (p * S) / (p * S + (1 - p) * (1 - G))

    p_new = posterior + (1 - posterior) * L

    return KCState(
        p_mastery=min(1.0, max(0.0, p_new)),
        p_learn=L,
        p_guess=G,
        p_slip=S,
        opportunity_count=state.opportunity_count + 1,
    )

The Four Parameters

BKT models each KC with four parameters, all probabilities between 0 and 1:

Parameter	Meaning	NumPath default
`p_mastery`	P(student has learned this KC)	0.10 (prior — low, conservative)
`p_learn`	P(learning occurs on this attempt, given not yet learned)	0.30
`p_guess`	P(correct answer given KC not learned)	0.20
`p_slip`	P(incorrect answer given KC is learned)	0.10

These are Phase 1 seed values — not calibrated against real student data yet. The parameter estimation problem (fitting p_learn, p_guess, p_slip per KC from observed attempts) is a Phase 4 task once the RCT produces enough data. For now they are reasonable priors from the BKT literature.

The Update Equations

After observing an answer, two steps happen.

Step 1 — Bayesian update (prior → posterior):

Correct:   posterior = p(1 - S) / [p(1 - S) + (1 - p)G]
Incorrect: posterior = pS       / [pS       + (1 - p)(1 - G)]

This is straight Bayes. A correct answer raises the posterior unless the student is likely to have guessed. An incorrect answer lowers it unless the student is likely to have slipped. A correct answer from a student with p_mastery=0.95 and p_slip=0.10 barely moves the needle — the model already thinks they know it. A correct answer from a student with p_mastery=0.10 and p_guess=0.20 moves it less than you might expect — the model discounts lucky guesses.

Step 2 — Learning update (posterior → next prior):

p_new = posterior + (1 - posterior) × p_learn

Even if the student answered incorrectly, there's a p_learn probability that learning occurred anyway. The posterior is never the final state — the learning update always nudges p_mastery upward slightly, reflecting that every attempt is an opportunity.

The Design Decision

We evaluated three approaches before choosing standard BKT:

Item Response Theory (IRT) — models item difficulty as well as student ability. More expressive, but requires calibrated item parameters we don't have. Rejected for Phase 1.

Deep Knowledge Tracing (DKT) — replaces the parametric model with an LSTM that learns latent student state from sequences of attempts. Better at capturing cross-KC transfer. Rejected for Phase 1 because it needs training data we haven't collected yet. It's on the Phase 2 roadmap.

Accuracy streak — raise difficulty after 3 correct in a row, lower after 3 wrong. This is what most commercial apps do. Rejected because it gives you no probability estimate, no per-KC granularity, and no way to distinguish a guesser from a learner.

Standard BKT is 30 years old and still the right choice when you're instrument-building before data collection. It gives you a per-KC probability estimate with interpretable parameters, it's fast to compute, and its failure modes are well understood.

One implementation choice worth noting: KCState is a frozen dataclass. update_bkt() returns a new KCState rather than mutating the existing one. This makes the update function a pure function — easy to test, easy to replay, and safe to call in parallel if we ever need to.

Why It Matters for the Research

The RCT compares learning outcomes for students using NumPath against a control group using static worksheets. To measure a difference, you need a measurement instrument. p_mastery is that instrument.

After a session, the teacher dashboard shows each student's p_mastery per KC as a progress bar. The adaptive engine uses it to pick the next problem. The LLM insight generator reads it to produce explanations like "Aiden's p_mastery on SUB_BORROW is 0.18 — the model has seen 11 attempts and is not converging." All three downstream consumers depend on the same number being meaningful.

BKT's key property for research purposes: it's falsifiable. If a student's p_mastery stays low after 20 correct answers, that's a signal worth investigating — either the parameters are wrong, or the student is consistently guessing, or there's a measurement problem. An accuracy percentage doesn't give you that.

What We Learned

The model is simple. Getting the parameters right is not.

p_learn=0.30 means the student has a 30% chance of learning the KC on any given attempt. That sounds reasonable. But it implies that after 10 attempts, a student who has not yet learned the KC has a 97% cumulative chance of learning it — which is almost certainly too optimistic. The seed parameters will need calibration.

The other thing we learned: opportunity_count is load-bearing. The adaptive engine uses it as a tiebreaker and the teacher dashboard shows it alongside p_mastery. It's not computed from the BKT model — it's just a counter that increments on every update_bkt() call. The frozen dataclass pattern makes this safe: the count in the database is always the count from the last update_bkt() return value, never a stale mutation.

What's Next

Phase 2 adds a DKT model alongside BKT — trained on the data collected during the pilot. The two models will run in parallel so we can compare their predictions against observed outcomes before the RCT begins.

Key Takeaways

BKT separates learning from performance — p_guess and p_slip let the model discount lucky correct answers and unlucky wrong ones; a 70% accuracy rate means something different depending on what the model thinks caused it
p_mastery is the measurement instrument for the RCT — every downstream consumer (adaptive engine, teacher dashboard, LLM insights) reads the same number, so getting it right matters more than getting it fast
Frozen dataclass + pure function = safe update chain — update_bkt() returns a new KCState; there's no shared mutable state, the update is replayable, and the test suite can verify every case in isolation

Two Cross-Platform Bugs in Our Go CLI (And How We Fixed Them)

Oscar Rieken — Wed, 27 May 2026 05:22:56 +0000

Go's cross-platform story is genuinely good. Write code once, compile for any target, mostly just works. But "mostly" hides a couple of sharp edges that bit us while building TestSmith. Both bugs were invisible on macOS and Linux, only surfaced on Windows CI, and had the same root cause: assumptions about path separators and filesystem traversal boundaries.

Bug 1: The Detector Boundary Escape

TestSmith has five language drivers, each responsible for detecting whether a directory is a project of its type. The Python driver walks upward from the starting directory, looking for pyproject.toml or setup.py. The Go driver looks for go.mod. And so on.

The bug: every driver would happily walk past a .git directory belonging to a different project and claim files in an ancestor project.

Here's what happened in practice. Our example projects live at:

testsmith/                  ← Go repo root (.git here)
  examples/
    python-service/          ← Python example project
      pyproject.toml

When you ran testsmith generate from inside examples/python-service/, the Python driver would detect it correctly. But when you ran it from examples/go-service/ and the Python driver was tried first during registry detection, it would walk upward, find no Python markers in go-service/, then continue upward, find no markers in examples/, then continue upward... find conftest.py at the testsmith repo root (left over from a previous test run), and claim the entire testsmith repo as a Python project.

The naive fix is "stop when you see .git." But that's wrong too — a legitimate project root can have both pyproject.toml and a .git directory. If you stop at the first .git you see, you'd refuse to detect projects that are also VCS roots.

The correct rule: check VCS stop markers only at ancestor directories, not at the starting directory.

func findRoot(startDir string) (string, error) {
    dir := startDir
    for {
        // Only check VCS boundaries at ancestor dirs — the starting dir
        // may legitimately have both a project marker and a .git directory.
        if dir != startDir {
            for _, stop := range stopMarkers {
                if _, err := os.Stat(filepath.Join(dir, stop)); err == nil {
                    return "", domain.ErrProjectNotFound
                }
            }
        }
        for _, marker := range rootMarkers {
            if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
                return dir, nil
            }
        }
        parent := filepath.Dir(dir)
        if parent == dir {
            break
        }
        dir = parent
    }
    return "", domain.ErrProjectNotFound
}

We applied this pattern to all five drivers. The key insight: .git is a traversal-stopping sentinel when found in an ancestor, but it's perfectly normal at the project root itself.

Bug 2: Hardcoded Path Separators

The Windows test failure was more direct:

    analyzer_test.go:179: DeriveTestPath("/proj/src/services/payment.py"):
        got "\\proj\\tests\\src\\services\\test_payment.py",
        want "/proj/tests/services/test_payment.py"

Two things wrong in that output: backslashes (expected on Windows, handled by the test via filepath.ToSlash), and src appearing in the output path when it should have been stripped.

The code:

func deriveTestPath(sourcePath string, ctx *domain.ProjectContext) (string, error) {
    rel, err := filepath.Rel(ctx.Root, sourcePath)
    if err != nil {
        return "", err
    }

    // Strip src/ prefix if present.
    if len(rel) > 4 && rel[:4] == "src/" {  // ← BUG
        rel = rel[4:]
    }
    // ...
}

filepath.Rel on Windows returns src\services\payment.py. The prefix check looks for src/ with a forward slash. On Windows, it never matches. The src component stays in the path, so the output becomes tests\src\services\test_payment.py instead of tests\services\test_payment.py.

The fix normalises to forward slashes before the check:

// Normalise to forward slashes for the prefix check so this works on
// Windows (where filepath.Rel returns backslash-separated paths).
slashed := filepath.ToSlash(rel)
if strings.HasPrefix(slashed, "src/") {
    slashed = slashed[4:]
}
rel = filepath.FromSlash(slashed)

filepath.ToSlash converts \ to /. strings.HasPrefix(slashed, "src/") works correctly on all platforms. filepath.FromSlash converts back to the OS-native separator for the subsequent filepath.Join call.

The same pattern applied to deriveModulePath, which had the identical bug.

The Pattern

Both bugs share a structure: an algorithm that works correctly on the development platform (macOS/Linux) but silently produces wrong results on Windows because it makes assumptions about the filesystem:

Bug 1: assumes .git presence implies "not a project root" (wrong at the starting dir)
Bug 2: assumes filepath.Rel uses forward slashes (wrong on Windows)

The remedies are similarly structured:

Bug 1: be explicit about which directories the invariant applies to
Bug 2: normalise to a known format before string operations, then convert back for OS operations

Go's filepath package is excellent — filepath.Rel, filepath.Join, filepath.Dir, filepath.Base all do the right thing. The problems arise when you mix filepath results with hardcoded string literals (like "src/") that embed platform assumptions. The rule: use filepath functions for path operations, filepath.ToSlash to convert before any string matching, and filepath.FromSlash to convert back before passing to OS calls.

CI as the Detector

Neither bug would have been caught by running tests locally on macOS. The Windows CI job was the only place they surfaced.

This is the case for a real cross-platform test matrix. It's not just about supporting Windows users — it's about finding any code that makes implicit platform assumptions. If your tests only run on one platform, that class of bug is invisible until a user reports it.

TestSmith is open source at github.com/orieken/testsmith. The full CI matrix runs on Ubuntu, macOS, and Windows with -race enabled on all three.

Two Knowledge Hierarchies: Structuring Context for AI Agents and LLMs

Oscar Rieken — Wed, 27 May 2026 05:22:10 +0000

TestSmith has two distinct audiences that need context about the project: AI agents that work on the TestSmith codebase (helping develop and extend it), and the LLM that generates test code for your project at runtime. These are different problems with different solutions.

Layer 1: Agent Context — CLAUDE.md Hierarchies

When an AI agent opens TestSmith to fix a bug or add a feature, it needs to understand the codebase structure without reading every file. A single large context file doesn't work well — an agent fixing a retry bug doesn't need to know the Java driver's fixture generation logic.

The solution is a CLAUDE.md hierarchy:

CLAUDE.md                              ← package map, invariants, dependency direction
internal/domain/CLAUDE.md             ← interfaces, key types, "add a field" checklist
internal/generation/CLAUDE.md         ← pipeline data flow, verifier selection
internal/llm/CLAUDE.md                ← middleware stack, batch vs fan-out, cache key
internal/projectknowledge/CLAUDE.md   ← TESTSMITH.md hierarchy, budget tiers
internal/drivers/CLAUDE.md            ← how to add an adapter or language driver

The root file is the map. The per-package files are the territory. An agent touching the LLM retry logic loads internal/llm/CLAUDE.md — it never sees the driver or generation docs.

The root file contains three things that every agent needs regardless of task:

Package map — what each internal package does and which files to read first
Dependency direction — the hard architectural constraint (domain never imports other internal packages; drivers never import generation)
Invariants — things that must remain true across all changes (e.g., GeneratedFile.Language must always be set; resolveAction has specific rules for fixture vs. non-fixture files)

Per-package files contain the "read this before touching this package" context: data flow diagrams for the pipeline, the middleware stack for the LLM layer, the adapter registration pattern for drivers.

When Claude Code loads a file in a package, it automatically reads that package's CLAUDE.md. The agent gets exactly what it needs, nothing more.

Layer 2: Runtime LLM Context — TESTSMITH.md

This is what TestSmith injects into prompts when generating tests for your project. It's a conventions file you maintain alongside your source code.

Two levels are merged at generation time:

<project-root>/TESTSMITH.md     ← always loaded; project-wide framework, mock style
<source-dir>/TESTSMITH.md       ← optional; package-level overrides

Example root TESTSMITH.md:

# Project conventions

Framework: pytest
Mock style: pytest-mock (use `mocker.patch`, not `unittest.mock.patch`)
Assertion style: plain assert statements

# Module structure
Services are in `src/services/`. Each service has a single public class.
Tests go in `tests/` mirroring the `src/` structure.

Example per-directory override in src/services/payment/TESTSMITH.md:

# Payment service conventions
This module integrates with Stripe. Mock all `stripe.*` calls.
Use `pytest.mark.vcr` for HTTP interaction tests.

The root file is loaded once at startup and cached in ProjectContext. The per-directory file is merged lazily — only when a file in that directory is being generated. A large monorepo never loads context it doesn't need.

Both files go into the system prompt, not the user prompt. This matters because the user prompt is subject to a configurable token budget (PromptTokenBudget, default 6,000 tokens) with a priority-based trim:

Priority	Content	Dropped when?
1 (never)	Source code	Never
2	Internal dep signatures	Budget exceeded after source
3	Style snippet from nearby tests	Dropped first

Project knowledge is exempt from this budget entirely — it stays in the system prompt regardless of how large the source file is.

Dynamically Mined Conventions

Beyond TESTSMITH.md, TestSmith also mines conventions from existing tests in the same directory — up to 5 files, capped at 80 lines total. This gives the model real examples of the project's test style without requiring the developer to maintain a conventions doc.

This is cheaper and more accurate than a hand-written guide: it automatically reflects the actual test patterns in use, and it updates itself as tests evolve. If your team starts using a new assertion pattern, the next generation run picks it up.

The Dependency Signature Index

The third piece is the dep index: at the start of a --all run, TestSmith analyses every source file once and builds a modulePath → SourceAnalysis map. When generating tests for payment.go, it can pull the public API signature of discount.go (which payment.go imports) from memory:

// In the prompt:
// Internal dependency signatures:
// discount.ApplyPromoCode(order Order, code string) (Order, error)
// discount.ValidateCode(code string) bool

This tells the model what the real interface looks like so it generates test doubles that match the actual signatures — not invented ones.

In watch mode, when a file changes, only that file's entry is refreshed. The rest of the index stays warm between regens.

Why the Separation Matters

The two layers solve different problems:

Agent context is about development-time navigation. It's hierarchical, human-readable, and loaded selectively. It describes architecture and invariants. It lives in the repo and is maintained alongside the code it describes.
Runtime LLM context is about generation-time quality. It's merged from two levels, injected into system prompts, and exempt from token budgets. It describes conventions and patterns specific to the target project — things an LLM can't infer from source code alone.

Conflating the two leads to either bloated system prompts (dumping agent context into every generation request) or under-informed agents (giving them only the user-facing conventions doc with no architectural guidance). Keeping them separate means each audience gets exactly what it needs.

Next: the cross-platform bugs we hit shipping a Go CLI — detector boundary escapes and Windows path separators.

Making LLM Calls Reliable: Retry, Semaphore, Cache, and Batch

Oscar Rieken — Sat, 23 May 2026 16:34:37 +0000

When TestSmith generates tests with --llm, it calls an LLM for every public member of every source file being processed. A project with 20 files and 5 public functions each means up to 100 API calls in a single run. That's a lot of surface area for things to go wrong.

Here's the reliability stack we built, layer by layer.

Layer 1: Retry with Exponential Backoff

LLM APIs fail transiently. Rate limits, timeouts, occasional 5xx responses — all of these are recoverable if you wait and retry. We built a retry middleware that wraps any Provider:

type RetryProvider struct {
    inner      Provider
    maxRetries int
}

func (r *RetryProvider) Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error) {
    var lastErr error
    for attempt := 0; attempt < r.maxRetries; attempt++ {
        if attempt > 0 {
            wait := time.Duration(math.Pow(2, float64(attempt))) * 100 * time.Millisecond
            select {
            case <-time.After(wait):
            case <-ctx.Done():
                return CompletionResponse{}, ctx.Err()
            }
        }
        resp, err := r.inner.Complete(ctx, req)
        if err == nil {
            return resp, nil
        }
        lastErr = err
    }
    return CompletionResponse{}, fmt.Errorf("after %d attempts: %w", r.maxRetries, lastErr)
}

MaxRetryAttempts defaults to 3. With exponential backoff: attempt 1 is immediate, attempt 2 waits 200ms, attempt 3 waits 400ms. Total worst-case wait per call is under a second — acceptable latency for a background tool.

Layer 2: Semaphore for Concurrency Control

With up to 100 calls to make, goroutine fan-out is the obvious approach. But hitting an LLM API with 100 concurrent requests triggers rate limiting immediately. A semaphore caps the in-flight calls:

type SemaphoreProvider struct {
    inner Provider
    sem   chan struct{}
}

func NewSemaphoreProvider(inner Provider, maxConcurrent int) *SemaphoreProvider {
    return &SemaphoreProvider{inner: inner, sem: make(chan struct{}, maxConcurrent)}
}

func (s *SemaphoreProvider) Complete(ctx context.Context, req CompletionRequest) (CompletionResponse, error) {
    select {
    case s.sem <- struct{}{}:
        defer func() { <-s.sem }()
    case <-ctx.Done():
        return CompletionResponse{}, ctx.Err()
    }
    return s.inner.Complete(ctx, req)
}

MaxConcurrentCalls defaults to 5. Each retry attempt acquires its own semaphore slot — this is important. If retry logic held a slot while waiting between attempts, other goroutines would be blocked unnecessarily. The retry wrapper is the outer layer; semaphore is the inner layer.

The middleware stack assembled by the factory:

retry → semaphore → raw provider

Layer 3: Result Cache

Many test generation runs touch the same files repeatedly — watch mode is the extreme case. Calling the LLM for the same source code twice is wasteful. A content-addressed cache avoids it:

type ResultCache struct {
    mu      sync.RWMutex
    entries map[string][]BodyGenResult
    hits    int
    misses  int
}

func cacheKey(req BodyGenRequest) string {
    h := sha256.New()
    fmt.Fprintf(h, "%s\n%s\n%s\n%s", req.Language, req.MemberName, req.SourceCode, req.Framework.Name)
    return hex.EncodeToString(h[:])
}

The key is a SHA-256 hash of the language, member name, source code, and framework. If the source file changes, the hash changes and the cache misses — you always get fresh results for changed code.

After a run, --verbose prints the cache stats:

LLM cache — hits: 12  misses: 8  entries: 8

Layer 4: Batch Generation

The fan-out approach makes one API call per public member. For a file with 10 functions, that's 10 calls. Batch generation collapses this to one:

func (g *LLMBodyGenerator) GenerateBatchBodies(
    ctx context.Context,
    reqs []BodyGenRequest,
) ([]BodyGenResult, error) {
    prompt := buildBatchPrompt(reqs)
    resp, err := g.provider.Complete(ctx, CompletionRequest{
        SystemPrompt:   batchSystemPrompt,
        UserPrompt:     prompt,
        Model:          g.model,
        MaxTokens:      g.maxTokens * len(reqs), // scale with request count
        Temperature:    g.temperature,
        ResponseFormat: "json_object",            // structured output
    })
    // ...
}

We use OpenAI's response_format: {"type": "json_object"} to get structured output. The model returns a JSON envelope with one entry per member:

{
  "tests": [
    {"name": "ProcessPayment", "code": "func TestProcessPayment(t *testing.T) { ... }"},
    {"name": "RefundPayment",  "code": "func TestRefundPayment(t *testing.T) { ... }"}
  ]
}

We parse that with a primary JSON parser, with a fallback to a delimiter-regex parser for providers that don't support structured output.

The pipeline checks for the BatchBodyGenerator interface via type assertion. If the generator implements it, batch mode is used. If not (or if the driver explicitly opts out), it falls back to goroutine fan-out with individual calls. This keeps the interface opt-in and backward compatible.

Observability: Cache Stats

With all this happening in the background, it's useful to know what actually ran. The cacheStatsReporter interface lets the CLI query stats without importing the llm package:

// In cmd/testsmith/generate.go — avoids importing internal/llm from the CLI layer
type cacheStatsReporter interface {
    CacheStats() (hits, misses, size int)
}

func printCacheStats(bg domain.BodyGenerator) {
    if !verbose {
        return
    }
    if r, ok := bg.(cacheStatsReporter); ok {
        hits, misses, size := r.CacheStats()
        fmt.Printf("LLM cache — hits: %d  misses: %d  entries: %d\n", hits, misses, size)
    }
}

This is the interface segregation principle at work: the CLI knows about domain.BodyGenerator (which it needs for the pipeline) and cacheStatsReporter (which it needs for stats output). It doesn't need to know anything else about the LLM implementation.

The Numbers

In practice, on a mid-size Go project with 40 source files and an average of 6 public functions each:

Without batch: 240 API calls, ~4 minutes at 5 concurrent
With batch: 40 API calls (one per file), ~45 seconds
Second run with warm cache: near-instant for unchanged files

The cache and batch generation together turn what would be a "go make coffee" operation into something you can run while you're still in the flow.

Next: how we structure context for both AI agents working on TestSmith itself and for the LLM generating tests for your project.

Language-Agnostic Code Generation: The Driver Plugin Model

Oscar Rieken — Sat, 23 May 2026 16:29:11 +0000

TestSmith generates test scaffolds for five languages: Go, Python, TypeScript, Java, and C#. Each language has its own project structure conventions, test frameworks, import styles, and code patterns. The naive implementation would be a big switch statement throughout the codebase. We chose a plugin model instead.

The Problem with Hardcoded Branches

When a codebase switches on language in multiple places, every new language requires touching every branch point. Miss one and you get a silent bug — the new language falls through to some default behavior that doesn't apply to it. This is the classic Open-Closed violation: you have to modify existing code to extend it.

The LanguageDriver Interface

Every language in TestSmith implements a single interface:

type LanguageDriver interface {
    // Detection
    DetectProject(dir string) (*ProjectContext, error)
    FileExtensions() []string

    // Analysis
    AnalyzeFile(path string, ctx *ProjectContext) (*SourceAnalysis, error)
    ClassifyDependency(dep ImportInfo, ctx *ProjectContext) DependencyCategory
    DeriveTestPath(sourcePath string, ctx *ProjectContext) (string, error)
    DeriveModulePath(sourcePath string, ctx *ProjectContext) (string, error)

    // Generation
    GenerateTestFile(analysis *SourceAnalysis, opts GenerateOpts) (*GeneratedFile, error)
    GenerateFixture(dep string, analysis *SourceAnalysis, opts GenerateOpts) (*GeneratedFile, error)
    GenerateBootstrap(plan *GenerationPlan, ctx *ProjectContext) (*GeneratedFile, error)

    // Framework config
    GetTestFrameworkConfig() TestFrameworkConfig
    SelectAdapter(ctx *ProjectContext) TestAdapter

    // LLM integration
    LLMContext(ctx *ProjectContext) map[string]string
    LLMVocabulary() map[string]string

    // Migration and validation
    ListMigrators() []Migrator
    ValidateFile(path string, ctx *ProjectContext) ([]ValidationIssue, error)
}

The generation pipeline, the CLI commands, and the watch mode all work against this interface. They never import a specific driver package.

How Detection Works

When you run testsmith generate, the first step is figuring out what language you're in. The registry tries each registered driver in turn:

func Detect(dir string) (domain.LanguageDriver, error) {
    for _, d := range drivers {
        ctx, err := d.DetectProject(dir)
        if err == nil && ctx != nil {
            return d, nil
        }
    }
    return nil, domain.ErrProjectNotFound
}

Each driver's DetectProject walks upward from the starting directory looking for its own project markers — go.mod for Go, pyproject.toml or setup.py for Python, package.json for TypeScript, pom.xml or build.gradle for Java, .csproj or .sln for C#.

One subtle requirement: a driver must not claim an ancestor project that belongs to a different language. If you run TestSmith from inside an example project that lives inside a Go repo, the Python driver shouldn't walk up past the Go project's .git boundary and claim the repo root. We solve this by checking VCS stop markers (.git, .hg, .svn) at ancestor directories only — not at the starting directory itself, since a legitimate project root can have both a project marker and a .git directory.

func findRoot(startDir string) (string, error) {
    dir := startDir
    for {
        // At ancestor dirs, stop at VCS boundaries first.
        if dir != startDir {
            for _, stop := range stopMarkers {
                if _, err := os.Stat(filepath.Join(dir, stop)); err == nil {
                    return "", domain.ErrProjectNotFound
                }
            }
        }
        // Then check for project markers.
        for _, marker := range rootMarkers {
            if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
                return dir, nil
            }
        }
        parent := filepath.Dir(dir)
        if parent == dir {
            break
        }
        dir = parent
    }
    return "", domain.ErrProjectNotFound
}

The Adapter Layer

Within a language, there can be multiple test frameworks. TypeScript has Jest, Vitest, and Mocha. Java has JUnit 4, JUnit 5, TestNG, and Spring Boot Test. Each framework has its own import style, mock library, assertion syntax, and file naming conventions.

We model this with a TestAdapter interface:

type TestAdapter interface {
    Name() string
    FileNamingConvention() FileNaming
    ImportStyle() ImportStyle
    MockLibrary() string
    AssertionStyle() string
    LLMVocabulary() map[string]string
}

Each driver has a registry of adapters and a SelectAdapter method that reads the project config (or sniffs package.json devDependencies, pom.xml dependencies, etc.) to pick the right one. The LLM prompt gets the vocabulary from the selected adapter — so the model knows to generate expect(x).toBe(y) for Jest but assert.Equal(t, x, y) for Go's testify.

Adding a New Language

Because everything flows through the interface, adding a new language driver is isolated:

Create a new package under internal/drivers/<lang>/
Implement domain.LanguageDriver — the compiler tells you exactly what's missing
Register it in internal/registry/registry.go
Optionally add a Verifier in internal/generation/verify.go for post-write compile checking

No other files change. The existing drivers are untouched. The pipeline, CLI, and watch mode pick it up automatically.

The Dependency Direction

The plugin model enforces a strict dependency direction:

cmd → generation → domain ← drivers
                           ← llm

domain defines the interfaces. drivers implement them. generation uses them via the interface. Neither generation nor drivers imports the other. This is the Dependency Inversion Principle applied at the package level — and it's enforced by Go's import cycle detector.

When you add a new driver, it's impossible to accidentally reach into the generation pipeline or the LLM layer — Go won't compile it. The architecture is self-enforcing.

Next in this series: making LLM calls reliable when you're hitting them for every public member of every source file.

Why We Rewrote Our Python CLI in Go (and What We Gained)

Oscar Rieken — Sat, 23 May 2026 16:23:38 +0000

TestSmith v1 was a Python CLI. It worked. Users could pip install testsmith, point it at a source file, and get a test scaffold back. But every team that tried to wire it into CI hit the same wall: Python environments.

The problem wasn't Python itself — it was distribution. A static analysis tool that requires a matching Python version, a virtual environment, and a pinned dependency tree is a hard sell for a step that runs on every push. We were shipping a tool, not a library. Tools should be frictionless.

The Decision

We rewrote TestSmith v2 in Go. The goal was a single static binary with no runtime dependencies — something you could drop into any CI runner, any Docker image, any developer's PATH, and it would just work.

Go was the right choice for three reasons:

Single binary. go build produces one self-contained executable. No pip, no venv, no requirements.txt. Users download a binary or brew install it and they're done.

Cross-platform with one build. The v1 CI matrix was a headache — different Python versions across Ubuntu, macOS, and Windows, with slightly different behavior on each. Go's cross-compilation gave us linux/amd64, darwin/amd64, darwin/arm64, and windows/amd64 from a single build step.

Native concurrency. Test generation is embarrassingly parallel — each file is independent. Go's goroutines and channels made the fan-out generation and the debounced file watcher straightforward to implement without pulling in async libraries.

What Changed

The command surface was cleaned up in the same pass. v1 used flags for everything:

testsmith <file>         # generate
testsmith --all          # generate all
testsmith --graph        # dependency graph
testsmith --prune        # prune stale fixtures
testsmith --watch        # watch mode

v2 uses proper subcommands:

testsmith generate <file>
testsmith generate --all
testsmith graph
testsmith prune
testsmith watch

This made shell completion, help text, and per-command flags much cleaner. Cobra's built-in completion generator gives us bash, zsh, fish, and PowerShell completion for free.

Architecture Change

v1 was monolithic Python — one codebase with hardcoded branches for each language it supported. Adding a new language meant editing multiple core files.

v2 uses a LanguageDriver interface. Each language (Go, Python, TypeScript, Java, C#) is a separate package that implements the interface. The core generation pipeline never knows which language it's dealing with — it just calls through the interface:

type LanguageDriver interface {
    DetectProject(dir string) (*ProjectContext, error)
    AnalyzeFile(path string, ctx *ProjectContext) (*SourceAnalysis, error)
    DeriveTestPath(sourcePath string, ctx *ProjectContext) (string, error)
    GenerateTestFile(analysis *SourceAnalysis, opts GenerateOpts) (*GeneratedFile, error)
    // ... and more
}

Adding a new language is now a matter of creating a new package and registering it — no changes to the pipeline.

What We Kept

The v1 Python package didn't disappear. It lives in archive/v1/ and continues to receive bug fixes during the transition period. Teams already using v1 in production don't need to migrate immediately. The v2 binary is a clean break for new users; v1 stays stable for existing ones.

Was It Worth It?

Yes, unambiguously. The CI story went from "install Python, set up venv, pin deps" to "download one binary." The Windows test matrix went from flaky (Python path issues) to clean. And the plugin architecture means we can add a Ruby or Rust driver without touching the core generation logic.

The rewrite took longer than a feature would have. But distribution friction is a silent project killer — nobody files a bug for "this was annoying to set up," they just stop using the tool.

TestSmith is an open-source CLI for generating test scaffolds across Go, Python, TypeScript, Java, and C#. The source is at github.com/orieken/testsmith.