DEV Community

Great Sage
Great Sage

Posted on

Grading free-form Japanese answers with an LLM: Django + pydantic-ai

#japanese #python #django #ai

I'm studying for the JLPT (aiming for N2 this cycle), and the thing that kept bugging me about every prep app I tried is that they're all multiple choice. You tap A/B/C/D, you get a green checkmark, and you feel like you're learning. But multiple choice trains recognition, not recall. You can pattern-match your way to a passing-looking score without being able to actually produce a single sentence.

So I started building BetterLingo — a JLPT study app whose daily practice is free-form. No multiple choice. You write a real answer, and an LLM grades it. This post is about the part that was actually interesting to build: grading a free-form Japanese answer reliably enough that a learner trusts the score.

The problem with "just ask the model to grade it"

The naive version is one prompt: "Here's the question, here's the student's answer, give it a score out of 100." It works in a demo and falls apart in real use:

  • Scores are unstable — same answer, different run, ±20 points.
  • The model hallucinates a rubric that doesn't match the question type.
  • You get back prose you have to regex, and sometimes it just... doesn't give a number.
  • No structured breakdown, so the learner can't see why they lost points.

For a study tool that has to be near-zero cost per grade, run reliably, and give feedback a learner believes, none of that flies.

Structured output with pydantic-ai

The single biggest reliability win was forcing the model into a typed schema instead of free text. I use pydantic-ai, so the grade is a Pydantic model:

from pydantic import BaseModel, Field
from pydantic_ai import Agent


class GradeBreakdown(BaseModel):
    grammar: int = Field(ge=0, le=100)
    vocabulary: int = Field(ge=0, le=100)
    naturalness: int = Field(ge=0, le=100)
    meaning: int = Field(ge=0, le=100)


class Grade(BaseModel):
    score: int = Field(ge=0, le=100, description="Overall score")
    breakdown: GradeBreakdown
    feedback: str = Field(description="Two or three sentences, specific")
    example_answer: str = Field(description="A model answer in Japanese")


grader = Agent(
    "openai:qwen3-next-80b",  # via an OpenAI-compatible endpoint
    output_type=Grade,
    system_prompt=GRADER_SYSTEM_PROMPT,
)
Enter fullscreen mode Exit fullscreen mode

Because the output is a validated model, I never parse prose. If the model returns something off-schema, pydantic-ai catches it and retries automatically. The ge/le bounds mean a score is always a real 0–100 int, never "about 85".

Four question types, four rubrics

"Free-form" isn't one task. The daily set mixes four types, and each needs a different grading emphasis:

  1. Answer — respond to a prompt in Japanese. Graded on meaning + naturalness.
  2. Fill in the blank — one right-ish token, graded strictly on grammar/collocation.
  3. Translate EN→JP — meaning fidelity first, then naturalness.
  4. Translate JP→EN — comprehension check; grammar of the English matters less.

The trick that stabilized scores was not having one universal rubric. Each type gets its own system prompt and its own weighting of the breakdown fields. A perfectly natural answer that misses the point of a translation task shouldn't score the same as one that nails the meaning.

Keeping grading cost near zero

This runs on my own study sessions daily, and I want it to scale without a bill that scales with it. So the two LLM jobs are split by cost profile:

  • Grading (high volume, runs every practice item) uses Qwen3-Next-80B on NVIDIA's free inference tier. Structured output means even a smaller/cheaper model stays on-rails.
  • Tutor + hints (lower volume, needs to be sharp) use Claude Sonnet via OpenRouter.

The tutor — I call her Rami — is the other half of the free-form bet. She sees your actual attempt: the question, what you wrote, the score, the per-field breakdown, the feedback. So when you ask "why is my answer wrong," she's responding to your specific mistake, not reciting a generic grammar FAQ. That context handoff is only possible because the grade is structured data, not a paragraph.

The boring stack (on purpose)

The rest is deliberately unexciting so I spend my time on the learning loop, not plumbing:

  • Django 6 + HTMX + Tailwind — no JS build step, server-rendered.
  • Postgres for everything.
  • granian for ASGI.
  • A traditional SM-2 SRS review mode and kanji study with level gates, for when free-form recall is too heavy and you just want to drill.

What I've learned so far

  • Structured output is the reliability lever. Switching from "parse the prose" to "the grade is a typed object" removed almost all the flakiness in one move.
  • One rubric per task type beats one clever universal prompt.
  • A cheap model with tight constraints often beats an expensive model with a loose prompt, especially for high-volume grading.
  • Passing structured feedback into the tutor makes the tutor feel like it's actually reading your work — because it is.

It's still rough — I'm the primary user, fixing things as I hit them in my own N2 prep. If you're building anything that grades or evaluates free-form user input with an LLM, I'm happy to compare notes on schema design and rubric splitting. And if you're prepping for the JLPT yourself, betterlingo.io is the thing I'm building — I'd genuinely like the feedback.

Top comments (0)