I built a tutor that circles your math mistake in red pen

This is a submission for the MLH x DEV Writing Challenge

What I Built

Somo (Swahili for "lesson") is a math tutor you use by taking a photo. You snap a problem along with your handwritten working. Somo finds the exact line where the working goes wrong, circles it in red pen the way a teacher would, and then teaches the idea in English, Swahili or Spanish.

What inspired it

I grew up in Tanzania. Primary school is taught in Swahili. Secondary school switches every subject to English, and you sit the NECTA national exams in a language you're still learning. Plenty of students understand the math and get stuck on the words. I see the same thing with English learners in New York.

Most AI tools will solve the problem for you. I wanted to explore something different: could an AI tell a student why they went wrong, in the language they think in? I built Somo in one day to find out.

How it works

1. Snap. Photograph the problem and your working.
2. Diagnose. Gemini reads the handwriting and returns structured JSON with the transcription, the concept, the first wrong line, and why it's wrong. The UI circles that line in red.
3. Confirm. Somo shows "Here's what I read," and you confirm it or correct it before any teaching happens.
4. Teach. A short lesson streams in. It covers the idea, where it went wrong, an everyday analogy (a market scale, a subway fare), a worked example with different numbers so the answer isn't given away, and the English exam terms you'll need.
5. Check. You answer one question. Get it wrong and you get a simpler re-explanation with a new analogy. Get it right and you can practice on a real past exam question found live on the web.

What I learned

Reliability is most of AI engineering. The prompts were the easy part. The work went into everything around them:

• Self-verification. A second, independent model call solves the problem without seeing the diagnosis. If the two answers disagree, the student sees "I'm not fully sure I read this right."
• Verification can't catch reading mistakes. One of my test photos was a printed question that had lost its formatting. Both model calls read it the same wrong way, and they agreed with each other. That's why the human-in-the-loop confirm step exists: the student can see the paper, and the model can't always.
• Failover is a feature. Gemini's free quota ran out mid-build. I routed every call through one function: try Gemini, retry once on a 503/429, then fall back to Claude, with a circuit breaker that skips Gemini for 60 seconds after a quota error. Diagnosis went from ~116 seconds (waiting on an overloaded model) to ~10 seconds.
• Honest labels. My pipeline once showed a UK worksheet as a "real past exam" question. Now every question is labeled past exam paper, practice question from the web, or written by Somo, and off-topic questions are rejected.

In testing, Somo found the exact wrong line on all three of my handwritten test problems: a halving error (2B = 8 → B = 3), a dropped minus sign when rearranging, and multiplying the two sides of an equation by different numbers. No students have used it; every test problem was written by me.

Demo

The stack is a FastAPI backend with Server-Sent Events for streaming, plus a single-file mobile interface styled like a school exercise book: grid paper, a red margin line, and a handwriting font for the teacher's notes.

Partner Technologies

Google Gemini (and Gemma 4)

Gemini (gemini-flash-latest through the google-genai SDK) does the hard multimodal work. The key was structured output. Instead of parsing free text, the diagnosis is validated against a Pydantic schema:

response = client.models.generate_content(
    model="gemini-flash-latest",
    contents=[photo, DIAGNOSE_PROMPT],
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema=Diagnosis,  # transcription, concept, error_step, final_answer...
    ),
)
diagnosis = response.parsed

Because error_step is a real field, the UI knows exactly which line to circle. Gemini also streams the lessons and writes the check questions.

Gemma 4 (gemma-4-26b-a4b-it, served through the same Gemini API) handles the frequent, low-stakes work: grading answers and deciding whether two answers are equivalent ("x = 4" vs "4"). It's fast and cheap, so it's a good fit for checks that run on every answer.

Nimble

Nimble turned "practice more" into "practice the real exam." When a lesson starts, a background job asks Nimble to search the live web for past papers on the student's concept, starting with official sites:

response = nimble.search(
    query=f"NECTA CSEE Basic Mathematics past paper {concept}",
    max_results=5,
    search_depth="standard",
    full_content=True,
    include_domains=["necta.go.tz", "maktaba.tetea.org"],
)

US students get NY Regents question banks (jmap.org, nysedregents.org) instead, and if the official sites return nothing, Somo searches the open web.

Nimble found the right documents: official NECTA (Tanzania's National Examinations Board) Basic Mathematics papers and examiner reports from 2017 to 2023. But for PDFs, search results came back as ~300–700 character summaries, with no questions in them. The fix was to treat Nimble as the finder: when a result is a short PDF, Somo downloads it, extracts up to 20,000 characters with pypdf (five papers in parallel), and Gemini picks one question that appears word for word and tests the same concept. NECTA's Candidates' Item Response Analysis reports turned out to be gold, because they quote every exam question verbatim.

The first real result was a matrix word problem from the NECTA CSEE 2023 Basic Mathematics paper, which gets shown with an exam stamp and a link to the source. Results are cached to disk with fuzzy topic matching, so each topic is searched only once.

Sentry

Somo makes a lot of calls to several providers, so I wanted to see what it was actually doing. Every Gemini, Gemma, Claude and Nimble call is a Sentry span, and background jobs (verification, question prep, past-paper search) are their own transactions:

with sentry_sdk.start_span(op="gen_ai.chat", name=f"gemini {label}") as span:
    span.set_data("model", model)
    result = gemini_call()

Every Gemini → Claude failover also raises a warning issue, so the issues list became a timeline of when and why the app switched providers.

Then Sentry found a real bug. The trace waterfall for one past-paper search showed a single NECTA server taking 3.6 minutes to send one PDF. I had a 12-second timeout, but httpx applies it per chunk, so a server trickling data never trips it. The fix was a hard deadline on the whole download:

deadline = time.time() + 12
with httpx.stream("GET", url, timeout=12) as r:
    for chunk in r.iter_bytes():
        data.extend(chunk)
        if time.time() > deadline:
            raise TimeoutError("download took over 12s")

The worst case dropped from about 4 minutes to about 30 seconds. I wouldn't have found that one from logs. No student photos or prompt text are sent to Sentry (send_default_pii=False).

Hackathon Experience

I attended the AI Engineer NYC Hackathon, a one-day MLH event on October 11, 2026. I came in without a team and without an idea, so the morning was brainstorming. I narrowed it down to education because of my own background, then spent most of the day building.

What I'll remember most:

• Building alongside coding agents. I started with Gemini CLI, which hit its usage limit partway through, so I switched to working with Claude. It changed how I worked: I spent my time on design decisions and testing with real handwritten problems, and less on typing boilerplate.
• The sponsor talks paid off. I hadn't planned to use Nimble until the sponsor presentation. It ended up being the feature I'm proudest of.
• Real-world friction. The venue wifi wouldn't let my phone reach my laptop, my Gemini quota ran out, and a government server was very slow. Each one forced a better design.

The 5 PM deadline was tight. I froze features at 3:45, then wrote the README, recorded a demo, and submitted with minutes to spare.

Thanks to MLH and the sponsors for a great day of building.