DEV Community: Jeman

আইতাৰ কথা (Aitar Kotha): I fine-tuned an open model to name plants the way my grandmother does

Jeman — Sun, 11 Oct 2026 15:33:47 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

What I Built

Ask ChatGPT what Ziziphus mauritiana is and you'll get "Indian jujube". Ask my Aita (grandmother) and you'll get বগৰী, bogori, plus a warning not to eat one before Saraswati Puja, and a reminder that the thorns curve backwards so you pick from the outside of the tree.

That second answer is what we lose when our field guides are written for somebody else. So I built Aitar Kotha (আইতাৰ কথা, "Aita's words"): a field guide for the plants of Assam that answers the way an Assamese grandmother would. Then, every time, it tells you to put the phone in your pocket and go and find the plant.

📷 Show Aita a leaf. An open vision model picks the plant from the photo.
🗣️ Or ask by name: bogori, sewali, ou tenga, or type it in Assamese script.
🌾 "In season now" shows what to look for this Assamese month (it's কাতি, Kati, as I write this: sewali flowers, amlokhi and robab tenga).
অ Everything works in English or অসমীয়া, with one tap.
⭐ Achievements you can only earn outside. Each plant you scan with the camera earns its badge. Collections like টেঙাৰ দল (the sour ones) and বিহুৰ গছ (Bihu plants) add up to আইতাৰ তৰা, Aita's Star, for all 43.
🔊 শুনা · Listen. With an ElevenLabs key, Aita reads the card aloud in Assamese, so you never have to look at the screen at all.

Demo

Two narrated walkthroughs, recorded with the real fine-tuned Aita and real Gemma running on my laptop:

▶️ Demo in English (1:05)
▶️ অসমীয়াত, in Assamese (0:58), with the whole app switched to Assamese

Both are narrated by Meta's open MMS-TTS model running on my laptop, which has an Assamese voice. Even the narration is open source.

Try the demo: https://jemankalita.github.io/aitar-kotha/ (works on your phone, in English or অসমীয়া). The live demo answers from the family notes; the open AI (fine-tuned Aita and photo scanning) runs when you run it yourself, which is the whole point: your family's knowledge, on your own device.

Code

jemankalita / aitar-kotha

আইতাৰ কথা: an Assamese plant field guide that talks like your grandmother. Fine-tuned open model (Tinker + Qwen), offline-first, bilingual, with badges. Hacktoberfest Open-Source AI Challenge Week 1.

আইতাৰ কথা · Aitar Kotha

Plants of Assam, the way Aita (grandmother) told it. Point your phone at a plant, or just say its name, and a small open-weight model fine-tuned on Assamese family plant knowledge answers like an Assamese grandmother would:

বগৰী — bogori (Indian jujube, Ziziphus mauritiana) A thorny small tree with small shiny oval leaves… Eaten raw with salt and chilli, or made into sweet-sour bogori achar dried in the winter sun Many families tell children not to eat bogori until after Saraswati Puja… Now put the phone away and go find one.

Built for the Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass.

Live demo: https://jemankalita.github.io/aitar-kotha/. The demo answers from the family notes in your browser (names, seasons, English ⇄ অসমীয়া, achievements). The open AI, meaning the fine-tuned Aita and photo scanning, runs when you run it yourself (below). That's the point: your…

View on GitHub

How I Built It

1. The knowledge comes from family, not the internet

data/plants.json holds 43 plants as my family knows them: the Assamese name in script and in roman letters, the English and scientific names, the season, how to recognise the plant, how we use it, one story, and any warning. data/plants_as.json holds the same notes in Assamese.

Plant knowledge in Assam lives in conversation, not on the web. It's in what an Aita says while pointing at a bush, or in which leaves go into which dish at Bihu. Books gave me the scientific names; the uses, the stories and the warnings are the part only family has. So every entry carries a verified flag, and the app shows a "not yet checked with elders" label on any card where it's still false. A note only counts once the family has heard it read back in Assamese.

Traditional remedies are written down as culture, not as medical advice, and any plant with a dangerous look-alike carries a warning.

Each card also shows a photo of the plant: 41 freely licensed photos from Wikimedia Commons, each credited to its photographer on the card.

2. Teaching an open model to answer like Aita (Tinker)

I turned the notes into 383 practice conversations and LoRA-fine-tuned Qwen3.5-4B on Tinker. The fine-tune trains only on Aita's replies, and the answer pattern is always the same: Assamese name first, then how to know the plant, how families use it, one story, any danger, and finally "now put the phone away." I also trained it to say "I don't know" about plants that aren't from Assam, and never to tell anyone to eat something they can't name for sure.

training_client = service.create_lora_training_client(base_model="Qwen/Qwen3.5-4B", rank=32)
renderer = get_renderer("qwen3_5", training_client.get_tokenizer())
data = [conversation_to_datum(c, renderer, max_length=2048,
        train_on_what=TrainOnWhat.LAST_ASSISTANT_MESSAGE) for c in conversations]
for step, batch in enumerate(batches):
    training_client.forward_backward([data[i] for i in batch], "cross_entropy")
    training_client.optim_step(tinker.types.AdamParams(learning_rate=learning_rate(step, total, 2e-4)))

Then I tested both models on 172 questions the model never saw, phrased differently from the training ones (e.g. "Koka told me to look for bogori today. What is it?", or বগৰীৰ বিষয়ে কোৱাচোন।):

	Base Qwen3.5-4B	Aita (fine-tuned)
Assamese script name	31%	77%
Roman Assamese name	42%	78%
Scientific name	40%	85%

The training run was 96 steps (LoRA rank 32, 4 epochs over 383 chats); loss fell from 31.1 to about 0.5. Neither model got the family notes during this test: this is only what the weights learned.

My favourite example: asked "Koka told me to look for sewali today. What is it?", base Qwen started reasoning aloud and guessed Sesbania grandiflora, the wrong plant. Aita answered শেৱালি, sewali (Night-flowering jasmine, Nyctanthes arbor-tristis). Look under the bush at dawn: small white flowers with bright orange tubes carpet the ground.

Honest caveats: answers were capped at 400 tokens and the base model spends many of them thinking out loud, so its score is a little pessimistic. And Aita isn't perfect from memory alone: asked about bogori with no notes, she got the name, season and uses right but described the bark of a different tree. That's why the app always hands her the family notes too.

Photo scanning, honestly

Gemma 3 (4B) runs on my laptop's 4 GB GPU, about 20–25 seconds per photo. On six test photos it got four right: sewali, tulsi, star fruit, ou tenga and water spinach were each identified at least once. Its self-reported confidence was 95–100% even when it was wrong, so I don't trust that number. A badge only appears after a scan, and Aita says so when she isn't sure.

3. The walking app

A small FastAPI server and a phone-first web app. When the fine-tuned model is unreachable, the app answers straight from the family notes, and a service worker keeps those notes on the phone so you can still ask by name on a trail with no signal. Photos are downscaled on the phone and their real file type is checked before anything reaches the vision model.

Why open matters for this

You can't fine-tune a closed model on your grandmother's knowledge and keep the result. Here the weights are mine: an adapter of a few megabytes that I can download and run in Ollama on my own laptop.
Small languages deserve better than "close enough". Assamese plant names, folk uses and Bihu customs are thin on the web, which is exactly where general models guess. With an open model I could teach it the real names, and test whether it learned them.
It runs without the cloud. The notes, the app and the exported model all work offline, so nothing about where you walk leaves the phone.
Honest about the one closed piece: the Assamese voice uses ElevenLabs. It's optional; without a key the Listen button just doesn't appear.

Built for outside

The app is designed so the screen is the shortest part of the walk:

The answer ends by sending you away. Every reply from Aita closes with "now put the phone away" and tells you what to look for: the curved thorns, the smell, the colour of the flower tube.
Badges only come from the camera. Typing a name teaches you something; earning the badge means you stood in front of the plant and took its photo.
It works with no signal. A service worker keeps all 43 plants' notes on the phone, so asking by name works in a bari or by a pond with no network.
It can speak. With an ElevenLabs key, "Listen" reads the card aloud in Assamese, so the phone can stay in your pocket.

I tested the scan on real photographs of the plants, not illustrations: tulsi, sewali, star fruit, ou tenga and water spinach were all recognised at least once by Gemma running on my own laptop, and the honest misses are in the section above. The open model is what makes a phone-in-pocket walk possible at all: there's no server to reach, and no record of where you walked.

What's next

More plants, and the names other communities of Assam use (Bodo, Mising, Karbi…)
Let elders record their own voice for each story
A printable pocket card for walks with no phone at all

What I read along the way

Building an Open-Source Indian Address Parser: From Raw MCA/Bank Data to a Fine-Tuned LLM by Gagan: a LoRA pipeline for Indian data, end to end
Translating 50,000+ Indian village names into six scripts by mchittineni: why English-only data fails rural India
LoRA Doesn't Approximate Fine-Tuning. It Changes What You Can Teach: useful framing for what a small adapter can and can't learn

Prize categories

Best Use of Tinker: Tinker LoRA-fine-tuned Qwen3.5-4B into Aita (383 conversations, 96 steps), evaluated on 172 unseen questions: the Assamese-script name went from 31% to 77%.
Best Use of Gemma: Gemma 3 (4B) does the photo scanning, entirely on my laptop through Ollama.
Best Use of GitHub Copilot: I used Copilot in VS Code to write the entire codebase.

Built with Github Copilot as my pair programmer.

HandNotes: teaching an open model to read my friend's handwriting, on his own laptop

Jeman — Sun, 04 Oct 2026 23:55:25 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My friend writes down everything in class. Every lecture, every assignment, page after page. The problem shows up the night before an exam: he photographs his notes, uploads them to an AI chatbot, asks for a summary, and gets nothing useful back. It can't read his handwriting.

When I ran his pages through a vision model myself, I found something worse than "can't read it". The model read most of it, and the mistakes looked right:

The Perennial Student (the book he was writing about) became The **Peripheral* Student*
"satirical" became "spatialical"
"his reputation" became "her reputation"
a date in a letter disappeared, and "5:00 to 7:00 pm" became "8:00 to 8:00"

And then there's his shorthand. He writes "acc" for according and "gitq" for given in the question, which no model is ever going to guess.

A tired student skimming a summary at 2 AM would never catch any of that. So I built HandNotes: a handwriting reader that runs entirely on his laptop, lets you correct what it got wrong, and remembers those corrections for the next page.

What it does:

Reads a photo of his notes live. Text streams in word by word from an open vision model running locally.
Highlights words it isn't sure about, with one-click jumps to each one so checking a page is fast.
Learns from corrections. Every fix is compared word by word with what the model wrote. His vocabulary and past misreads go into the prompt for the next page, and a misread you've corrected twice gets fixed automatically.
Knows his short forms. Teach it "gitq = given in the question" once, either by adding it or just by expanding it while correcting a page, and it expands it on every page after that.
Shows its work. An accuracy chart per page, a list of his usual misreads, and a "compare with plain model" view that shows, word by word, what the memory changed.
Turns notes into revision material: a summary with likely exam questions, flip-to-reveal flashcards, and export to PDF, Word or text.

What he said: when I showed it to him, he really liked it, and he told me he's going to use it on a regular basis. Coming from the person who photographs his notes because typing them up is too much effort, that's the review I was hoping for.

Demo

It runs locally (that's the point), so the GIF at the top is a real run recorded on my laptop, sped up 4x: adding his short forms, reading one of his pages live, comparing against the plain model, and making flashcards.

Plain model vs. with his memory, on the same page. Struck through is what the plain model read; green is what it read with his memory:

The revision summary, generated locally from the transcript:

Code

jemankalita / Hacktoberfest-Build-for-a-Friend

A handwriting reader that learns one friend's handwriting from corrections. Gemma 3 via Ollama, runs fully offline on a laptop.

HandNotes

A handwriting reader that learns one person's handwriting, running entirely on your own laptop.

My friend writes down everything in class, then photographs his notes the night before an exam and asks an AI chatbot to summarize them. It can't read his handwriting. And when a model does read most of it, the mistakes look right: in our first test a book title turned into a different word, "his" became "her", and a date quietly disappeared.

HandNotes reads his notes with an open vision model (Gemma 3 via Ollama) on his own machine, lets you fix what it got wrong, and remembers those fixes so the next page comes out better. Nothing is uploaded anywhere.

Built for the DEV Hacktoberfest Weekend Challenge: Build for a Friend.

Real run at 4x speed: adding his short forms, reading a page live, comparing with the plain model, and making flashcards.

Features

…

View on GitHub

Setup is three commands: ollama pull gemma3:4b, pip install -r requirements.txt, python server.py.

How I Built It

The model: Gemma 3 4B, an open-weight vision model, served locally by Ollama. On my laptop (GTX 1650 Ti, 4 GB) Ollama splits it about half GPU, half CPU, and a page takes 30 to 40 seconds.

The app: a small FastAPI server bound to 127.0.0.1, with a hand-written HTML/CSS/JS front end. Fonts and icons are bundled, so even the UI makes no network requests.

The prompt matters more than I expected. My first prompt let the model "tidy up" his notes: it dropped a whole line with a date in it. Telling it to copy literally, skip nothing, and mark unclear words with [?] brought the date back.

The learning is a memory, not retraining. Fine-tuning a vision model in a weekend on a 4 GB GPU wasn't realistic, so HandNotes keeps a small per-writer memory file:

The model transcribes the page.
You fix the transcript and save.
A word-level diff (difflib) between the model's output and your fix records each misread ("Peripheral" → "Perennial"), the words he uses, and the page's accuracy.
On the next page, those words and misreads are added to the prompt, and misreads confirmed at least twice are replaced automatically. Short words like his/her are never auto-replaced, because they depend on context.
Short forms are detected separately. If one short word gets replaced by a longer phrase that starts with the same letter and contains its letters in order (gitq → given in the question), it's stored as his shorthand. The model is told to copy those exactly as written, and HandNotes expands them after every read.

How well does it work? I ran four of his pages in order, letting the memory build up, and scored each against a transcript checked word by word against the photo. For pages 2 to 4 I also read the page with the plain model for comparison:

Page	With memory	Plain model
1. Literature assessment	94.1%	(no memory yet)
2. Research overview	95.5%	95.5%
3. Formal letter	96.0%	95.2%
4. Literature assessment	90.1%	91.9%

That table isn't the neat upward line I hoped for, and it taught me more than one would have:

The memory does fix names and terms. Re-reading the first page with memory, "Peripheral" became "Perennial" and "spatialical" became "satirical".
It can also learn the wrong lesson. My friend spells "behavior" on one page and "behaviour" on another. The memory learned one spelling and "fixed" the other into it, which is why page 4 came out worse with memory. On another re-read, it turned a correct "Coles" into "Cole".
Pronouns are the stubborn error. Even when told to copy literally, the model kept turning "his" into "her".
Errors travel. On one run the model invented a word at the end of a page ("distressed"), and it flowed straight into the generated summary. That's why the review step comes before the study tools.

I also found a bug in my own scoring along the way: curly and straight apostrophes (student’s vs student's) were counted as misreads. I fixed it with a test and re-ran everything, so the numbers above are from the corrected version.

The next step is clear from that list: let the user mark a change as "his slip, don't learn this" versus "the model misread this", and flag words he genuinely uses instead of auto-replacing them. After that, I want to cut each corrected word out of the page and use that library to fine-tune a small open handwriting model on his writing specifically.

Why Does Open Innovation Matter?

For this project, open isn't a nice-to-have. It's the reason the project can exist at all.

It can be taught one person. A hosted chatbot can't be taught my friend's handwriting or his shorthand. An open model I run myself, with a memory I control, can. And because every correction is stored locally, his own handwriting dataset builds up as he uses it. With an open model I can actually fine-tune on that later.
His notes stay his. These are photos of his assignments and personal letters. Nothing is uploaded anywhere: the server only listens on 127.0.0.1.
It works where he studies. No internet needed once the model is downloaded, which matters in a hostel room the night before an exam.
It costs nothing to run. No subscription and no API bill. For a student who's already "too lazy" to type up his notes, anything that costs money or effort won't get used.
I could see and change everything. When the model dropped a line, I could rewrite the prompt. When the memory learned the wrong lesson, I could see exactly why in a JSON file. I could swap models with one environment variable.

My Agent Session

I built this pairing with Claude Code over one long night. It helped with the code, the tests (37 of them, covering the memory, short forms, the API with the model mocked, and the exporters), the demo recording, and the measurements above.

Prize Categories

Best Use of Gemma: Gemma 3 4B does all the reading, summarizing and flashcard generation, running locally through Ollama.