Mathaka (Memory): “Did I take my medicine?” - answered by open AI on an old phone

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

In many Sri Lankan homes, the hardest medicine question is not when. It is "did I already take it?"

I built this for my grandmother. She has mild dementia, and she often forgets her pills. Some days she misses a dose. Other days she isn't sure, so she asks us again and again: "Did I take my medicine?" We aren't always home to answer. I wanted her phone to give her that answer: kindly, the same way every time, and without guessing.

We tried pill reminder apps, but none of them fit her:

• They only say when. The alarm rings and she takes the pills. Ten minutes later she can't remember doing it. Then she either skips the dose or takes it twice, and both are dangerous.
• They are made for young people. Small text, menus, swipes and settings confuse her. One wrong tap and the reminder is gone.
• They don't speak her language. Most are in English only. She reads and speaks Sinhala.
• They don't know her pills. Her clinic gives loose tablets in small paper envelopes, often with no label. An app that shows drug names doesn't help when she can't match a name to a white tablet.
• They need the internet and an account, and they send her health data to someone's cloud. Her phone is old and cheap, and the signal at home isn't always good.

So I built Mathaka (මතකය, "memory" in Sinhala). It is a medicine helper for older people with memory loss. It works in Sinhala, Tamil and English. It is one Android app with two roles:

On the elder's phone (this can be an old, cheap phone):

• One gentle reminder, in the family's own recorded voice if they want.
• Only the pills for this time of day, shown as big pictures with what they are for ("sugar medicine"), never a drug name.
• A big button: "Did I take my medicine?" The app answers from its own log, in the same calm words every time: "Yes, Ammi. You took your morning medicine at 8:00. The next one is at lunch time."
• A double-dose guard. If she tries to take it again, nothing is logged twice, and the app tells her kindly that she already took it.
• Huge text, 7:1 contrast, a high-contrast theme, a real clock, "Say it again", and no swipes or double taps.

On the family's phone:

• Set up the medicines once, by scanning the prescription and photographing each pill.
• See the day at a glance: doses taken, missed doses, low stock, the clinic date and a week of history.
• Get alerts over home Wi-Fi, or by plain SMS when there is no internet.
• Print a "fridge card" with every pill on paper, for days when the phone is off.

There is also an optional Arduino pill box. The slot that is due lights up, and pressing it means "I took it".

Home	This slot's pills	"Did I take it?"	Family dashboard

Demo

The 4.5-minute video shows the elder phone, the family phone, how the AI works, the pill measuring, the Arduino box and the test results.

• Try it in a browser: clone the repo, then run npm install and npm run web. Demo mode fills in a sample family and a week of history.
• Try the Arduino box with no hardware: open the Wokwi ESP32 simulator. Paste in firmware/esp32/mathaka_box/mathaka_box.ino and diagram.json, then type LIGHT 1 in the serial monitor.

Code

Bin4yi / Mathaka

Mathaka (මතකය)

A dementia-friendly medicine helper for older people in Sri Lanka. Android first, Sinhala / Tamil / English. All AI runs on the phones, with open-weight models. There is no cloud.

A reminder is not remembering. Alarm apps say when. Mathaka also answers whether you already took it, and stops a second dose.

Two open AIs. Gemma can see. Laya decides but cannot speak. Grandma only hears words her family wrote.

Status: prototype for the DEV Hacktoberfest "Build for a Friend" challenge. Not a medical device. See Honest limits.

Demo video (4.5 min)

▶ Watch the demo (MP4). It covers the elder flow, the family phone, the on-device AI, pill vision, the Arduino box and the evaluation The video is made from HTML with HyperFrames. Its source is in videos/mathaka-promo.

Screenshots

Real screens from the web build (npm run web…

View on GitHub

It's an Expo / React Native app (TypeScript), a shared pure-TypeScript core, an ESP32 Arduino sketch, an optional Python laptop helper, and an evaluation folder with frozen test sets. It has 124 automated tests.

How I Built It

My rule was: heavy AI once, light AI every day, and no cloud anywhere.

1. Setup happens once, on the family phone, with Gemma

A family member photographs the prescription. Gemma 3 4B (vision, Q4_K_M GGUF) runs on the phone through llama.rn, which is llama.cpp for React Native. It reads the prescription into rows. It also reads the number printed on each pill and writes a short description for the family.

Clinic books are handwritten, so I don't trust the model blindly:

• Every answer must fit a JSON schema. It's checked with zod and retried once.
• A person confirms every row before anything is used.
• Gemma is never loaded on the elder's phone. The code refuses.

2. The pill is measured, not guessed

You put each pill on a printed mat with four ArUco markers and a grey patch. Then plain TypeScript (no big model) does the work:

• finds the markers and flattens the perspective
• converts pixels to millimetres
• fixes the colour using the grey patch
• cuts out the pill and measures its size, shape, colour, two-colour halves and score line

For the demo pill it measured 19.2 × 9.2 mm, oblong, white, with a score line, in about 200 ms. The same code runs on the phone, in the browser and in the tests.

3. One file moves the setup to the elder's phone

A second person must approve the whole setup. Then it is saved as one file (.mtk, a zip with a SHA-256 hash for every part). You can send it by Bluetooth, a USB cable or WhatsApp-to-yourself. The elder's phone rejects any file that is broken, changed, older or not approved. There's no account and no server.

4. The AI that is not allowed to speak

This is the part I care about most. No AI ever writes the words the elder hears. Every sentence is a fixed template in Sinhala, Tamil or English, filled in with facts from her own log. A unit test blocks words like "take another".

For understanding what she means (like "mama beheth biwwada?", Singlish for "did I take my medicine?"), I tested Laya. It's an open, non-generative decision model: it can only pick one option from a short list and give a probability. It cannot write a sentence.

5. Alerts without a cloud

When a dose is missed, the family finds out:

1. over home Wi-Fi (the two phones are paired once with a QR code)
2. by plain SMS if Wi-Fi fails, with no mobile data needed
3. by retrying in the background until the alert gets through

6. The Arduino pill box

An ESP32 with 4 LEDs and 4 buttons talks to the phone over Bluetooth LE (or USB serial). The phone sends LIGHT 1, and the box sends back PRESS 1. The box never decides anything. A press goes through the same double-dose guard as the screen.

What the tests say

I froze the test sets before measuring.

Test	Result
"Did I take it?" answers (40 hand-written cases)	40 / 40
Double doses stopped	4 / 4
Unsafe sentences said to the elder	0
Pill matching, known pills found (synthetic photos)	100% (colour alone: 62%)
Pill matching, new pills rejected	100% (colour alone: 92%)

The honest part: Laya, without training, acted wrongly on about 1 in 4 Sinhala and Singlish questions. So it stays switched off. The elder's phone keeps big buttons, which can't misunderstand her. I'd rather ship buttons than a confident wrong answer to someone with dementia. I've prepared a fine-tune on Sinhala care phrases as the next step.

Why Does Open Innovation Matter?

• Privacy that is real. Medicine photos, the dose log and a family member's recorded voice never leave the two phones. With open weights I can check that, instead of trusting a privacy policy.
• It works in a village. After a one-time model download, everything runs offline. Alerts go by plain SMS. A cloud API stops working when the signal does.
• No monthly bill for a pensioner. Running the model costs nothing per use, and the elder's phone can be old and cheap.
• I could measure the risk. Because Laya is open, I could test exactly how often it would act wrongly before letting it near a vulnerable person. That test is the reason it's switched off.
• I could change what matters for us: Sinhala and Tamil templates, a pill pipeline built for loose clinic pills in paper envelopes, and later a model fine-tuned for Singlish that I can give back.

My Agent Session

I built Mathaka with Claude Code as my pair programmer, working on Windows with no Android emulator. We used the web build and headless Chrome to test the screens. The demo video was also made with code: HyperFrames HTML scenes, an offline Kokoro voice and MusicGen music, all in videos/mathaka-promo.

Prize Categories

• Best Use of Gemma: Gemma 3 4B vision runs on the family phone through llama.rn. It reads prescriptions and pill imprints and writes pill descriptions, always behind a JSON schema and a human check.
• Best Use of Arduino: the optional pill box is an Arduino sketch for the ESP32 (Bluetooth LE + USB serial), with a Wokwi diagram so anyone can try it without hardware.

Honest limits

• It's a prototype. It is not a medical device, and doctors haven't tested it.
• A pill photo can't safely identify a medicine. Mathaka uses it only as an extra check, after the schedule and the box slot.
• The pill numbers come from synthetic photos so far. I'm collecting real ones.
• It doesn't replace a carer. It gives the carer one less worry.

Credits

Thanks to the Gemma, llama.cpp / llama.rn, Laya, js-aruco2, Expo and HyperFrames teams. The photos in the video are from Wikimedia Commons: Ayena (public domain), Stevepb (CC0), trokilinochchi (CC BY 2.0) and Edwiyanto (CC BY-SA 4.0).