This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
What I Built
Madras Field Notes is an installable PWA that turns your phone into a pocket naturalist for Chennai — and it works entirely in airplane mode.
- Identify: Point your camera at a bird or plant. A CLIP zero-shot image classifier — running inside your browser, against a curated list of 15 species common to IIT Madras campus and the Pallikaranai marsh — tells you what you're looking at, with an honest "not sure" state below its confidence threshold.
- Learn: Every species gets a card from a hand-built knowledge pack: 3 facts, identifying features, and one safety note — in both English and Tamil. No screen time bloat; read the card, look back up.
- Ask: A Qwen2.5-0.5B open-weight model running locally in the browser tab answers follow-up questions ("Is this bird migratory?") grounded strictly in the species card — it cannot hallucinate edibility or medical advice because it never sees the open internet.
- Log: Identifications save to an on-device field journal, exportable as JSON for citizen-science projects.
Live demo: https://madrasfieldnotes.onrender.com (open it once online, then switch on airplane mode — everything still works)
Source: https://github.com/pkbhalla/MadrasFieldNotes
Demo flow: load once → airplane mode → identify a rose-ringed parakeet from a photo → read its card in Tamil → ask the local LLM a question → check the field log. Zero network requests after first load.
How I Built It
- Stack: Vite + TypeScript, no backend, no API keys, no analytics. Hugging Face Transformers.js v3 runs both models in Web Workers so the UI never blocks.
- Identify: Xenova/clip-vit-base-patch32 zero-shot image classification on WASM, against label phrases built from the knowledge pack.
- Chat: onnx-community/Qwen2.5-0.5B-Instruct at q4 quantization (~300 MB), with the species JSON injected into its system prompt as the only allowed knowledge source.
- Offline: Service Worker caches the app shell; the Cache API persists model weights; navigator.storage.persist() guards against browser eviction.
Where Open Beat Closed
This project only exists because the models are open — and my biggest failure proves why that matters.
1. I could see the real cost. Closed APIs hide their footprint behind a spinner. Open weights forced me to confront the physics: my first build loaded a 1B model's fp32 weights (~2 GB) and Chrome killed the tab the moment CLIP and the LLM were resident together. Android caps a browser tab around 1–1.5 GB no matter how much RAM the phone has. Because the weights were mine to inspect and swap, the fix was a real engineering tradeoff — q4 quantization, a smaller instruct model, identification-first design — not a support ticket. Chat now ships desktop-first, with mobile chat as the honest next step.
2. Zero per-query cost means zero reason to stay inside. No API bill means the app nags nobody, tracks nobody, and never degrades when a free tier ends. The user "closes" the screen because using it worked.
3. The knowledge pack is the forkable product. species.json holds Chennai's birds and plants with Tamil names. Any community — Coimbatore, Colombo, Kuala Lumpur — can fork the repo, swap 15 JSON entries, and ship their field guide, in their language, without asking anyone for permission or a bill. That's what "gets people off the screen and into the world" looks like at ecosystem scale.
4. Privacy is architectural, not a policy page. Photos of what you find on a trail never leave the device, because there is no server to send them to.
What I'd Do Next
- Shard-free sub-500 MB total budget so chat survives mobile tab limits
- Bird-call identification via the open-source BirdNET browser port
- Community-contributed packs (crowdsource species.json per region)
Built in ~24 hours for the hackathon. The screen is the shortest part of this experience — by design, and by memory ceiling.
Top comments (1)
grounding a 0.5b on the card alone is the right call. did you try webgpu for the chat model, or does wasm stay fast enough on the cheap phones people carry on a marsh walk?