Picture a busy hospital OPD in India and the first five minutes of a consultation. The doctor types. The patient waits. Now make that patient a 70-year-old who is most comfortable in Telugu, speaking the half-Telugu, half-English mix that real consultations sound like. A text-first app can be a wall for people like that.
In an emergency, typing may not be an option at all. So I built AyuSwar (Ayu means life, Swar means voice) to explore one question: how far can voice alone take a health app? Speak to it in your language, and let it listen, understand and answer back.
Honesty up front: AyuSwar is a prototype built on synthetic data. It is not a medical device, it is not clinically validated, and it does not give diagnoses. In a real emergency, dial 112 or 108.
The architecture: Listen, Keywords, Risk, Guidance
Everything in AyuSwar hangs off one 4-stage pipeline. Each stage carries a status chip, and I want to be clear about what those chips are: they are my own labels for where each stage stands, not an independent audit. Here is what they currently say:
- Listen (LIVE): ambient doctor-patient audio intake through the browser Web Speech API, with a local AudioContext FFT running alongside for signal analysis.
- Keywords (DEMO): a hand-built mapping links colloquial vernacular idioms to WHO ICD-10 and SNOMED CT concept codes.
- Risk (DEMO): non-diagnostic acoustic signal checks and triage urgency flags. Rules, not models, wherever safety matters.
- Guidance (PLANNED): HL7 FHIR R4 document bundles and spoken prescription instructions in the patient's language.
The live /technology page draws this pipeline in real time, with LIVE, DEMO and PLANNED chips on each stage.
The hardest part: what patients actually say
Many patients do not say "I am experiencing palpitations". In my native Andhra, someone is more likely to say "గుండెల్లో దద" (gundello dada). So one core piece of the app is a vernacular mapping table: regional idioms in Telugu, Hindi, Tamil, Malayalam, Kannada, Marathi and Bengali, linked to standard clinical concept codes. This is a real excerpt from the repo:
// src/data/indicVernacularOntology.js (real excerpt, trimmed)
{
id: 'ont_01',
vernacular: 'గుండెల్లో దద', // Telugu, Rayalaseema / Coastal Andhra
transliteration: 'Gundello Dada',
clinicalTerm: 'Palpitations & Cardiac Flutter',
icd10: 'R00.2', // WHO ICD-10
snomed: '80313002' // SNOMED CT
}
Two things to be clear about. First, this is a developer-authored mapping. I am a student builder, not a clinician, and the table is unvalidated: it is a structure waiting for clinician review, not an authority on symptoms. Second, that is exactly why it is plain data anyone can read and challenge on GitHub, instead of a black box.
Code-mixing is the default, not the edge case. A real sentence sounds like "doctor garu, naa chest lo burning sensation undi". Anything that only understands clean English or clean Telugu will fail people who speak like this, so the language handling assumes mixed input.
Audio-Rx and the rest of the system
The Audio-Rx studio turns a prescription into spoken instructions in the patient's own language, with word-by-word karaoke highlighting over colour-coded pill cards. It can then format the schedule as text and open WhatsApp with it ready to send; the send itself is always a human tap. The OPD Ambient Scribe drafts a structured SOAP note from a consultation conversation and is designed around a rule the site states openly: a doctor reviews before anything is exported. The Acoustic Signal Lab measures physical signal properties in the browser (F0, spectral centroid, energy flux) and labels itself on the page as non-diagnostic. The Sanjivani-108 emergency console triages a spoken distress call and prepares a hospital handover, with dispatch clearly marked as simulation and a 110 BPM CPR metronome on screen.
The Audio-Rx studio with sample medication cards - synthetic demo data, not real prescriptions and not medical advice. On-screen wording like "60 FPS" and "direct WhatsApp dispatch" is the site's own UI copy: aspirational, not measured performance, and the WhatsApp send is always a human tap.
The stack
| Layer | Choice |
|---|---|
| Front end | React 19 + Vite on Vercel, installable PWA |
| Voice | Web Speech API; Gnani Prisma/Timbre adapters wired behind a browser-speech fallback (provider keys pending verification) |
| Acoustics | Web Audio API FFT in the browser: F0, spectral centroid, energy flux |
| Clinical vocabulary | Developer-authored Indic vernacular mapping to ICD-10 + SNOMED CT codes (unvalidated, open for review) |
| Maps | MapLibre GL + OpenStreetMap hospital discovery |
| Scanning | Tesseract.js OCR for blister strips and prescriptions |
| Mobile app | Planned: Capacitor dependencies are in place for an Android build, nothing built yet |
| Backend | Supabase-ready client; falls back to local demo storage when keys are unset |
Testing the boundaries, not just the features
The most deliberate part of the codebase is the test suite, because much of it checks what the app must NOT do. Alongside the repo's Node test files, the /benchmarks page defines a contract-verification suite that runs in the browser against the app's own services; the assertions are written and wired, and I am still verifying a full green run end to end. They include: the assistant must refuse to prescribe drug doses, an emergency phrase must trigger a 112/108 warning, the acoustic analyzer must return isDiagnostic: false with a disclaimer, and no secrets may leak into the client bundle.
The /benchmarks runner is built to execute these assertions in the browser and export a software test report; a full verified green run is still in progress.
Most of my tests check what the app must never do.
What broke, and what I learned
- Test on a phone early. Most users will meet AyuSwar on a small screen, and the first mobile feedback I got was blunt and useful: pages overflowing, a floating widget covering content, a whole route rendering blank.
- Code-mixed speech is the default, not the edge case. Design for the half-Telugu, half-English sentence from day one.
- Ask for the harsh review. An engineer at Gnani AI sent me a detailed list of what was wrong: numbers I could not evidence and claims the app could not keep. I have been cutting them since, some cleanup is still in progress, and the project is better for it.
- Label reality. The simulated modules carry SIMULATION or DEMO labels, and the goal is a demo people can trust, not one that only impresses.
Honest limitations, and what is next
What is unfinished: the OPD scribe's full voice-capture pipeline is still under end-to-end test; the Gnani Prisma (speech-to-text) and Timbre (text-to-speech) adapters are wired with a browser-speech fallback while the deployed provider configuration is verified; and all patient data on the site is synthetic. The vernacular mapping needs review by someone with clinical training before I would trust it beyond a demo. Next steps are technical: finish the scribe pipeline, complete and verify the provider integration, expand the contract tests, and keep hardening the mobile layout.
Built for the Great Indian AI Internship Challenge 2026 by Gnani AI. All patient data shown is synthetic. #GreatIndianAIInternshipChallenge #GnaniAI
Top comments (0)