RememberMe CareGrid: Local Gemma 4 for dementia memory and safety

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

What I Built

I built RememberMe CareGrid, a local AI dementia-care prototype powered by Gemma 4 E2B running through Ollama.

I built it as a solo participant, from the dashboard and backend to the Android relay, local model pipeline, and demo flow.

The project is built for people living with dementia, Alzheimer's, or memory loss, and for the caregivers who support them every day.

The moment I designed for is painfully ordinary: Rajamma steps outside, becomes confused, and cannot remember where she is, why she left, or who is standing in front of her. Her caregiver is not physically beside her. A neighbour may want to help, but should not receive private medical details. Later, a doctor may need a clean timeline, but the caregiver may only remember fragments.

RememberMe CareGrid turns that moment into a coordinated care flow.

It gives the patient calm memory context, helps caregivers understand what happened, and lets trusted community helpers respond with only the information they need.

This is not a diagnosis tool. It is not surveillance. It is a consent-aware memory and safety network.

The working prototype includes:

• Lumo Companion for patient-facing memory cues
• Trusted-person enrollment and recall
• SafePath geofencing for wandering safety
• SOS escalation with location and risk context
• CareCircle task coordination for trusted helpers
• CareLearn training cards generated by Gemma 4
• Reward wallet gamification for community participation
• Doctor Brief generation for clinic-ready summaries

The guiding idea is simple:

Help patients remember less alone, and help caregivers care less alone.

Demo

Direct video link: https://youtu.be/Lz3u-ljxhvA

The demo shows the dashboard, Lumo Companion, local Gemma 4 reasoning, SafePath, CareCircle, CareLearn, trusted-person recall, reward wallet, and Doctor Brief generation.

Code

ladiesmans217 / RememberMe-Caregrid

RememberMe CareGrid

AI memory, wandering safety, community care coordination, and Gemma 4-powered dementia training for Indian families.

RememberMe does not just track dementia patients. It gives them back context: who they met, where they are, what was said, and who can help.

Problem

Dementia care in Indian families is usually handled by one exhausted caregiver. Patients may forget trusted people, recent conversations, medicine routines, and safe routes. Existing tools often stop at GPS tracking or family-only reminders.

Solution

RememberMe CareGrid creates a consent-aware memory layer around a dementia patient. It recognizes enrolled trusted people, remembers meaningful conversations, maps safe places, alerts caregivers during wandering risk, coordinates community helpers, generates CareLearn training resources, and prepares doctor-ready summaries.

Community Impact

The product is built around three rings of care:

• Patient: Lumo Companion, SmritiLens, memory cues, SafePath.
• Family: caregiver dashboard, alerts, timeline, doctor brief.
• Community: neighbour, ASHA worker, pharmacy partner, RWA volunteer, student…

View on GitHub

Repository: https://github.com/ladiesmans217/RememberMe-Caregrid

A few places I would point reviewers first:

• src/lib/ai/gemma.ts - Gemma/Ollama reasoning wrapper
• src/lib/ai/ollama.ts - local Ollama helper and provider behavior
• src/app/api/watch/talk/route.ts - watch-style text conversation route
• src/app/api/watch/talk-audio/route.ts - local audio-to-care-response path
• src/lib/watch-talk-response.ts - structured response sanitization
• src/app/api/generate-doctor-report/route.ts - doctor brief generation
• src/app/api/carelearn/generate-training-card/route.ts - Gemma 4 CareLearn cards
• src/components/app-shell.tsx - dashboard shell and care navigation

How I Used Gemma 4

Gemma 4 is not used as decoration here. It is the local reasoning layer behind the care workflows.

I used Gemma 4 E2B because this use case needs local, privacy-conscious intelligence more than it needs the largest possible model. Dementia care involves sensitive context: confusion episodes, family relationships, trusted people, location, caregiver notes, and safety status. Sending every interaction to a remote model is not the architecture I wanted to demonstrate.

The core pipeline is:

patient input
  -> local transcription or text
  -> Gemma 4 E2B via Ollama
  -> validated structured response
  -> dashboard / phone / watch-style output

Gemma 4 powers:

• patient-facing memory responses
• structured Lumo conversation output
• SafePath safety guidance
• caregiver summaries
• CareLearn training cards
• doctor-ready weekly briefs
• care reasoning and action recommendations

For example, when the patient asks "Who am I?" or "Where am I?", Gemma 4 receives the care context and generates a short, calm response. When SafePath detects that the patient is outside the safe zone, Gemma 4 helps produce a reassuring safety cue instead of a cold alert. When CareLearn needs a neighbour-facing training card, Gemma 4 turns the situation into practical instructions.

The model output is not blindly trusted. Patient-facing responses and actions go through structured JSON handling and sanitization because this is a safety-sensitive workflow. The app expects fields like reply, intent, risk level, action, and escalation state. If the model output is malformed, the app falls back to safer behavior instead of breaking the care loop.

Why I Chose E2B

I chose E2B intentionally. The patient-facing workflow needs short, calm, structured responses, not the largest possible model.

A 31B Dense model would be useful for heavier reasoning, but it would make the local-first care loop slower and harder to run on modest hardware. E2B fits the product goal better: private local inference, fast enough responses, and enough reasoning ability for structured care actions.

For this project, the best model was not the biggest model. It was the model that made local dementia-care assistance practical.

Code Snippets

1. Local Gemma 4 through Ollama, forced into JSON mode

Source: src/lib/ai/ollama.ts

This is the core local model contract. The app calls Ollama locally, keeps Gemma warm, disables streaming for structured route responses, and asks the model for JSON so downstream patient actions can be validated.

export async function generateJson<T>(prompt: string, fallback: T): Promise<T & AiProviderMetadata> {
  const started = Date.now();

  try {
    const res = await fetchWithTimeout(`${OLLAMA_URL}/api/generate`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        model: OLLAMA_MODEL,
        prompt,
        stream: false,
        format: "json",
        keep_alive: OLLAMA_KEEP_ALIVE,
        options: { temperature: 0.2, num_predict: 512 }
      })
    }, OLLAMA_TIMEOUT_MS);

    const data = (await res.json()) as OllamaGenerateResponse;
    const text = data.response || data.message?.content || "";
    if (!text.trim()) throw new Error("Ollama returned an empty JSON response");

    const parsed = JSON.parse(extractJson(text));
    return withMetadata(parsed as T, {
      _provider: "ollama",
      _model: data.model || OLLAMA_MODEL,
      _elapsed_ms: Date.now() - started
    });
  } catch (error) {
    return withMetadata(fallback, {
      _mock: true,
      _provider: "ollama",
      _model: OLLAMA_MODEL,
      _error: errorMessage(error),
      _elapsed_ms: Date.now() - started
    });
  }
}

2. Watch audio becomes local transcription, then Gemma reasoning

Source: src/app/api/watch/talk-audio/route.ts

Gemma 4 is not treated as an audio transcription model. The watch audio is transcribed locally first, then the transcript is sent into the same Gemma-powered care reasoning path.

async function processAudioTalk(input: AudioTalkInput, progress?: ProgressWriter) {
  progress?.("audio_received", {
    audio_bytes: input.audioBytes,
    mime_type: input.mimeType,
    asr_url: getAsrConfig().url
  });

  const asr = await transcribeLocalAudio({
    audio: input.audio,
    filename: input.filename,
    mimeType: input.mimeType
  });
  const transcript = asr.transcript.trim();

  if (!asr.ok || !transcript) {
    const code = asr.error ? "local_asr_failed" : "local_asr_empty_transcript";
    await logAudioFailure(input, code, `Local ASR failed for session ${input.sessionId}.`);
    return audioFailure(input, code, { asr: summarizeAsr(asr) });
  }

  progress?.("transcribed", { transcript, asr: summarizeAsr(asr) });
  progress?.("thinking", { message: "Thinking with local Gemma" });

  return handleWatchTalkTranscript({
    transcript,
    patientId: input.patientId,
    sessionId: input.sessionId,
    route: "/api/watch/talk-audio",
    inputMode: "audio",
    diagnostics: { audio_bytes: input.audioBytes, asr: summarizeAsr(asr) }
  });
}

3. Gemma output is sanitized before it can affect the patient flow

Source: src/lib/watch-talk-response.ts

For a dementia-care workflow, the model cannot be allowed to return arbitrary action strings. This sanitizer constrains the reply, intent, risk level, action, and session-ending flag before anything is shown or executed.

export function sanitizeWatchTalkResponse(input: unknown, fallback: WatchTalkResponse): WatchTalkRouteResponse {
  const source = input && typeof input === "object" ? input as Record<string, unknown> : {};
  const reply = cleanReply(source.reply, fallback.reply);
  const intent = oneOf(source.intent, intents, fallback.intent);
  const risk_level = oneOf(source.risk_level, riskLevels, fallback.risk_level);
  const action = oneOf(source.action, actions, fallback.action);
  const should_end_session = typeof source.should_end_session === "boolean" ? source.should_end_session : fallback.should_end_session;

  return {
    reply,
    intent,
    risk_level,
    action,
    should_end_session,
    ...metadata(source)
  };
}

function cleanReply(value: unknown, fallback: string) {
  const reply = typeof value === "string" ? value.replace(/\s+/g, " ").trim() : "";
  return (reply || fallback).slice(0, 360);
}

function oneOf<T extends string>(value: unknown, allowed: ReadonlyArray<T>, fallback: T) {
  return typeof value === "string" && (allowed as ReadonlyArray<string>).includes(value) ? value as T : fallback;
}

4. The same Gemma helper powers care features beyond chat

Sources: src/app/api/carelearn/generate-training-card/route.ts and src/app/api/generate-doctor-report/route.ts

The same local Gemma helper is reused for two different care workflows: community training and doctor-ready summaries. This is the part that makes the project feel like a care system instead of a single chat screen.

// CareLearn: role-specific dementia-care training cards
export async function POST(request: Request) {
  const body = await request.json().catch(() => ({}));
  const role = (body.role || "neighbour") as Role;
  const useCase = body.use_case || "wandering";
  const eventContext = body.event_context || "Rajamma left safe zone near MSRIT gate.";
  const data = await generateJson(trainingPrompt(role, useCase, eventContext), mockTrainingCard);
  return NextResponse.json(data);
}

// Doctor Brief: clinic-ready caregiver summary
export async function POST(request: Request) {
  const body = await request.json().catch(() => ({}));
  const fallback = mockDoctorReport();
  const generated = await generateJson(doctorReportPrompt(body.state || {}), fallback);
  return NextResponse.json({
    ...fallback,
    ...generated,
    id: generated.id || uid("doctor_report"),
    patient_id: generated.patient_id || DEMO_PATIENT_ID,
    created_at: generated.created_at || new Date().toISOString()
  });
}

Consent Model, Not Just a Promise

It is easy to say "this is not surveillance." The important part is designing the system so that statement is true.

RememberMe CareGrid uses a consent-aware flow:

• trusted people must be enrolled before they can be recognized
• unknown people are not labeled as identities
• unknown faces are not turned into memory profiles
• patient-facing capture flows ask for consent before saving names, photos, or transcripts
• helper tasks expose limited role-based information instead of full medical history
• safety alerts share location and risk context only when escalation is needed

That distinction matters. The phone relay can capture a frame, Firebase can sync care state, and Twilio can send SOS messages, but those tools are wrapped around a care workflow with enrollment, consent, retention, and role limits.

The goal is not to watch the patient. The goal is to reduce the time between confusion and safe help.

UX Choices for Dementia Care

I designed the AI responses around three rules:

• answer in one or two calm sentences
• never shame the patient for forgetting
• always give one safe next step

That is why Lumo does not behave like a normal chatbot. A long answer may be technically impressive, but it can overwhelm someone who is already confused.

The same principle applies to community helpers. They receive limited role-based instructions, not the patient's full private history. The product is designed to make help easier without exposing more information than needed.

Main Workflows

Lumo Companion

Lumo is the patient-facing companion. It gives short, respectful memory cues.

The patient can ask:

• "Who am I?"
• "Where am I?"
• "Who is Ananya?"
• "Who is in front of me?"

The response is intentionally not a long chatbot answer. In dementia care, the best answer is often the one that reduces fear and helps the next safe action happen.

Trusted-Person Recall

A caregiver can enroll trusted people. The system stores consented trusted-person embeddings.

If the patient cannot remember who is standing in front of them, the phone relay captures a frame and compares it only against enrolled trusted-person embeddings. If there is a match, the patient receives a gentle cue.

Unknown people are not identified or stored. The system recognizes trusted people, not strangers.

SafePath

SafePath checks whether the patient is inside or outside a safe zone.

If the patient leaves the intended radius, the system creates a risk event and can trigger escalation with location context. The patient message remains calm, for example:

You are outside the safe zone. You are not alone. Please stay near a familiar place. Ananya is being contacted.

SafePath is not only a map. It is the part of the system that turns location into a care decision: reassure the patient, notify the caregiver, and give helpers enough context to respond safely.

CareCircle

CareCircle turns alerts into role-based tasks.

The important design choice is that not everyone gets the same information. A caregiver may need a detailed event timeline. A neighbour may only need a simple request: "Rajamma may need help near the temple gate. Please contact Ananya." A community health worker may need visit context. A pharmacy partner may need refill-related context. A local safety volunteer may need an emergency protocol.

This is where RememberMe CareGrid becomes more than a patient app. It becomes a coordination layer for trusted care.

CareLearn and Rewards

CareLearn uses Gemma 4 to generate role-specific dementia-care training cards.

The training is tied to real care moments. If a wandering event happens, the system can generate a card for a neighbour explaining how to approach calmly, what not to say, and when to notify the caregiver. If medication confusion happens, it can generate a different card for a pharmacy partner or caregiver.

The reward wallet adds a small gamification layer. Helpers earn points for completing training and can redeem them in the demo for care-related rewards such as pharmacy support, safety kits, or caregiver respite vouchers.

The point is not points for the sake of points. The point is to make community readiness visible and repeatable.

Doctor Brief

Doctor Brief converts scattered care events into a clinic-ready summary.

It can include medication adherence, wandering events, known visitors, memory journal topics, health snapshots, caregiver notes, community health worker notes, and suggested discussion points. The goal is to help a caregiver walk into a doctor visit with structured context instead of relying on memory during a stressful appointment.

Technical Architecture

The architecture is split around responsibility: Gemma 4 reasons, local transcription handles speech, Firebase synchronizes care state, the Android relay handles camera capture, and the app validates every model response before it becomes a patient-facing action.

The prototype uses:

• Next.js for the web dashboard
• Firebase for live care state, cues, GPS events, alerts, relay state, and training progress
• Android phone relay for trusted-person recall
• Gemma 4 E2B through Ollama for local reasoning
• local speech-to-text before Gemma reasoning
• face embeddings for consented trusted-person matching
• SafePath geofencing
• Twilio for SOS-style call and SMS workflows
• structured JSON validation for patient-facing responses and actions

This is not only a UI mockup. The demo connects model reasoning, phone relay, live sync, geofence safety, community tasking, training, rewards, and doctor summaries into one care network.

What Was Hard

The hardest part was not calling Gemma 4. The hard part was making local Gemma 4 behave like one component inside a safety-sensitive product.

The first design problem was speech. A remote multimodal API can hide transcription and reasoning inside one call. Gemma 4 E2B in this local setup should not be treated as an audio transcription model. So I split the pipeline: local speech-to-text first, then Gemma reasoning over clean text and care context. That made the system more honest and easier to debug.

The second problem was structure. A normal chatbot answer is not enough here. The watch flow needs to know whether the response is just an answer, a caregiver notification, a camera-frame request, or an end-session action. That is why the output is constrained and sanitized before the app uses it.

The third problem was timing. In the trusted-person recall flow, the phone relay, dashboard state, and watch-style cue delivery all have to line up. A result that arrives too late is not useful to the patient. I had to treat latency and state sync as product problems, not just backend problems.

The main lesson was that assistive AI is not only about model capability. It is about the boundary around the model: what context it receives, what it is allowed to output, and how the product behaves when something fails.

What I Would Improve Next

The next version would replace the phone relay with a low-cost wearable camera module, improve offline deployment on mobile or wearable hardware, refine localized voice UX, expand caregiver-controlled consent settings, and validate the workflows with elder-care groups, clinics, and community health workers.

I would also continue optimizing local Gemma 4 inference so the patient-facing experience feels faster on modest hardware.

Closing

RememberMe CareGrid is built around one belief:

Dementia care should not depend on one exhausted caregiver remembering everything alone.

Gemma 4 makes it possible to build local, private, useful intelligence around people who need support in the moment.

This prototype proved to me that local AI can be more than a private chatbot: it can become a careful coordination layer between a patient, a caregiver, and trusted helpers.

RememberMe CareGrid helps patients remember less alone, and helps caregivers care less alone.