DEV Community: Naomi Pereira

It scored 100%. Its note scored 0%

Naomi Pereira — Fri, 09 Oct 2026 20:31:44 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

I kept wiping the chat.

A model sees a hidden rule through examples. Four rounds, six items each, with the right answers after every round. Then it has 120 words. Then the conversation is deleted. A fresh reader gets the note and a set of questions it has never seen.

Each dot below is one of those episodes. Left to right: how the writer scored while the examples were still on screen. Bottom to top: how the fresh reader scored from the note alone. The orange dots are the ones that bothered me. The writer was at least 90 percent right. The reader, holding only the note, landed at 50 percent or worse.

The rules are made up. Strings, numbers, a tiny grammar, categories. Three difficulties each, one episode per cell, same grid for every model. The generator holds the answer key, so a model never grades another model.

Under every note I also ran two controls on the same questions. No note at all. And a perfect note, written from the true rule, capped at the same 120 words. Teaching Efficiency is the share of that gap the model's own note managed to close. A 1.0 means the note did what the truth did. A 0 means the reader would have done as well with a blank page. Under 0, the note pushed the reader the wrong way. A run that died halfway is left blank. It is not a zero.

For that teaching score the fresh reader is always Gemini 3.1 Flash-Lite. I wanted the note to work for a small model. A note that only its author can use is just a diary.

After that I flipped the setup. Every model read the whole pile, 320 notes, plus the perfect note and the blank, and answered the same questions. That score is Reading Efficiency. I only compare readers when the perfect note actually works for them.

Models Tested

Twenty-five models finished the teaching run. I wanted three kinds of company in the same table.

Neighbours. Sonnet 4.5 beside Sonnet 5. Opus 4.8 beside Opus 5. Gemini from 2.5 Flash up through the 3.x Flash line to Gemini 3.1 Pro.

An open model in with the hosted ones. Gemma 4 31B sits on the same chart as GPT-6 Astra and Gemini 3.5 Flash.

And the small ones that usually get left out of a write-up. GPT-5.4 nano, GPT-5.4 mini, both Flash-Lites, Gemma 4 26B.

Two names are missing, and I am not filling them in. gpt-oss-120b never got through a run. The provider was overloaded. Grok 4.20 non-reasoning rejected the reasoning setting everyone else used. They stay off the chart.

Findings

This is the teaching board. The bar is how much of a perfect note the model's own note was worth. Gemini 3.5 Flash is at the top, 1.02. Sonnet 5 is one of the short ones, 0.75, under several models that cost less and learned less.

Gemma 4 31B reached 99 for $0.15. Gemini 3.5 Flash is the only model that spent more and still lifted the score, to 102 at $0.78. GPT-6 Astra spent about a dollar to land where Gemma already was.

The note failed its own examples

Sonnet 5 learned the set at 0.92 and taught it at 0.75. On one number rule the truth is 4x − 5. The last practice rounds were perfect. Then it wrote that the rule was 4x − 1, listed 57→223, 1→−1, and 25→95, and said the examples match. 4×57 − 1 is 227, not 223. The fixed reader scored 0 out of 12. So did a fresh Sonnet 5, handed only that note. The rule was in the chat. It did not survive the paragraph.

Opus 5 went the other way. It learned at 0.78, the weak end of the frontier, and its notes scored 0.99. On both easy string rules the last practice round was 0 for 6. The note still got the small reader to 12 for 12. One note said to reverse the word. The other said where to move the first letter. A fresh copy of Opus, reading its own note, came out at 1.04 against the perfect note.

Ten episodes in the full set look like Sonnet's. The writer finishes practice at 90 percent or better. The reader, with the note, is at 50 percent or worse. Nano learned to reverse a word at 100 percent and described an interleave. The reader scored 0, below the blank-page score. A mini grammar note announced the wrong word order, then used the right order in its own example. The reader scored 0. DeepSeek-R1 learned a string rule cold. The small reader got 46 percent from the note. DeepSeek's own fresh copy got 100 percent from the same page.

Some notes are still thinking

Ten notes still have the scratch work in them. "Wait." "Actually." "Let's look." Those ten teach at 0.47. The other 288 teach at 0.91. On the four where the writer had already learned the rule, the scratch-work notes teach at 0.33. The notes that just say the rule teach at 0.97.

Flash-Lite had a reversal at 92 percent, then talked itself into a vowel swap. The reader scored 8 percent. With a blank page that same reader scored 31 percent.

Extra examples did little. Notes with a pair of worked examples taught at 0.90. Notes without them taught at 0.88. Shorter was fine. Gemma 4 31B averaged about 44 words and taught at 0.99. Sonnet 4.5 averaged about 94 and taught at 0.92. Sonnet 5 averaged about 65 and taught at 0.75.

Three notes landed more than ten points under the blank page. That is rare. The usual miss is a sure sentence parked next to examples that disagree with it.

A good note, and the readers bunch up

Each row is a writer. Each column is a reader who never saw the lesson, only the note. The last two rows are the perfect note and the blank page. Opus 5's column is the washed-out one. Those are empty replies.

Where the note is solid, the readers land in a tight band. Opus 4.8, Grok 4.20, and GPT-5.5 each got about 93 percent of the model-written notes. Gemini 3.6 Flash and Gemini 3.1 Pro got about 92 and 93. The perfect note was essentially 100 percent for them. Gemini 3.5 Flash, best teacher in the whole run at 1.02, read other people's notes at about 92 percent. Reading Efficiency 0.90. Ordinary.

Gemma 4 31B taught at 0.99 and the run cost $0.15. Astra taught at 1.01 and the run cost $0.88. Gemma's notes were the short ones. As a reader it scored 91 percent on everyone else's notes.

Sonnet 5, author of the 4x − 1 note, is better at other people's notes than at its own. Its notes, read by itself, scored 0.75. Opus 5's notes, read by Sonnet 5, scored 0.99.

The ratio will crown the wrong reader

Green is the perfect note. Blue is whatever the model actually wrote. Orange is a blank page. The blue line is only a fair grade where the green line is still up near 1.

Gemma 4 26B's Reading Efficiency is 0.96. That looks like the top of the table. On the notes themselves it scored 80 percent. The perfect note only got it to 83 percent. With nothing, it scored 0. The ratio is large because the bottom of the fraction is empty. Qwen3 Next 80B rhymes with it: ratio 0.95, notes at 80 percent, perfect note at 83 percent.

Opus 5, reading, is a different kind of wreck. Empty replies on 9 of 12 perfect notes, and on 129 of 296 model notes. Fifty percent on the notes. Twenty-two percent on the perfect note. A modest numerator over a broken ceiling comes out above 2. I am not putting that on a leaderboard. An earlier probe hit the same silence when one sentence about a hidden rule was in the prompt.

Nano's 0.93 is a smaller version of the Gemma problem. It scores about 6 percent with no note, so the ratio looks sharp. On the notes it is at 88 percent, middle of the pack.

Flash-Lite is what a low score looks like when the perfect note still works. Eighty-five percent on the notes, 16 percent with nothing, 99 percent on the perfect note, Reading Efficiency 0.83. That 0.83 can be compared with the others. Gemma's 0.96 needs the three raw rates sitting next to it, or it will be misread.

What I would keep

Print the blank-page score and the perfect-note score beside the ratio. A teaching score of 1.0 means the note matched the truth. A reading score of 0.96 can mean the reader was lost with nothing and only middling with a perfect note.

Grade the note with a stranger. DeepSeek's own fresh copy scored 100 percent on a note that the small reader scored 46 percent on, after DeepSeek had learned the rule completely.

Count the blank page. Three notes did worse than no note. They added a rule that was false.

Leave empty replies and dead runs out of the average. Opus 5's blanks inflated a ratio. gpt-oss never finished. Those two facts do not belong in the same cell.

What I would measure next

Hand the note to a second model and make that model write the next note. Same rules. Count how many hops the rule lasts. Sonnet's "the examples match," written over examples that do not match, is the sentence I want a model to catch before it passes the page on.

My Benchmark

Teaching and reading are separate tasks. Teaching Efficiency is the first number. Reading Efficiency is the second, and a reader whose perfect note failed is marked not comparable. Twenty-five models have a teaching score. Twenty-four finished reading. Sonnet 4.5 has a teaching score. Its reading run was still going when I wrote this. gpt-oss-120b and Grok 4.20 non-reasoning are not in the scored rows.

Teaching: https://www.kaggle.com/benchmarks/tasks/naomiandreapereira/amnesiac-teach

Reading: https://www.kaggle.com/benchmarks/tasks/naomiandreapereira/amnesiac-read

Passion Brought Me to Open Source. PatchQuest Helps Others Get Started.

Naomi Pereira — Sun, 12 Jul 2026 21:36:17 +0000

PatchQuest: Turning Passion into Open Source Contributions

This is a submission for *Weekend Challenge: Passion Edition*.

🚀 What I Built

When I saw this weekend's theme, Passion, my mind immediately went to open source.

Every day, thousands of developers star repositories, bookmark issues, and tell themselves they'll contribute "one day." But that first contribution can feel overwhelming. You have to understand an unfamiliar codebase, pick up new concepts, and somehow know where to begin.

I wanted to build something that makes that first step feel a little less intimidating.

That's how PatchQuest came to life.

PatchQuest transforms real GitHub issues into personalized learning quests. Instead of simply giving you the answer, it helps you understand the problem, learn the concepts behind it, and build the confidence to contribute.

The workflow

Start with any GitHub issue.
PatchQuest analyzes the repository using Google Gemini.
Receive a personalized learning quest based on the skills you'll need.
Learn the concepts before diving into the code.
Pair program with a live AI mentor using voice and screen sharing, powered by Google Gemini and ElevenLabs.
Generate a PR Readiness Report before opening your pull request.
Keep learning on the go with the Wear OS companion app, where you can track your progress, active quests, reminders, and streaks.

Personalized Learning Quest

Gemini Live Pair Programmer

PR Readiness Report

‹ ›

🎥 Demo

You can try PatchQuest here:

🌐 Live Application

https://patchquest.vercel.app

Or watch the demo below.

Note: Developer logs are intentionally visible during the demo to make the AI pipeline easier to follow. The live application showcases the polished production UI without those logs.

💻 Code

The complete source code is available on GitHub.

NaomiiAP / patchquest

🐉 PatchQuest

Your AI pair programmer for open source. Learn, build, and contribute with confidence.

Turn GitHub issues into personalized learning quests powered by Google Gemini Built for the DEV Weekend Challenge: Passion Edition using Google Gemini, Gemini Live, ElevenLabs, and Wear OS.

🎥 Demo

🌍 Live Website	patchquest.vercel.app
📖 DEV Post	dev.to/naomiiap/patchquest

🔥 What is PatchQuest?

Open source is hard to break into. You find a promising GitHub issue, read it once, and immediately feel lost — "What even is a race condition? What's a mutex? Where do I start?"

PatchQuest fixes that. Paste a GitHub issue URL and watch it transform into a structured learning quest — skill gaps identified by Gemini AI, curated resources, a step-by-step solution outline, and a voice tutor to walk you through it all. It even includes a Wear OS companion app to keep your learning journey with you wherever you go.

This…

View on GitHub

⚙️ How I Built It

I wanted PatchQuest to feel like more than another AI coding assistant. The goal was to build something that helps developers learn while contributing to open source instead of simply generating answers.

Google Gemini

Google Gemini powers the core experience. It analyzes GitHub issues, understands repositories, identifies the skills required for each task, creates personalized learning quests, powers the live AI pair programmer through voice and screen sharing, and generates PR Readiness Reports before a pull request is submitted.

ElevenLabs

For the live mentoring experience, I integrated ElevenLabs to provide natural voice interactions. I wanted conversations to feel less like talking to a chatbot and more like pair programming with another developer.

Wear OS Companion

One feature I especially enjoyed building was the Wear OS companion app. Learning doesn't stop when you step away from your computer, so the watch lets you keep track of your active quest, progress, reminders, and streaks without needing to open the web app every time.

The biggest challenge wasn't building each feature individually. It was making repository analysis, learning quests, live mentoring, PR review, and the smartwatch companion all feel like one seamless experience.

🏆 Prize Categories

I'm submitting PatchQuest for the following prize categories.

🤖 Best Use of Google AI

Google Gemini powers:

Repository understanding
GitHub issue analysis
Skill mapping
Personalized learning quests
Live AI pair programmer with voice and screen sharing
PR Readiness Reports

🎙️ Best Use of ElevenLabs

ElevenLabs powers the live AI mentor's voice, making mentoring sessions feel natural and conversational while pair programming.

Thanks for taking the time to check out PatchQuest! I'd genuinely love to hear your thoughts and feedback.

CareSync: A Local Health Memory Agent for Family Caregivers

Naomi Pereira — Sun, 31 May 2026 18:20:02 +0000

This is a submission for the *Hermes Agent Challenge*: Build With Hermes Agent
(had to edit as code block wasnt rendered properly)

What I Built

CareSync is a CLI-based caregiving assistant that helps me keep track of everything related to caring for my grandmother.

I'm Naomi, a 21-year-old engineering student. Between classes, assignments, and part-time work, I also help care for my grandma, Kamala. Over time, I realized that caregiving is surprisingly difficult to manage.

A symptom might be mentioned over breakfast. A medication change could happen after a doctor visit. A few weeks later, someone asks, "Has this happened before?" and suddenly I'm digging through WhatsApp messages, notes, and my memory trying to find the answer.

I built CareSync to solve that problem.

CareSync acts as a personal caregiving command center. It stores observations, symptoms, medications, reports, and important events in a local SQLite database, making them searchable and easy to organize.

For example, I can quickly log a symptom:

./caresync add "Kamala felt dizzy after breakfast"

Search past events:

./caresync search --person Kamala --query dizziness

Generate a doctor-ready summary:

./caresync brief --person Kamala --days 14

Or create handoff notes when another family member takes over care:

./caresync handoff --person Kamala --for-person Aunt

CareSync isn't just a record-keeping tool. It helps answer questions, identify recurring patterns, prepare for appointments, and keep family members aligned.

Key capabilities include:

Quick logging of symptoms, meals, medications, vitals, and observations
Searching historical events and recurring symptom patterns
Medication tracking and report imports
Appointment briefs and doctor question generation
Caregiver handoff summaries
Audit logs for transparency
Natural-language workflows powered by Hermes Agent
Local-first storage using SQLite

CareSync does not provide medical advice. Its goal is to help caregivers remember less, stay organized, and feel more prepared.

Demo

The demo follows a typical caregiving workflow using CareSync.

It starts with logging a new symptom after Grandma mentions feeling dizzy. From there, CareSync is used to search past observations, helping answer an important question before a doctor visit: "Has this happened before?"

Next, the demo shows how CareSync can generate suggested questions for the doctor and create a concise appointment summary based on recent observations.

The demo also highlights Hermes Agent's natural-language capabilities. Instead of manually searching records, I can ask questions like:

Has Grandma been dizzy before?

Hermes searches the caregiving history, identifies relevant events, and returns a summarized answer with supporting context.

Finally, the demo demonstrates transparency through the audit log, showing how actions performed by the agent can be reviewed and traced.

Example commands shown in the demo:

./caresync add "Kamala felt dizzy after breakfast"
./caresync search --person Kamala --query dizziness
./caresync timeline --person Kamala
./caresync questions --person Kamala
./caresync brief --person Kamala --days 14
./caresync handoff --person Kamala --for-person Aunt
./caresync chat "has grandma been dizzy before?"
./caresync audit

Code

Repository: https://github.com/Byte-Sized-Brain/caresync

Architecture

My Tech Stack

Core Technologies

Hermes Agent – Natural-language orchestration and skill execution
Python 3.12 – Core application and CLI implementation
SQLite – Local-first storage for observations, medications, reports, and audit logs
agentskills.io – Custom Hermes skill definitions and execution
Terminal-based CLI – Primary interface for caregivers
Nous Portal – Development, testing, and agent deployment

Hermes Skills

CareSync is powered by seven custom Hermes skills:

log-health-event – Record symptoms, meals, vitals, medications, and observations
symptom-search – Search historical records and identify recurring patterns
medication-tracker – Manage medications, doses, and prescriptions
import-report – Import reports and prescription information into memory
appointment-brief – Generate summaries for upcoming doctor visits
handoff-notes – Create caregiver transition summaries for family members
doctor-questions – Suggest follow-up questions based on observations and reports

Storage Philosophy

CareSync is designed to be local-first. All caregiving information is stored in a local SQLite database rather than a cloud service, giving caregivers full ownership of their data while enabling long-term searchable memory.

How I Used Hermes Agent

CareSync started as a collection of CLI tools for managing caregiving information. While the commands are useful, caregiving often involves answering questions rather than running specific commands.

That's where Hermes Agent comes in.

Hermes acts as an orchestration layer on top of CareSync's tools and memory. Instead of remembering which command to run, I can describe what I need in natural language and Hermes determines the appropriate actions.

For example, I can ask:

Has Grandma been dizzy before?

Hermes searches historical observations, reviews matching events, identifies patterns, and returns a concise answer based on information stored in CareSync.

The biggest advantage came from multi-step workflows.

Consider the request:

Prepare me for tomorrow's doctor appointment.

Instead of manually reviewing notes and running several commands, Hermes can:

Search recent symptoms and observations
Review medication history
Generate suggested doctor questions
Create an appointment summary

All from a single request.

Hermes is also useful for family coordination. When another family member takes over caregiving responsibilities, the agent can gather recent observations, medication updates, and upcoming appointments to generate a clear handoff summary.

The terminal tool was particularly valuable because Hermes performs real actions against the CareSync CLI rather than generating responses from memory alone. Every action interacts with actual data stored in SQLite and is recorded in an audit log.

This makes CareSync feel less like a collection of commands and more like an assistant that understands caregiving workflows. Hermes handles the reasoning and orchestration, while CareSync provides the tools and long-term memory.

Why I Built This

I built CareSync because I experienced this problem myself.

As a student, I help care for my grandmother, Kamala. Important details like symptoms, medication changes, and doctor recommendations were often spread across messages, notes, and memory.

CareSync gives me one place to record, search, and organize that information, making it easier to prepare for appointments and keep family members informed.

Its goal is simple: reduce the mental load of caregiving and help families stay organized.