DEV Community

Cover image for Chat memory for an on-device notes assistant, without a bigger prompt
Georgii Kharlampiiev
Georgii Kharlampiiev

Posted on Originally published at mindscendinc.com AI-assisted

Chat memory for an on-device notes assistant, without a bigger prompt

Originally published on the Mindscend blog.

Mindscend Notes (formerly MindPage) is our notes app for handwriting, PDFs and books on iPhone, iPad, Mac and the web. Notes stay on your devices and in your own iCloud Drive or Google Drive; the AI runs on the device. Version 1.1.1 just shipped on the App Store, and the free version no longer has limits: unlimited folders and notebooks, sync and on-device AI, no account.

This post is the technical part: how "Ask my notes" got chat memory without giving the model the conversation, what the search under it does, and a SwiftUI hit-testing bug we managed to ship twice.

The constraint: a 4,096-token model and a search that decides

Ask my notes is retrieval plus Apple's on-device Foundation Model. The pipeline:

  1. Pages are split into ~700-character passages with ~20% overlap, cut at sentence or line ends.
  2. Each passage is embedded with its context: "<document topic>, page N: <passage>". The topic is the PDF's own title or the notebook's name. (A file name like 1707.06347v2 (1) as context made search worse.)
  3. Hybrid ranking: BM25 over passage text + EmbeddingGemma per passage + EmbeddingGemma per page, fused with reciprocal rank fusion (k = 60).
  4. Small to big: the 3 best pages × 2 best passages go to the model.
  5. The answer is structured (@Generable), must quote the notes, and passes a grounding check. Apple's session is 4,096 tokens, so notes are trimmed to a budget and retried with half on overflow.

How much each search layer matters, on our 21-question test library:

Search Right page first In top 3
Words only 12/21 13/21
Apple NLEmbedding 15/21 18/21
EmbeddingGemma 20/21 21/21

The embedding model in the app is our fine-tune, MindPage Search v2 (Apache-2.0, 300M, Q8_0 GGUF via llama.cpp, Matryoshka-truncated to 256 dims). It was trained on what Apple Vision actually reads from handwriting, misreadings included. On 234 held-out handwritten pages, it ranks the right page first 85.9% of the time vs 80.3% for v1.

Chat memory lives in search, not in the prompt

The 1.1 Ask was one question, one answer. Users then type follow-ups:

when is the physics exam?
and where is it?

The textbook fix is to pass the chat history to the model. That fails here twice: the context window is already spent on retrieved passages, and the model never sees the right page anyway, because "and where is it?" retrieves nothing.

So we detect follow-ups and fix the query:

// A sketch of the idea, not the app's code
func isFollowUp(_ q: String) -> Bool {
    let words = q.split(whereSeparator: \.isWhitespace)
    return words.count <= 3
        || q.lowercased().hasPrefix("what about")
        || words.contains { pronouns.contains($0.lowercased()) }
}

let searchText = isFollowUp(q) ? previous + " " + q : q
let prompt     = isFollowUp(q) ? "\(q) (follow-up to: \(previous))" : q
Enter fullscreen mode Exit fullscreen mode

Retrieval sees the previous question's words, so it finds the exam page again. The model keeps its one-question prompt, so none of the answer evals had to change. Chats are stored as chat:<ulid> root keys in the library's Automerge document, with messages in a map keyed by ULID, so they sync and merge like everything else.

Without Apple Intelligence, there's no writing model, so Ask shows the one line of the notes that answers the question: EmbeddingGemma scores each line and line pair of the two best pages, with the page heading as context. 15/15 on our extraction eval.

The SwiftUI bug we shipped twice

In the iPad library's list view, tapping a notebook row only worked on the text. Tapping the middle of the row did nothing.

Button { open(doc) } label: {
    HStack { Text(doc.title); Spacer(); Text(doc.date) }
}
.buttonStyle(.plain)
Enter fullscreen mode Exit fullscreen mode

With .plain, only drawn content is hit-testable. A Spacer draws nothing, so most of the row wasn't a button. The fix:

.buttonStyle(.plain)
.contentShape(.rect)
Enter fullscreen mode Exit fullscreen mode

We hit the same thing in October on the Mac pen bar (only an icon's drawn pixels took clicks). Lesson recorded twice now: any .plain button with a Spacer or thin icon needs .contentShape. A UI test (testListRowOpensNotebook) now taps the middle of a row on iPhone and iPad.

That test came from a batch written for features that had no UI tests: empty library, folders, export, recordings, sync choices, largest accessibility text, landscape, flashcards. The batch also found an export sheet that could spin forever (the sheet showed a URL set by a separate Task; now the sheet runs the export itself) and flashcards unreadable in Dark Mode.

One more from the test suite: a test that queried a sticker the wrong way made XCUITest snapshot the whole accessibility tree, which took up to 15 minutes. Checking it as a button first fixed it.

What's still weak

  • The native Mac app isn't on the Mac App Store yet; our latest Mac run caught Summarize stuck on its spinner.
  • Reading position syncs on the web; the iOS apps remember it per device for now.
  • Kanban boards show on the web but are edited in the apps.

App Store: Mindscend Notes · Web: mindscend.academy · Models: huggingface.co/Mindscend

Top comments (0)