Every chatbot you talk to starts from zero. Close the tab, open it again, and it has no idea who you are. That is not a bug — it is how most cloud APIs are built.
We have been shipping a local memory layer for AI agents (this is our own product, HyperMarrow, so take the comparison with that in mind). The core question for any team building AI features is simple: where should the memory live — on the user's machine, or in someone else's datacenter?
Three things actually change
Privacy. A cloud API means every conversation leaves the device. Local memory keeps the raw transcript on the user's disk and uploads nothing by default. For internal tools handling customer data, that difference is the difference between a security review that passes and one that does not.
Cost. API calls are billed per token. If you re-send the user's entire history on every request to fake "memory," your bill scales with conversation length. A local store retrieves only what is relevant, so the context you ship stays small.
Control. When memory is local, you decide what is kept, what is forgotten, and when. With a third-party API, retention is their policy, not yours.
How the local model works
We split memory into four parts: intercept what the user actually does, record it, consolidate fragments into something durable, and recall the right piece on demand. The consolidation step is the one most cloud setups skip — they just stuff more tokens in the prompt and hope.
If you want to see how a local-first memory layer feels, the project is at https://hm.qianshi.cool/?from=devto (30-day full trial, no card). Curious what people build once their agent actually remembers — happy to compare notes in the comments.
Top comments (0)