I grew up watching Iron Man, so when I started this project the working name wasn't ORION, it was JARVIS. Obviously. I only changed it later because half of YouTube and Instagram is full of devs building their own "JARVIS" right now and I didn't want to be one more thumbnail in that pile.
But the name was never really the point. The point was two things I kept running into with every AI assistant I tried.
First, I live in the terminal. I don't want a browser tab, I don't want a desktop app, I don't want to click anything. If I'm going to talk to something every day it has to be where I already am.
Second, almost every "personal AI" project I saw was welded to one provider. OpenAI, or Claude, pick your poison. I didn't want that. I wanted to be able to run this thing on DeepSeek, on Claude, on GPT, on Gemini, or on a local Ollama model, or even on one of the newer Chinese open-source LLMs, and just switch whenever I felt like it.
So that's what ORION is. Terminal-only, no UI, memory in Obsidian, works on your code, doesn't care which model is behind it.
It worked great, right up until it didn't
The first version of the memory system was dumb in the good way — simple and it just worked. Ask something, ORION searches the vault, pulls in whatever's relevant. Day one, it already felt useful.
Where it fell apart was the writing side. My first idea was: every exchange becomes a note. Question, answer, saved to the vault. More notes = more memory, that was the logic.
Except it wasn't. A week in, my vault was full of junk. Dozens of near-identical notes like "user asked about X, I said Y," most of which nobody would ever want to read again. It stopped feeling like a second brain and started feeling like a chat log someone dumped into my notes folder. Nothing was technically broken, it just wasn't good.
Honestly this is the part that ate most of my time on the whole project. Not the terminal interface, not wiring up five different model providers — the memory. Getting it to actually behave like something with judgment.
Teaching it what NOT to write down
The fix wasn't a smarter prompt or a bigger model, it was accepting that the model has to make a call on every exchange: is this worth keeping, or not.
Roughly, it looks like this now — during a conversation ORION is quietly checking whether something is worth persisting (a fact, a decision, something I'll want next week) or whether it's just noise that shouldn't touch the vault at all. If it decides yes, it writes a proper note with frontmatter and links to related notes. If not, nothing gets written, it just stays in that conversation's working memory.
each exchange -> is this durable knowledge or just noise?
yes -> write a linked note
no -> keep it in working memory only, skip the vault
end of session -> write ONE summary note for the whole session
The other piece that came out of this, almost by accident: when I close a session, ORION now writes a single summary note of what happened in that session. Not every message — just the gist. Next time I open the terminal it can read that note and pick up roughly where we left off, instead of me re-explaining context from zero, which was the exact problem I built this thing to avoid in the first place.
I didn't plan that feature. It fell out of fixing the "too many notes" mess.
What I'd do differently
If I'm honest, I should have designed the "what deserves to be remembered" logic before I wrote any storage code, not after my vault was already a mess I had to clean up. Retrofitting judgment into something built to log everything is a lot more annoying than just building it in from day one. If you're building anything with agent memory — don't start with "store everything, sort it later." You'll regret it, I promise.
Where it's at
ORION is open source, MIT license, still early. DeepSeek by default, but Claude/GPT/Gemini/local Ollama all work too. The memory system above is the part I'm most proud of and also the part I'd love feedback on, because I don't think there's one correct answer to "what should an AI remember about you."
Repo's here if you want to poke at the code or try it:
https://github.com/BotirBakhtiyarov/orion-second-brain
There's a good-first-issue list if anyone wants to dig in. I'll be around.
Top comments (2)
"The fix wasn't a smarter prompt or a bigger model" is the conclusion most people reach late, after their vault is already a chat log with extra steps. Making the model call on every exchange whether something is durable knowledge or noise is the right primitive, and the end-of-session summary note falling out of it is the part I would keep forever.
If you ever revisit it: the failure mode I hit with a similar setup is that the filter drifts. Early on it drops trivia you wanted; months later, after you have tuned it, it hoards near-duplicates of the same fact from different sessions. A cheap defence is a merge pass at write time that checks whether a note already covers the fact, or a decay score so unread notes sink. Curious whether your frontmatter links are already doing that job.
Good catch, and actually there's already a mechanism for this, though it's softer than what you're describing. The system prompt has a rule that before creating a note, the model has to call search_notes first, and if something relevant already exists it's supposed to use append_to_note or update_note instead of creating a new one. There's also dedup on the linking side, link_notes won't duplicate an existing link.
The honest caveat: this is a prompt-level rule, not a hard constraint in code. Nothing stops the model from skipping search_notes and just writing a fresh note anyway, that's an LLM being an LLM, not a guarantee. So the drift you're describing is still possible, just less likely than in the naive version. Which makes me think the real fix might be closer to what you said originally, enforcing the search_notes check in code rather than trusting the prompt to make the model do it every time. Might turn that into an issue. Appreciate you pushing on this.