DEV Community: Shivendu(Shivu)

Building a Shared Brain for My AI Agents — and Everything That Broke Along the Way

Shivendu(Shivu) — Sun, 05 Jul 2026 10:06:38 +0000

Meet Passport 🧳 — a shared memory layer for your AI coding agents.

I have three AI coding agents open right now. Claude Code in one terminal, Cursor in a window, Codex in another. And every single one of them is an amnesiac.

I'll spend ten minutes explaining my stack to Claude Code — we use pytest, the DB is Postgres, auth is JWT, don't touch the legacy billing module. It nails the task. Then I switch to Cursor for something else, and I'm back to square one. Same explanation. Different tool. Blank stare. By the third time I typed "we use pytest, not unittest" into a fresh chat, something in me snapped.

These tools are brilliant.

They're also goldfish.

And weirdly, none of them talk to each other.

What Claude Code learns about my codebase dies the moment I close the tab, and Cursor never even knew it happened.

So when I saw Cognee's hackathon — literally themed "Where's My Context?" — I didn't need to brainstorm.

I'd been living the problem for months.

The idea (and the doubt that came with it)

The pitch was simple:

One shared memory that every agent reads from and writes to.

Teach one, and they all know it.

I called it Passport — because your memory should travel with you, into whatever tool you open next.

But here's the first honest thing I'll admit:

For about an hour, I was scared it was fake.

Cognee already ships an MCP server. My judges are Cognee. If all I did was wire their memory tool into a couple of agents, I'd be handing the people who built the memory layer a tutorial and calling it a project.

That fear sat in my chest the whole first day.

The only way out was to build something that wasn't in anyone's tutorial:

Provenance (which agent taught what)
Conflict detection (what happens when two agents disagree)
Real isolation between users

Those three became the whole point.

Everything else was table stakes.

I also made a call I second-guessed a dozen times:

Skip the flashy browser extension, go MCP-only.

A browser extension that captured ChatGPT would've been a killer demo — memory following you into the web.

But it's fragile, and a demo that breaks live is worse than a demo that's slightly less magical.

I chose the boring, robust path.

(Spoiler: I'd make the same call again. The magical path breaks. Ask me how I know.)

The moment it actually worked

I'll never forget this one.

I told Claude Code — Anthropic's agent —

"remember that we use JWT for auth and the frontend is React."

Then I opened Codex — OpenAI's agent, a completely different company's model, a fresh session that had never seen a word of that conversation — and asked:

"what's our auth and frontend?"

It said:

JWT and React.

I actually said "no way" out loud to an empty room.

Two rival AI systems, sharing a brain, through a memory layer I'd wired up that afternoon.

That's the moment the project stopped being an idea and started being real.

Cross-vendor memory.

It shouldn't feel emotional.

It did.

And then, of course...

Everything started breaking.

Break #1: Codex crashed and I couldn't see why

Claude Code worked.

Codex threw:

[Errno 22] Invalid argument

on every recall and I had no idea why.

Same server.

Same code.

Different client.

One works, one dies.

I burned an hour on this.

The breakthrough was almost stupid:

MCP talks over stdout, and Cognee logs a lot to stderr during a recall.

On Windows, if the client doesn't drain that stderr pipe, it fills up (~64KB), and the next write crashes the whole tool.

remember was quiet enough to sneak through.

recall was chatty enough to blow the pipe.

The fix was three lines — route stderr to a log file before Cognee gets imported.

But finding those three lines meant understanding exactly how pipes, buffering, and MCP's transport actually work.

That's the thing nobody tells you about these bugs:

The fix is tiny, the understanding is not.

Break #2: the feature that vanished

Cognee's memory lifecycle is:

remember
recall
improve
forget

I'd built my whole conflict-reconciliation story on improve() — the "memify" step that reweights the graph.

I called it against the cloud and got a flat:

404 Not Found

memify isn't exposed on the cloud tier I was using.

A feature I'd designed around, gone.

For a second I felt the floor drop.

Then I did the thing you actually have to do in a hackathon:

Do I actually need it?

And I didn't.

Reconciliation doesn't require reweighting a graph — it requires recording an authoritative decision and making recall respect it.

I wrapped improve() in a try/except, logged it honestly as "OSS-only," and moved on.

The demo got simpler.

And more honest.

Sometimes the constraint is a gift.

Break #3: the night Doug the wolf leaked into everything

This is the one that nearly broke me.

Multi-tenancy — real isolation between users — was supposed to be the crown jewel.

Two users, alice and bob, each with their own private brain.

I wrote the test:

alice stores a secret
bob stores a different secret
neither should ever see the other's

I ran it.

alice recalled her own fact perfectly.

And bob...

recalled

"the mascot is a wolf named Doug."

Doug.

The.

Wolf.

Doug was test data from hours earlier — a throwaway fact I'd stored a dozen times while debugging something unrelated.

And here he was, materializing inside a brand-new tenant's supposedly-isolated memory.

My isolation was an illusion.

If this were real users, one company's secrets would be bleeding into another's.

That's not a bug you ship.

That's the bug that ends the product.

I tried the obvious fix — filter by the tenant's tags — and it got worse.

Now both users got Doug.

I sat there staring at a wolf that would not die, at 1 in the morning, genuinely wondering if the whole multi-tenant premise was broken.

So I stopped guessing and got methodical.

I stored one unique fact —

"the office mascot is a phoenix named Ember."

—and queried it through every retrieval mode Cognee offered, one at a time, watching which ones leaked and which held.

GRAPH_COMPLETION → leaked Doug

RAG_COMPLETION → leaked Doug

CHUNKS → returned "a phoenix named Ember."

Clean.

Isolated.

Correct.

The graph-completion modes were searching the whole shared knowledge graph and letting a heavily-reinforced node like Doug bleed through dataset boundaries.

Raw chunk retrieval respected the dataset scope.

The fix wasn't a filter.

It was choosing the right kind of recall.

I switched to chunk-scoped retrieval, re-ran the isolation test, and got the two words I'd been chasing for two hours:

ISOLATION PASSED

I may have fist-pumped.

At a wolf.

It's fine.

And the beautiful part:

The fix made the design better.

Passport now returns each tenant's own faithful facts and lets the calling agent do the reasoning — cleaner, more correct, and more honest than a black-box completion.

The bug forced a better architecture.

They usually do.

The question I was most afraid to ask

Late in the build, my teammate asked me the question I'd been avoiding:

"Is this thing actually intelligent, or are you just hardcoding keywords and fooling yourself?"

That one stung because it was fair.

Part of my ranking used a keyword heuristic for "importance."

Was I dressing up if-statements as AI?

So I did the honest thing.

I ran an adversarial test designed to fail if it was fake.

I stored:

"the quarterly board meeting is the first Tuesday of each month."

Then queried it with:

"when do senior executives gather to review company performance?"

No shared words at all.

A keyword system fails this instantly.

It returned the meeting.

Because the recall is real semantic understanding — embeddings that know "executives gathering to review performance" means "board meeting."

That's not string matching.

That's the real thing.

But I also didn't let myself off the hook.

The importance score was a keyword heuristic, and that's not intelligence.

So I upgraded it:

Now Cognee's own LLM rates each memory's importance 1–10.

I tested it — a company policy scored a 9, idle chatter scored a 2.

Real judgment.

And I kept the one honest non-AI piece — trust weights — exactly as explicit config, because deciding "how much do I trust this source" is a governance call, and letting a model guess it is how you get memory-poisoning attacks.

Knowing what shouldn't be AI is its own kind of engineering.

What it became

By the end, Passport wasn't a wrapper.

It was:

A shared brain across Claude Code, Cursor, and Codex — proven cross-vendor.
Provenance on every memory — a live graph colored by which agent taught what.
Conflict detection via Cognee's LLM — it catches "Postgres vs MySQL" and reconciles it.
Real multi-tenant isolation — measured, zero leakage.
Intelligent recall — semantic retrieval, LLM-scored importance, recency, source trust.

And I made myself measure it instead of vibe it:

✅ Cross-agent recall 100%
✅ Isolation 100%
✅ Semantic recall proven against zero-keyword queries

Numbers, not adjectives.

I'll be honest about the limits too, because a blog that only brags is a lie:

memify only runs self-hosted
my conflict detection is high-signal but not exhaustive
trust weights are static, not learned

All fixable.

All on the roadmap.

None of them hidden.

The thing I actually took away

There's a poetic irony I didn't plan:

I built a memory for AI agents while pair-programming with an AI agent (Claude Code — disclosed, and honestly the whole reason I could move this fast).

The tool that forgets,

helping me build

the thing that remembers.

But the real lesson wasn't about memory.

It was that every hard bug — the pipe that crashed Codex, the feature that 404'd, the wolf that wouldn't die — didn't just cost me time.

Each one forced a better decision.

The crash taught me how the transport really works.

The 404 made the design simpler.

Doug the wolf handed me a cleaner, more correct architecture than I would've written on my own.

We keep asking our AI where its context went.

Turns out the answer was never a bigger context window.

It was giving it a memory —

one it owns,

one that travels with it,

one that remembers who taught it what.

Passport is that memory.

Doug is finally gone.

And my agents, at last, remember last night.

🔗 Live Demo

https://memlayer.streamlit.app/

💻 GitHub

https://github.com/ShivenduShivu/MemoryLayer_for_Agents

Made with 💓 & Built for Cognee's "Where's My Context?" hackathon — powered by Cognee's graph-vector memory.

How High-Frequency Trading Systems Remove Every Microsecond of Latency

Shivendu(Shivu) — Wed, 06 May 2026 19:52:21 +0000

I recently went down a rabbit hole connecting OS internals with real-world low-latency systems.

While learning about process management in operating systems, I kept wondering:

“Where does this level of optimization actually matter?”

That eventually led me to High-Frequency Trading systems — one of the few domains where microseconds can literally mean money.

So I decided to break down how modern HFT systems push OS, hardware, and networking to their limits.

What is HFT (Really)?

At a surface level, HFT sounds simple:

“Buy low, sell high — very fast.”

But in reality, it looks more like this:

Receive market data → analyze → decide → send order → repeat — all within microseconds.

A simplified pipeline:

Exchange → Market Data → Strategy → Order Execution → Exchange

It looks straightforward on paper.

In practice, every step has to happen faster than your brain can even register what’s going on.

Why Speed is Everything

In HFT:

1 millisecond is already slow
1 microsecond is competitive
1 nanosecond is where things get serious

Even a 5–10 microsecond delay can mean:

Someone else gets the trade
You miss the opportunity
Or you lose money

So engineers start asking uncomfortable questions:

“What if we remove everything unnecessary… including the operating system?”

Where the Operating System Becomes the Bottleneck

Normally, when data arrives:

Network Card → OS Kernel → Application

The OS does a lot of useful things:

Handles interrupts
Manages memory
Schedules processes

All of this is great for general-purpose systems.

But in HFT, it introduces:

Context switches
Memory copies
Scheduling delays

Which adds up to tens of microseconds — far too slow for this domain.

This is where things start getting crazy.

The Big Hack: Bypassing the OS

Yes, this is exactly what it sounds like.

HFT systems often bypass the OS kernel entirely.

Normal flow

NIC → Kernel → App

HFT flow

NIC → User Space (Direct)

Technologies like:

DPDK
RDMA
AF_XDP

allow applications to access network packets directly without going through the kernel.

It’s essentially skipping all the middle layers and going straight to the source.

Interrupts? Not Really

In a typical system:

The network card interrupts the CPU
The OS handles the interrupt

In HFT systems:

The CPU continuously polls the network card

Why?

Because waiting for an interrupt introduces latency.

Polling may use more CPU, but it removes unpredictability.

And in this world, predictability matters more than efficiency.

CPU Pinning: One Core, One Responsibility

Instead of letting the OS freely schedule tasks:

Core 1 handles market data
Core 2 runs the strategy
Core 3 handles order execution

This reduces:

Context switching
Cache invalidation

It’s a simple idea:

Fewer interruptions, more consistency.

NUMA Awareness (Memory Isn’t Uniform)

Not all memory access is equal:

Local memory is fast
Remote memory is slower

HFT systems carefully align:

CPU cores
Memory allocation

on the same NUMA node.

Because even a few nanoseconds can make a difference.

Lock-Free Programming

Traditional code often looks like this:

lock();
update();
unlock();

In HFT systems, you’ll often see:

atomic_update();

Using:

Atomic operations
Lock-free queues
Ring buffers (like LMAX Disruptor)

Locks introduce waiting and unpredictability.

Both are things you want to avoid here.

FPGA Acceleration

At some point, even optimized CPU code isn’t enough.

So firms move parts of the system into hardware using FPGAs (Field Programmable Gate Arrays).

These chips:

Run custom logic
Process data with extremely low latency
Avoid OS overhead entirely

What runs on FPGA?

Market data parsing
Order book updates
Sometimes even trading logic

The result is latency measured in nanoseconds.

At this point, engineers basically start fighting physics.

Co-location: Physical Distance Matters

HFT firms often place their servers:

Inside the exchange’s data center

Because:

Shorter distance means lower latency

At this level, even physical distance becomes a competitive advantage.

The Final Optimized Pipeline

A modern HFT system might look like:

FPGA NIC → User-space processing → Lock-free queue → Strategy → Order → Exchange

Typical latency breakdown:

Packet processing: ~0.1 µs
Strategy logic: ~3 µs
Total: ~4–5 µs

That’s significantly faster than anything humans can perceive.

Current Limitations

Even with all these optimizations, there are still hard limits.

1. Physics

Data travels at the speed of light.

You can optimize software and hardware,

but you can’t go faster than physics allows.

2. Jitter

Even if average latency is low, variability can hurt performance.

Sources include:

Cache misses
OS noise
Hardware behavior

Consistency matters just as much as speed.

3. Complexity

These systems are:

Difficult to build
Difficult to debug

A small mistake can have large financial consequences.

4. Cost

FPGA hardware
Specialized networking
Co-location

All of this adds up quickly.

The Future: Where Things Are Heading

There’s still room to push further.

Full Hardware Pipelines

The goal is to move the entire pipeline onto hardware:

No CPU
No OS

Just direct processing from input to output.

Smart NICs

Network cards are becoming more capable:

Processing packets
Running custom logic

They’re starting to behave like small computers.

RDMA Everywhere

Remote Direct Memory Access allows:

Direct memory communication between machines
Minimal CPU involvement

This reduces latency even further.

Minimal Operating Systems

Instead of general-purpose OSes:

Use stripped-down, specialized systems
Remove unnecessary components

The focus is on predictability and control.

Low-Latency AI

Applying machine learning in HFT is challenging because:

Inference takes time

Solutions include:

Hardware acceleration
FPGA-based inference

Final Mental Model

Normal systems:

App → OS → Hardware

HFT systems:

App → Hardware

Future direction:

Hardware → Hardware → Exchange

Closing Thought

HFT sits at the intersection of:

Operating systems
Hardware design
Networking
Physics

And the goal is simple:

Be faster than everyone else — even if it’s by a few microseconds.

This topic genuinely changed how I think about systems engineering.

You start realizing that performance isn’t just about writing faster code — it’s about removing friction from every layer of the stack.

If you’ve worked on low-latency systems, kernel tuning, networking, or HFT infrastructure, I’d genuinely love to hear your thoughts.