DEV Community

mage0535
mage0535

Posted on

From Chaos to Context: Building a Knowledge Pipeline for AI Agents

#ai #memory #opensource

I've been running AI agents for research and coding tasks, and the hardest part hasn't been the prompts—it's keeping them fed with the right context. My agents kept forgetting what I'd saved last week: that PDF paper, the YouTube lecture I watched, the blog post with the clever architecture. I needed a system that could collect anything, process it automatically, and make it available the next time my agent needed it.

That's why I built Knowledge & Memory Management (KMM)—an open-source extension layer for the Hermes Memory Sidecar. It's not another vector database; it's a full pipeline from content ingestion to multi-tier storage, with cloud sync baked in.

The Pipeline at a Glance 

Collection (40+ tools) → Analysis (AI processing) → Storage (3 tiers) → Sync (rclone)
Enter fullscreen mode Exit fullscreen mode

Collection Layer

40+ tools covering:

  • Web scraping engines (9)
  • Video/audio via yt-dlp (12)
  • Article and content extractors (10)
  • Documents and OCR via MarkItDown (10)
  • Knowledge retrieval and management (11)

Everything normalizes to a common format before it hits the analysis layer.

Analysis Layer

Automatic note generation, knowledge graph extraction, NLI fact-checking, and knowledge discovery. For example, when I save a video, KMM downloads it, transcribes it, generates a summary, and extracts key entities—all without me lifting a finger.

Storage: Three-Tier Memory

  • Hot – Fast, in-context tool for recent conversations
  • Warm – Semantic vector index (Hindsight, ~10K nodes) for your active knowledge base
  • Cold – Full-text archive (gbrain, ~11K pages) for historical deep cuts

The three tiers balance retrieval speed and depth. Most queries hit warm memory first, falling back to cold when needed. This keeps agents responsive without losing long-tail knowledge.

Cloud Sync via rclone

My memory database syncs automatically to OneDrive (or Google Drive, Dropbox, S3, WebDAV, etc.). That means I can run my agent on my laptop in the morning and on my server in the afternoon—same context, same knowledge.

A Typical Workflow

  1. I find a research paper or a conference talk video.
  2. kmm collect <url> – it downloads, analyzes, and stores the result in warm memory.
  3. Later, my agent asks: "What was the key finding from that paper?"
  4. The agent retrieves the answer from warm memory via semantic search.
  5. If the query is historical, it falls back to cold memory—still accessible without latency spikes.

All of this works with the Hermes Memory Sidecar, which provides the base memory runtime. KMM is the knowledge supply chain.

Why I Built This (and Why It's Not Just RAG)

RAG is great for one-off retrieval over a fixed corpus. But my knowledge is always growing—new articles, bookmarks, video courses. I wanted a pipeline that ingests, processes, and organizes continuously. KMM treats knowledge as a living asset, not a query-time afterthought.

It's early (v0.0.2), MIT-licensed, and written in Python 3.10+. The code is straightforward—each module is self-contained with clear docs. I'm using it daily and it works for me. If you're building agents that need persistent, ever-growing context, it might work for you too.

Check it out on GitHub: Knowledge & Memory Management

And if you want the base memory runtime, see the sibling project: Hermes Memory Sidecar

Feedback and contributions welcome. Happy building!

Top comments (0)