v0.0.2 is a clean release. If you've been tracking this project, the headline change is simple and overdue: every hardcoded personal path is gone, replaced with a portable $AGENT_HOME base directory. No more /home/someone/... leaks in configs, no more absolute paths baked into storage indexes. Set one environment variable and the entire knowledge store relocates cleanly.
This release is about two things working together: collecting knowledge from external sources and managing the resulting memory so it stays useful. Let's walk through what actually changed.
The $AGENT_HOME Path Shift
Previous versions had a bad habit — they embedded absolute paths into the collection metadata and memory store. That made backups, container moves, and multi-machine sync a pain. v0.0.2 replaces all of that with a single, resolved base directory.
The tool reads $AGENT_HOME at startup. If it isn't set, it falls back to a default location, but the recommendation for any real deployment is explicit:
export AGENT_HOME=/opt/agent-data
mkdir -p "$AGENT_HOME"/{collections,memory,index}
All subsequent writes — raw collected content, extracted transcripts, normalized article text, and the memory index — go under $AGENT_HOME. Relative paths inside the index are now the norm. You can move the entire store by changing one variable and re-running the re-index step. This is the "clean release" promise: no stale user-specific paths left in the data files.
Knowledge Collection: Web, Video, Articles
The collection layer in v0.0.2 accepts three source types, each with its own ingestion path:
- Web pages: The fetcher pulls the raw HTML, strips boilerplate, and stores the normalized article or page content. Metadata (URL, fetch time, title, site) is captured alongside. The key improvement here is that the stored record is self-contained — no live dependency on the original URL for basic retrieval.
- Video: Video ingestion is transcript-driven. The tool retrieves the transcript or subtitle track and stores it as the primary content, with the video URL and duration as metadata. This keeps the memory store text-searchable and avoids storing large binary files unless you explicitly enable media caching. For developers: think of video input as "get the transcript, not the bytes."
- Articles: Article ingestion handles structured content — typically from RSS/Atom feeds or direct article URLs. The parser extracts the main body, author, publication date, and any attached tags. It's stricter than the generic web fetcher because articles are expected to have a clear content boundary.
Each collected item gets a unique ID and is written to $AGENT_HOME/collections/<source-type>/. The raw fetch is kept separate from the normalized content, so you can re-process or debug without losing the original.
Memory Management: What Happens After Collection
Collection without management is just a pile of files. The memory layer is what makes this a knowledge store rather than a folder.
In v0.0.2, memory management includes:
- Deduplication by content hash: When a new item is collected, the tool computes a hash of the normalized text. If an identical hash already exists in the memory index, the new item is flagged as a duplicate and either merged or skipped, depending on your config. This prevents the same article or video transcript from accumulating copies.
- Title and entity normalization: Titles are cleaned of trailing junk ("- YouTube", "| Site Name"). Entity mentions are extracted heuristically and stored as tags, so later retrieval can match on concepts, not just exact strings.
-
A single append-only memory index: Every collection event appends a record to
$AGENT_HOME/memory/index— a newline-delimited JSON file, one record per line. This is not a database; it's a log. You can tail it, process it with standard Unix tools, or load it into your own search engine. Keeping it append-only makes backups trivial and avoids the complexity of a live server process. - Forget and prune operations: Memory management isn't only about adding. The release supports removing items by ID and pruning the index to drop references to missing files. This is not decay or automated forgetting — it's explicit maintenance — but it gives you control when collections get stale.
The design philosophy is that the memory index should always be reproducible from the collection folders. If you delete the index, you can rebuild it by scanning $AGENT_HOME/collections/ and re-hashing every item. The index is a convenience, not the source of truth.
What v0.0.2 Does Not Do
Keep expectations accurate. This release does not include semantic embeddings or vector search — that's a later milestone. It does not crawl the web autonomously; collection is pull-based, triggered by explicit input. Video ingestion works from transcripts and subtitles, not from audio transcription. If you need those features, they're not here yet.
Migration and Testing
If you're upgrading from v0.0.1, expect to edit config files. The old path patterns are not migrated automatically. The recommended move is to set $AGENT_HOME, re-run collection on your key URLs, and accept that old index records with absolute paths won't resolve. The data itself is readable — it's plain text and JSON — but the index entries are stale by design.
Run with a temporary $AGENT_HOME first, collect a single web page and a single video transcript, then inspect the stored files. If the paths look right and the index records reference $AGENT_HOME-relative locations, you're good.
v0.0.2 is a foundation release. It gets the plumbing right — portable storage, clean collection, maintainable memory log. That's enough to build on.
Top comments (0)