DEV Community: Amritesh Kumar

From Flat Files to a Living Memory: Building Graph-Based Semantic Memory for PocketPaw

Amritesh Kumar — Fri, 10 Apr 2026 10:45:08 +0000

TL;DR — I upgraded PocketPaw’s memory from a line-based file store to a hybrid semantic memory: (1) vector-backed retrieval for relevance-aware recall, and (2) a small knowledge graph for entity/relationship tracking. The result: the agent remembers semantically, connects concepts across sessions, and surfaces context proactively.

Link: https://github.com/pocketpaw/pocketpaw/issues/455

Why change the memory system?

PocketPaw's original memory (memory/file_store.py) stores facts as plain text lines. That model is simple and reliable, but it fails at several important agent behaviors:

Lexical-only recall: users must repeat keywords to be found.
No relationships: facts stay isolated (e.g., "Project X uses React" doesn't connect to "React 19 has breaking changes").
Poor cross-session continuity: facts are hard to prioritize/retrieve semantically across long histories.
No decay, timestamps, or metadata to shape relevance over time.

These limitations make the agent brittle for longer-term, proactive assistance. To fix that, we implemented two complementary features: vector-backed retrieval and a lightweight knowledge graph.

Design goals

Semantically meaningful retrieval (no more exact keyword matching).
Cross-session continuity and contextual prioritization.
Local-first: keep data on-device unless the user opts in.
Modular/backfill-friendly: fall back to file store and allow gradual migration.
Low operational overhead: support ChromaDB, Qdrant, or SQLite-vec as vector backends.

High-level architecture

Ingest pipeline (on writes)
- Normalize fact text, add metadata (timestamp, source, session_id).
- Produce an embedding using a configured embedding model.
- Store vector + metadata in a vector store (Chroma, Qdrant, or SQLite-vec).
- Run an entity/relationship extractor LLM to identify entities and relations; upsert into a lightweight graph (NetworkX or SQLite adjacency tables).
Retrieval pipeline (on agent turn)
- Query vector store for top-K relevant documents to the current prompt (RAG).
- Use graph queries to expand context (e.g., related entities, recent facts).
- Compose memory context to include top results + related graph edges; feed to the LLM.
Management
- Memory pruning, TTL/decay heuristics, and manual UI for inspection & edits.

Diagram (simplified):
User ↔ Agent → Writes → [Embeddings → Vector Store] + [LLM extraction → Graph DB]
Agent ← Retrieval ← Vector Store + Graph DB ← Context

Phase 1 — Vector-backed memory (what I implemented first)

Goals: make memories retrievable by semantic similarity while preserving the file-store fallback.

Key components:

Embedding model: nomic-embed-text (local model via Ollama or similar).
Vector stores: ChromaDB, Qdrant, SQLite-vec (selectable via config).
Minimal metadata: id, text, created_at, session_id, source, tags, vector_id.

Configuration (env):

POCKETPAW_MEMORY_BACKEND=vector     # file | vector | mem0
POCKETPAW_EMBEDDING_MODEL=nomic-embed-text
POCKETPAW_VECTOR_STORE=chromadb     # chromadb | qdrant | sqlite-vec

Write flow (pseudo):

def save_memory(text, source, session_id, tags=None):
    metadata = {
        "text": text,
        "source": source,
        "session_id": session_id,
        "created_at": now_iso(),
        "tags": tags or [],
    }
    embedding = embed_text(text)  # calls the configured embedding model
    vector_store.upsert(id=uuid4(), vector=embedding, metadata=metadata)
    # Graph extraction step runs here (Phase 2)

Retrieve flow (pseudo):

def retrieve_memories(query, top_k=8):
    q_emb = embed_text(query)
    hits = vector_store.query(vector=q_emb, top_k=top_k, filter=None)
    return [hit.metadata for hit in hits]

Fallback: If POCKETPAW_MEMORY_BACKEND != "vector" or vector store unreachable, continue using file_store.py.

Embeddings generated via Ollama (nomic-embed-text) and stored as JSON blobs in SQLite. Retrieval is pure cosine similarity in Python — load candidate vectors, compute dot products, rank by score. Candidate fetch is bounded at min(max(limit * 10, 200), 2000) to prevent unbounded memory allocation as the store grows.

Implementation notes:

I wrapped vector-store clients behind a simple abstraction (VectorStore) with upsert/query/delete methods so new stores can be added easily.
Embeddings loaded lazily and cached so we don't re-init heavy processes unless needed.

Phase 2 — Knowledge graph (what I added next)

Why a graph? Vectors give relatedness but lack explicit relationships and typed entities. The graph stores facts like "Project X — uses → React" and allows queries such as "What projects use Python?" or "What is related to deployment issue #42?"

Graph choices:

In-memory/ephemeral: NetworkX for development and small-scale setups.
Persistent: SQLite with adjacency tables for production/local persistence without needing a separate DB server.

Graph schema (SQLite, simplified):

nodes: id, label, type, metadata (json)
edges: id, source_node_id, target_node_id, relation, metadata (json)
node_refs: vector_id, node_id (to link vector docs to graph nodes)

Entity/relationship extraction:

Use a light LLM extraction prompt that asks for JSON: entities (name, type, span), relations (source, relation, target), confidence.
We run this asynchronously (non-blocking) to avoid delaying writes; low-latency systems can await it.

Example extraction prompt:

Input: conversation snippet or saved fact
Output (JSON):

{
  "entities": [
    {"name": "Project X", "type": "project"},
    {"name": "React", "type": "library"}
  ],
  "relations": [
    {"source": "Project X", "relation": "uses", "target": "React"}
  ]
}

Graph upsert:

Node deduplication by name + type + fuzzy matching.
Edge upsert with relation type and provenance metadata (source fact id).

Using the graph at retrieval:

After vector hits are found, expand the result set by traversing 1–2 hops in the graph for high-confidence relations.
Optionally prioritize recent nodes (timestamp decay) or nodes connected to high-importance tags.
Entities and relationships are persisted in knowledge_graph.sqlite3 across 6 tables, with PRAGMA user_version used for schema versioning and migrations. A key engineering constraint was SQLite's 999-variable limit — the initial DELETE ... WHERE doc_id NOT IN (...) cleanup query would crash on stores with 1000+ memories. This was replaced with a temp-table approach: valid IDs are batch-inserted (500 per batch) into a temporary table, and the delete joins against it instead of using an unbounded IN clause.

Putting it together — RAG + Graph hybrid

When the agent is about to produce an answer:

Generate a query from the current user message + short system prompt.
Retrieve top-K vector memories.
Expand context with graph neighbors of the top nodes (configurable depth).
Compose a memory context block containing:
- Short summaries of top vector memories
- Notable graph facts (entity relations)
- Timestamps + provenance
Feed the context to the LLM with the user prompt for generation.

This hybrid approach gives semantic relevance and structured connections.

Example: A user conversation

User: "Remind me why Project X is blocked."

Retrieval:

Vector search returns: "Project X blocked due to API refactor" (saved 2026-02-10)
Graph shows: Project X — depends_on → internal-api-v2
Graph shows: internal-api-v2 — breaking_change_in → Auth module

Agent response (composed using memory context):
"Project X is blocked because it depends on internal-api-v2, which recently introduced breaking changes in the Auth module — see note saved on 2026-02-10. Suggested action: pin internal-api-v2 to last working commit or open a patch for Auth compatibility."

Memory management: pruning, TTL, and UX

TTL/decay: each memory gets a scoring function that decreases with age unless referenced frequently.
Manual pruning: an admin UI lets users edit, delete, or merge memories and graph nodes.
Automatic consolidation: similar memories within a time window can be merged into a single canonical fact (configurable).

Testing

Large vector indexes can be paginated with approximate nearest neighbor (ANN) backends (Chroma uses hnsw, Qdrant has optimized indexing).
I added unit tests for:
- Embedding pipeline (mocked embeddings)
- Vector upsert/query semantics
- Graph upserts and traversal
- The RAG composition pipeline (integration tests, small in-memory stores)

Privacy & local-first constraints

A core design constraint: nothing leaves the user's machine by default.

Embedding model chosen can run locally (Ollama or other local providers).
Vector store choices include local SQLite-vec.
Graph stored locally in SQLite or as JSON files.
Documented that any cloud-backed vector store requires explicit opt-in.

Lessons learned

Don’t conflate similarity with semantics — vectors are phenomenal at relatedness but we still need typed edges for deterministic queries.
Keep the ingestion pipeline idempotent — upserts with deterministic IDs make retry and backfill simple.
Async extraction helps UX: write latency shouldn't be dominated by a second LLM call for entity extraction.

- Config-driven backends reduce friction for contributors and users — they can pick Chroma, Qdrant, or SQLite-vec.

Closing

Building semantic memory for PocketPaw transformed its behavior: it became better at recall, context stitching, and proactive assistance. The hybrid approach — vectors for fuzzy recall and a graph for explicit relations — balances flexibility with structured reasoning.

If you want the migration script, config examples, or the exact prompts I used for entity extraction, I can paste them here or open a docs PR for the repo.

Acknowledgements: thanks to the PocketPaw contributors and to the issue that seeded this work — https://github.com/pocketpaw/pocketpaw/issues/455
PR - https://github.com/pocketpaw/pocketpaw/pull/707

Author: AMRITESH240304 — Contributor to pocketpaw/pocketpaw

Make something Interesting

Amritesh Kumar — Mon, 09 Feb 2026 20:44:08 +0000

Swift On Server's

Amritesh Kumar — Wed, 24 Dec 2025 17:47:51 +0000

Vapor Drawing Backend - Project Overview

A collaborative note-taking backend built with Vapor (Swift) that supports real-time collaboration, authentication, and drawing strokes persistence.

Tech Stack

Framework: Vapor 4.115.0
Database: MongoDB with Fluent ORM
Authentication: JWT (JSON Web Tokens)
Real-time: WebSockets
Language: Swift 6.0

Features Implemented

1. Authentication System

JWT-based authentication with user registration and login functionality.

// User model with email and password
final class User: Model, Content, Authenticatable {
    static let schema = "users"

    @ID(key: .id)
    var id: UUID?

    @Field(key: "email")
    var email: String

    @Field(key: "password_hash")
    var passwordHash: String
}

Endpoints:

POST /api/v1/auth/register - Register new user
POST /api/v1/auth/login - Login user
GET /api/v1/auth/me - Get current user (protected)

2. JWT Middleware

Custom authentication middleware that verifies JWT tokens and protects routes.

struct AuthMiddleware: AsyncMiddleware {
    func respond(to request: Request, chainingTo next: any AsyncResponder) async throws -> Response {
        let payload: UserToken = try await request.jwt.verify(as: UserToken.self)

        guard let user = try await User.find(payload.userID, on: request.db) else {
            throw Abort(.unauthorized, reason: "User not found")
        }
        request.auth.login(user)

        return try await next.respond(to: request)
    }
}

3. MongoDB with Fluent ORM

Database configuration using FluentMongoDriver with automatic migrations.

// Configure MongoDB
try app.databases.use(
    .mongo(
        connectionString: Environment.get("MONGODB_URI") ?? "mongodb://localhost:27017"
    ),
    as: .mongo
)

// Run migrations
app.migrations.add(User.Migration())
app.migrations.add(NotesModel.Migration())

4. Notes Management

Full CRUD operations for notes with drawing strokes support.

final class NotesModel: Model, Content {
    @Field(key: "title")
    var title: String

    @Field(key: "strokes")
    var strokes: [DrawingStroke]

    @Parent(key: "user_id")
    var user: User
}

struct DrawingStroke: Codable {
    let points: [DrawingPoint]
    let color: DrawingColor
    let width: Double
    let timestamp: Date
}

Endpoints:

POST /api/v1/notes - Create note
GET /api/v1/notes - Get all notes
GET /api/v1/notes/get/:id - Get single note
PUT /api/v1/notes/:id - Update note
DELETE /api/v1/notes/:id - Delete note

5. Real-time WebSocket Collaboration

WebSocket manager for real-time collaborative editing with note session management.

final class WebSocketManager {
    private var connections: [UUID: WebSocket] = [:]
    private var noteCollaborators: [UUID: Set<UUID>] = [:]

    func joinNoteSession(noteID: UUID, userID: UUID)
    func leaveNoteSession(noteID: UUID, userID: UUID)
    func broadcastToNote(noteID: UUID, message: String, excludeUserID: UUID?)
}

WebSocket Features:

Join/leave note sessions
Real-time stroke updates
Broadcast to all collaborators (excluding sender)
Personal messaging

Endpoint: WS /api/v1/auth/handleInvite (protected)

6. Note Sharing

Share notes via JWT tokens with external users.

// Share token endpoint
GET /api/v1/notes/shared/:shareToken

// Verifies JWT token and returns note
let payload = try await req.jwt.verify(shareToken, as: ShareTokenPayload.self)

Project Structure

Sources/NoterPlayBackend/
├── Controllers/
│   ├── AuthenticationController.swift
│   ├── NotesController.swift
│   └── InviteController.swift
├── Middleware/
│   └── AuthMiddleware.swift
├── Models/
│   ├── User.swift
│   ├── NotesModel.swift
│   ├── UserToken.swift
│   ├── ShareTokenPayload.swift
│   └── InviteModel.swift
├── WSManager/
│   └── WebSocketManager.swift
├── configure.swift
└── routes.swift

Getting Started

# Install dependencies
swift package resolve

# Run the server
swift run

# With Docker
docker-compose up

Vapor Drawing Github

A lot to more to Fix it Up exploring just a small prototype.

Built with ❤️ using Vapor and Swift