Integrating ByteDance deer-flow: Vector Memory Persistence Patterns for Long-Horizon Agents

Context: Why Long-Horizon Agent Memory is Hard

When evaluating ByteDance's deer-flow for our multi-hour autonomous research pipeline, the core challenge wasn't task orchestration—it was semantic memory persistence under high concurrency. Agents running 90+ minute sessions need to recall prior decisions, avoid redundant API calls, and maintain coherent context across sandbox restarts.

Most agent frameworks treat memory as an afterthought. deer-flow takes it seriously with a pluggable memory backend, but integrating it into our existing PostgreSQL + pgvector stack exposed real architectural friction points.

Architecture Overview: Memory Chain in deer-flow

deer-flow splits memory into three layers:

1. Session buffer (ephemeral, in-process)
2. Vector store (semantic retrieval via embeddings)
3. Relational metadata (timestamps, task IDs, lineage)

The critical insight: hybrid search must fuse vector similarity with temporal recency and task ancestry filters. Pure cosine similarity retrieves semantically related memories but ignores causality.

Deliverable 1: PostgreSQL Schema with HNSW Indexing

Here's our battle-tested schema integrating deer-flow's memory interface with pgvector:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE agent_memories (
  id BIGSERIAL PRIMARY KEY,
  agent_id TEXT NOT NULL,
  task_id TEXT NOT NULL,
  content TEXT NOT NULL,
  embedding vector(1536) NOT NULL,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  parent_task_id TEXT
);

-- HNSW index for sub-100ms vector search at 100K+ rows
CREATE INDEX ON agent_memories 
  USING hnsw (embedding vector_cosine_ops) 
  WITH (m = 16, ef_construction = 64);

-- Compound index for filtered vector search
CREATE INDEX ON agent_memories (agent_id, task_id, created_at DESC);

Critical tuning lesson: Setting m=16 and ef_construction=64 gave us 12ms p99 latency under 500 concurrent inserts. Default pgvector settings caused 2+ second insert stalls during batch memory writes.

Deliverable 2: Hybrid Search with Reciprocal Rank Fusion

Pure vector search failed during complex multi-step tasks—agents would retrieve tangentially related memories from 3 hours prior while ignoring critical decisions from 10 minutes ago.

Here's our Python RRF implementation bridging deer-flow's memory interface:

import asyncpg
import numpy as np
from typing import List, Tuple

class HybridMemoryRetriever:
    def __init__(self, pool: asyncpg.Pool, k: int = 60):
        self.pool = pool
        self.k = k  # RRF constant

    async def retrieve(
        self, 
        agent_id: str, 
        query_embedding: np.ndarray, 
        task_id: str,
        limit: int = 5
    ) -> List[Tuple[str, float]]:
        async with self.pool.acquire() as conn:
            # Vector similarity ranking
            vector_results = await conn.fetch(
                """
                SELECT id, content, 
                       1 - (embedding <=> $1::vector) AS similarity
                FROM agent_memories
                WHERE agent_id = $2 AND task_id = $3
                ORDER BY embedding <=> $1::vector
                LIMIT 20
                """,
                query_embedding.tolist(), agent_id, task_id
            )

            # Recency ranking
            recency_results = await conn.fetch(
                """
                SELECT id, content, created_at
                FROM agent_memories
                WHERE agent_id = $1 AND task_id = $2
                ORDER BY created_at DESC
                LIMIT 20
                """,
                agent_id, task_id
            )

            # RRF fusion: 1 / (k + rank)
            scores = {}
            for rank, row in enumerate(vector_results, 1):
                scores[row['id']] = scores.get(row['id'], 0) + 1 / (self.k + rank)

            for rank, row in enumerate(recency_results, 1):
                scores[row['id']] = scores.get(row['id'], 0) + 1 / (self.k + rank)

            # Retrieve top-k by fused score
            top_ids = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:limit]

            final = await conn.fetch(
                "SELECT content FROM agent_memories WHERE id = ANY($1)",
                [id for id, _ in top_ids]
            )

            return [(r['content'], scores[r['id']]) for r in final]

Battle scar: We initially used a simple weighted sum instead of RRF. This failed catastrophically when vector scores clustered near 0.85-0.90—recency signals got drowned out. RRF's rank-based scoring proved far more stable.

Production Integration Points

When wiring deer-flow into our existing stack:

Embedding normalization trap: deer-flow doesn't enforce L2 normalization. We saw 22% recall degradation until we added:

def normalize_embedding(vec: np.ndarray) -> np.ndarray:
    norm = np.linalg.norm(vec)
    return vec / norm if norm > 0 else vec

Concurrency bottleneck: HNSW index locks during bulk inserts caused 5+ second write stalls. Solution: batch memories and use INSERT ... ON CONFLICT DO NOTHING with a 50ms debounce window.

Benchmarking methodology: When testing multi-provider fallback latency through B-Lost's relay gateway (baseURL: "https://b-lost.com/v1"), we clocked 127ms average embedding generation (text-embedding-3-small) which became the bottleneck—not database writes. Consider pre-generating embeddings asynchronously.

Operational Trade-offs

What works:

• Pluggable memory abstraction makes swapping PostgreSQL for Qdrant trivial
• Subagent lineage tracking via parent_task_id enables causal retrieval filters
• Clean separation between ephemeral buffers and durable storage

What hurts:

• No built-in memory compaction—our agents hit 50K+ rows per session, causing index bloat
• Lack of automatic embedding cache invalidation when upstream models change
• Hybrid search logic lives in application code, not the framework

The Open Question

Our biggest unresolved dilemma: when should agents forget? Unlimited memory growth killed performance after 6-hour sessions. Naive LRU eviction broke causal chains. Time-decay scoring helped but introduced new failure modes during context switches.

How do you balance memory retention with retrieval performance when agents run for hours? Curious what approaches others have battle-tested in production long-horizon systems.