Agent Memory Is Not "Vector Search Pasted Back Into the Prompt"

Most agent-memory demos can be summarized in one line:

conversation → chunks → embeddings → top-k → prompt

That is useful, but it is not yet a reliable memory system.

We learned this while building SPM-Polaris, a provider-neutral memory and context layer. Once the system was placed inside real Chat/Responses/Messages request paths, the harder questions appeared after retrieval:

• Is the retrieved text tied to a source?
• Is it a user statement, an observed fact, or an assistant proposal?
• Is it safe to use now, or has it been superseded?
• Can an old exchange be removed without breaking a tool call or provider-managed state?
• Does deletion remove vectors, candidates, and delayed jobs as well as the primary record?
• Does an MCP integration actually invoke memory at the right time?

The result was a shift in our design vocabulary. We stopped treating "memory quality" as one retrieval number and started treating memory as a request and data lifecycle.

1. Long context did not eliminate the memory problem

Long windows are valuable, but they do not make all historical content equally useful. Research on long-context utilization and long-context RAG shows that information position and hard negatives can affect answer quality. A provider's stateful API may simplify lineage, but growing history can still carry token cost.

For a week-long coding task, replaying everything is often both expensive and counterproductive. The system needs a way to retain the latest task and protected protocol state, retrieve relevant evidence, and remove only history that can be safely represented by that evidence.

2. We separated "found" from "allowed to use"

SPM-Polaris has a candidate-retrieval stage and an evidence-admission stage. Similarity is not sufficient. A result must retain source boundaries and satisfy the gate before it can replace historical exchanges.

That separation changed how we report results. Evidence containment ("did the payload include the source material?") and answer accuracy ("did the downstream reader produce the right answer?") are different measurements, and quoting a containment figure as a memory score is misleading. We previously published internal numbers for both; we have since withdrawn them pending a reproducible manifest — including one result we invalidated ourselves after finding protocol drift — rather than let them circulate without their evidence pack.

When nothing qualifies at the gate, the system returns UNKNOWN and injects nothing, instead of guessing with plausible-looking context.

3. Compression is mainly a state-integrity problem

Deleting the oldest messages until a token budget is met is unsafe for a tool-using agent. A request can contain system/developer instructions, function-call identifiers, tool results, multimodal blocks, provider reasoning state, and conversation lineage.

Our rule is deliberately conservative: an old complete exchange is removable only when admitted evidence covers its precise source set and no protected provider state is damaged. When the proof is missing, the request passes through unchanged.

This creates a seemingly odd but important product behavior: sometimes the correct compression ratio is zero.

4. Proxy and MCP are different experimental paths

The Hosted or Local Proxy is inline. It can automatically recall memory, compile context, and record the result. MCP exposes tools, but the host or model decides whether to invoke them and where to insert the response.

The same memory backend therefore does not guarantee the same outcome. A proper comparison needs matched models, tool schemas, invocation policy, return size, placement, judge, and seeds. We now treat integration_path as part of the benchmark manifest.

5. What we can currently report

Single observations from our production path — each with its scope, none of them averages, SLOs, or third-party certification:

• one eligible hosted Provider Proxy request: 66,265 original tokens → 365 forwarded + 279 recalled (a single request; ineligible histories may show no reduction at all);
• one measured Starter rate-limit window: a 45-request burst against a 30-requests-per-minute window returned 30 successful requests and 15 HTTP 429 responses;
• one production MCP remember/status round trip: ready in 8.23 seconds;
• one production MCP remember-to-read round trip: stored and read text matched byte-for-byte;
• one warm production MCP recall: about 4.93 seconds.

Every agentic request also emits an auditable receipt, including original and forwarded token counts for elided tool output, so removed history appears as real savings instead of a passthrough zero.

The limitations are just as important:

• the raw request and receipt manifests for the observations above are still being prepared for public inspection;
• we do not yet have public p99, sustained soak, or 100+ concurrency data;
• production charging is disabled;
• several moderation, abuse-protection, and reservation-recovery items remain open;
• Proxy results do not automatically transfer to MCP.

6. The questions we think matter next

The next generation of agent-memory evaluation should measure more than retrieval:

1. write fidelity and source authority;
2. evidence admission under conflicts and hard negatives;
3. whether admitted evidence actually enters context;
4. provider-state integrity after compression;
5. updates, supersession, and abstention;
6. deletion completeness under delayed jobs and reindexing;
7. answer quality, token cost, latency, and task success by integration path.

Our current view is simple: an agent-memory system should be judged not by how much it stores, but by whether it can explain why a memory was used, what it safely replaced, and what happened when the evidence was not good enough.

Disclosure: I am affiliated with the team building SPM-Polaris at Veridical Tech. The figures above are single first-party engineering observations, not independent third-party certification. Documentation: https://docs.spmos.ai/