AI Agent Memory: Add Semantic Search Without a Vector Database

#python #ai #tutorial #programming

📦 Clone and ⭐ stop-ai-agents-losing-memory-sample-for-aws

The agent's memory holds the answer. The user asks the question. And retrieval returns nothing.

stored:   dietary_notes: "Vegetarian; severe shellfish allergy — strictly no
          crustaceans or mollusks."

asked:    "What should I avoid eating when I go out for dinner on this trip?"

keyword scan: 4 hits, answer found: False

That's a real run, not a thought experiment. The question names no key and shares no words with the stored note, so the key-value memory from the previous post never finds it. The answer was in the store the whole time.

This is the dividing line for semantic search: do you know the key, or only the intent? When questions stop matching keys, you retrieve by meaning: embed each memory once, embed the question, return the nearest neighbors by cosine similarity. This post measures two things (whether semantic search finds what keyword search misses, and which vector store fits your deployment) using the same embeddings and the same memories in the companion repo.

(Post 2 of a series; the intro maps all the memory types. The code uses Strands Agents, an open source SDK; the pattern carries over to any agent framework.)

Why does key-value memory miss the question?

Because a key-value read is a lookup someone designed in advance, and this question maps to no key. The demo stores 10 memories about a traveler (profile facts, notes, episodes) and asks the dinner question against three stores. The key-value store has exactly two moves, and both fail honestly:

Keyword scan: match question words against keys and values. It returns 4 hits, none of them the allergy note, because "avoid eating at dinner" shares no words with dietary_notes or "shellfish". Answer found: False.
Dump-all fallback: give the model the entire memory and let it read. It works, at a price that grows with every memory you add. For these 10 memories that's 647 characters per question; for hundreds of notes it's thousands of tokens, every single question, forever.

This isn't a bug in key-value memory. Profile lookups ("what's my preferred cabin?") stay exact, instant, and free of embedding costs, which is why the previous post built them that way. The limit only appears when the question is semantic. That's the signal to add a second way in, not to replace the first.

How does semantic search find it?

By comparing meanings instead of words. Every memory is embedded once at write time into a vector (here: Amazon Titan Text Embeddings V2, 1,024 dimensions). At query time, the question is embedded and the store returns the nearest neighbors by cosine similarity:

top hit: "Vegetarian; severe shellfish allergy — strictly no crustaceans
          or mollusks."  (score 0.231)
answer found: True

No shared words between question and note. They're close in meaning, and meaning is what got indexed. Both backends below return this same top hit, because they use the same embeddings; what differs is everything around the query.

FAISS or Amazon S3 Vectors? Same accuracy, different deployment

Both are embedding vector stores. They use the same model (Titan V2), the same algorithm (cosine similarity), and they return the same top hit with the same score. The accuracy is identical this is not a quality trade-off.

Store	Finds the answer	Similarity score
Key-value (keyword scan)	No	— keyword miss
FAISS — Facebook AI Similarity Search, Meta's in-process vector index	Yes	0.231
Amazon S3 Vectors (managed cloud)	Yes	0.231

The difference is deployment. FAISS is an in-process library — zero infrastructure, a pip install, runs local to the process. It provides disk persistence via faiss.write_index / faiss.read_index. In this demo the index is not persisted and is rebuilt from scratch each run. S3 Vectors is a managed AWS service: the index lives in a cloud bucket, reachable from any process with AWS credentials, with no cluster to run or scale.

The demo uses the same AWS credentials for both: Titan embeddings via Bedrock and S3 Vectors via boto3; the same aws configure setup powers both, which is why this requires no extra setup inside a Strands Agents workflow. The demo self-provisions the bucket and index on first run: create_vector_bucket → create_index (1,024 dims, cosine) → put_vectors / query_vectors.

The cost both backends share: embedding the question costs ~510 ms with Titan V2. Index query time (0.09 ms for FAISS, 195 ms for S3 Vectors) is secondary to that. Plan for the embedding call in any latency-sensitive path, regardless of which vector store you pick.

So, do you need a vector database?

It depends on the query pattern. AWS positions S3 Vectors as "ideal for workloads where queries are less frequent", which describes agent memory exactly: an agent queries a user's memories a handful of times per conversation, not thousands of times per second.

	FAISS	Amazon S3 Vectors	Dedicated vector database
Type	In-process library	AWS vector storage	Full database engine
Examples	—	—	OpenSearch, Qdrant, Weaviate, Milvus, pgvector, Chroma
Semantic accuracy	✅ same	✅ same	✅ same
Infrastructure	None — pip install	None — fully managed	Self-hosted or managed
Max vectors	Process memory	Up to 2 billion per index	Depends on deployment
Query latency	~0.09 ms	~100–200 ms	Sub-10 ms at high QPS
Hybrid search	❌	❌	✅ most support it
Best for	Prototype / local agent	Cloud agent, infrequent queries	High QPS, advanced filtering, production search

The decision:

You need	Pick	Why
Facts under known keys (profile, preferences)	Key-value (post 1)	Exact and instant; don't pay ~510 ms of embedding for a lookup
Semantic search, local / prototype	FAISS	Zero infrastructure, pip install, in-process
Semantic search, cloud / infrequent queries	S3 Vectors	Purpose-built AWS vector storage, subsecond latency, up to 2 billion vectors, no infrastructure to manage
High QPS, hybrid search, or advanced filtering	Dedicated vector DB	OpenSearch, Qdrant, Weaviate, Milvus, pgvector, Chroma
Multi-hop questions over relationships	Graph (next post)	Semantic search finds pieces; it can't follow edges between them

What this demo does not cover: FAISS and S3 Vectors are storage backends. They store vectors and retrieve by similarity. Building what to remember (extracting specific facts from conversations, deduplication, structured memory across sessions) is handled by managed memory services like Amazon Bedrock AgentCore Memory. That technique is the topic of a future post in this series.

How does the agent choose between key lookup and semantic search?

From the tool docstrings, on its own. The demo's last test attaches both recall tools to one Strands agent:

@tool
def recall_by_key(key: str) -> str:
    """Recall a memory when the question maps to a known identifier.
    Use when the user asks about a stored field: "my preferred cabin",
    "my home airport"..."""

@tool
def recall_semantic(question: str, top_k: int = 3) -> str:
    """Recall memories by meaning when no key is obvious.
    Use for open questions: "what should I avoid eating on this trip?"..."""

Asked the dinner question, the agent calls recall_semantic; asked "what cabin do I prefer?", it calls recall_by_key. No routing logic, no prompt engineering. The when to use this sentence at the top of each docstring is what the model reads to decide. Write that sentence carelessly and the agent pays embedding latency for profile lookups.

How do you ask an AI coding assistant to build this?

The quality of the semantic search implementation your assistant builds depends on the decisions you name in the prompt. Unnamed, it will default to embedding everything and querying one big index. These five instructions encode what this post measured:

"Add semantic search only for questions that don't map to keys; keep profile facts in key-value state." Otherwise the assistant defaults to embedding every query, including exact lookups that already have a known key.
"Embed each memory once, at write time; only the question gets embedded at query time." Assistants love re-embedding the whole store per query.
"Use one embedding function for storage and queries, and state the model and dimensions." Mixed embedders produce silent garbage similarity scores.
"Give me two recall tools with 'when to use' docstrings: by key, and by meaning." The agent routes per question from those sentences; no router code.
"Make persistence explicit: in-process index for a prototype, managed vector storage for anything that must survive a restart, and prove it with a fresh-client test that still sees every vector."

The companion repo implements and measures all five. Run it to see each decision play out.

How do you run the demo?

git clone https://github.com/elizabethfuentes12/stop-ai-agents-losing-memory-sample-for-aws
cd stop-ai-agents-losing-memory-sample-for-aws/02-vector-memory-demo
uv venv && uv pip install -r requirements.txt
uv run python test_vector_memory.py

Needs AWS credentials (aws configure) for Titan embeddings and S3 Vectors. The demo creates the vector bucket and index automatically if they don't exist. OPENAI_API_KEY is only needed for the agent conversation in the notebook (or swap one line for Amazon Bedrock); the retrieval measurements run without any LLM.

FAQ

Is a vector database the same as AI agent memory?
No. A vector database is one possible backend for one memory type (retrieval by meaning). Agent memory is the whole system: key-value state, vector or graph storage, selection rules, and hygiene. Many production agents need vector retrieval without a vector database.

Can I use a vector database as agent memory?
Yes, for memories you'll query by meaning. But route keyed facts (preferences, settings) to key-value storage first: a direct lookup costs nothing, while every vector query pays the question-embedding call (~510 ms with Titan V2) before the index is even touched.

When do I need something beyond S3 Vectors?
When your query pattern changes. Dedicated vector databases such as OpenSearch, Qdrant, Weaviate, Milvus, pgvector, and Chroma are built for high QPS, hybrid keyword+vector search, aggregations, and advanced filtering. S3 Vectors is purpose-built for infrequent queries: it handles up to 2 billion vectors per index with subsecond latency, which covers agent memory workloads well past prototype scale.

What's the real latency difference between FAISS and S3 Vectors?
Measured on the same memories: 0.09 ms vs 195 ms per query. But embedding the question adds ~510 ms to both, so end-to-end it's ~0.5 s vs ~0.7 s. The index is rarely your bottleneck; the embedding call is.

Why did my semantic search return the wrong memories?
The most common causes: the store and the queries use different embedding models or dimensions, memories were embedded with stale text, or keyed facts polluted the index. Keep one embedder for everything, embed at write time, and keep profile facts out of the vector store.