My home cluster runs a small fleet of agents. They tell me which bin goes out tonight, what is on sale at the supermarket, and how many articles live in my blog repo. They are genuinely useful, and they have the memory of a goldfish.
Every message starts from nothing. Ask "what did I ask you yesterday?" and the router replies, with total confidence, that it has no access to any history. It is not wrong. There was no history — just traces piling up in an observability tool that no agent could read.
The fix is a vector database — Qdrant, in this case. The interesting part is not the fix. It is the four things between "Qdrant is running" and "Qdrant actually answers", none of which are in anyone's documentation.
Why Cloud, and why the URL matters
The cluster is three Raspberry Pis. They have opinions about being asked to run a vector database, back it up, and snapshot it. Qdrant Cloud has a free tier with a gigabyte of storage, which for conversational memory is roughly forever, and it moves storage and backups off hardware that has none to spare.
You get a cluster and an endpoint. The endpoint looks like this:
https://11111111-2222-3333-4444-555555555555.eu-west-1-0.aws.cloud.qdrant.io:6333
That trailing :6333 is the first thing that will cost you an afternoon. The console shows you the bare hostname, and a bare hostname resolves to port 443, where the REST API is not listening. The client does not tell you that you are talking to the wrong port — it just fails to connect, and you go looking at network policies and DNS instead of at the string you pasted.
The hostname is not a secret. The API key is, and it lives in a Secret created out of band, exactly like every other credential in this cluster:
kubectl create secret generic qdrant-cloud -n kmcp-system \
--from-literal=QDRANT_API_KEY='<key>'
Not in the repo. Not in a values file. The repo knows the name of the secret and nothing else.
The architecture
flowchart TD
A["kagent agents — Telegram front door"] -->|"emit OTLP spans"| B["Collector — drops framework noise"]
B -->|"HTTPS"| C[("Langfuse Cloud — trace store")]
D["CronJob — every 30 minutes"] -->|"public read API"| C
D -->|"embed with FastEmbed — 384 dims"| E[("Qdrant Cloud — conversations")]
F["MCP server — read only"] -->|"qdrant-find"| E
G["Recall agent"] -->|"MCP over HTTP"| F
H["Router agent"] -->|"A2A delegation"| G
A -.->|"same fleet"| H
Two paths, deliberately separate. Writes come from a scheduled job that reads traces the agents already emit. Reads go through an MCP server that any agent in the fleet can call.
The alternative was letting each agent save its own conversation as it goes. It sounds cleaner and it is worse: it depends on the model remembering to call a tool, it writes the same conversation several times when one agent delegates to another, and it adds an embedding round-trip to every reply the user is waiting on. Reading traces that already exist costs the agents nothing and cannot be forgotten.
For the people who sign off on this
Skip to the next section if you enjoy stack traces.
Three things worth knowing before anyone approves a similar build.
The data was already there. Nothing new is being collected. The agents were already emitting traces for observability; this reads the same traces and makes them searchable. No new consent surface, no new source of truth, one storage location added.
Running cost is zero and stays that way. Free tier at both ends — a gigabyte of vectors and a trace quota that a household does not threaten. Embeddings are computed on hardware that is already paid for. The cost line that matters is not the bill, it is the half hour a month somebody spends when a version pin needs moving.
The honest scope. This makes "what did we decide about X" answerable from a search box instead of from somebody's memory. At household scale that is a nice trick. At team scale — runbooks, architecture decisions, incident write-ups, onboarding — the same shape stops being a trick, because the cost of not having it is five people asking the same question every quarter and getting four different answers.
Where the conversation actually lives
The first version of the ingest job read input and output from each trace, embedded them, and stored zero points. Both fields were null on all 97 traces.
They are null because kagent does not set them at trace level. The conversation is in the observations — 57 of them for a single question, most of which are framework plumbing. The user-facing exchange is in the handful of GENERATION observations whose input is a chat-message list:
def conversation_turn(observation):
raw_input = observation.get("input")
if not isinstance(raw_input, list) or not raw_input:
return None
# Internal model calls carry a system_instruction and return a
# function_call. Embedding those fills the index with prompt
# boilerplate instead of anything a human said.
There is a second layer to peel. The user's turn arrives double-encoded — a JSON string holding a list of content blocks — so the raw field is '[{"type": "text", "text": "which bin goes out tonight?"}]'. Embed that verbatim and you have embedded the punctuation as much as the question.
The lesson generalizes past this stack: look at one real record before you write the parser. I wrote a plausible parser against a plausible schema, and the schema was fiction.
Deduplication you cannot forget to do
Each run looks back over a window wider than its own schedule, so a failed run heals on the next one. That only works if re-reading a trace is free:
POINT_NAMESPACE = uuid.UUID("6f9619ff-8b86-d011-b42d-00c04fc964ff")
point_id = str(uuid.uuid5(POINT_NAMESPACE, trace_id))
A point's ID is a UUIDv5 of the trace ID, so re-reading a conversation overwrites its own point instead of adding a twin. No cursor, no checkpoint, no state to corrupt. The overlap is free by construction rather than by bookkeeping, which is the only kind of free that survives a job being killed halfway through.
Four constraints nobody wrote down
This is the part worth the read.
Named vectors, not anonymous ones. The official MCP server stores and queries named vectors — internally vector={name: embedding} and searches with using=name. A collection created with a plain VectorParams accepts every write and rejects every read:
400 Bad Request: Wrong input: Not existing vector name error: fast-all-minilm-l6-v2
The name is derived from the model, so derive it the same way rather than pasting the string:
def vector_name(model_name):
return f"fast-{model_name.split('/')[-1]}".lower()
Filters address a nested path. Once I wanted date filtering, the tool arguments appeared and returned nothing at all. Filters resolve against metadata.<field>, not the payload root. Flat top-level keys are still handed back to the caller, which is exactly why this is confusing — the data is visibly there and invisibly unfilterable. Everything except the document text moved under a metadata key.
One condition per field. A filterable field carries a single comparison, and the set is keyed by name, so a range needs two fields with different names. Hence day_from compared with >= and day_to with <=, both written with the same value:
filterableFields:
- name: day_from
field_type: integer
condition: ">="
- name: day_to
field_type: integer
condition: "<="
The types on offer are keyword, integer, float, boolean — none of which can order an ISO timestamp. The date is stored a second time as a plain 20260818 integer, purely so it can be compared.
Range queries need an index, and the reader cannot create one. Qdrant refuses a range filter on an unindexed key:
400 Bad request: Index required but not found for "metadata.day_from"
The MCP server runs with QDRANT_READ_ONLY=true, so it has no business creating indexes and does not. The ingest job owns them and reconciles them on every run, because collections built before the filters existed have none.
Field notes
A filtered count equal to your search limit is truncation, not a leak. Filtering to a single day returned 25 results and I briefly concluded the filter was broken. There were 46 conversations that day and the limit was 25. Check the underlying distribution before you debug the filter.
Read the installed package before you build the missing feature. I was ready to write a second MCP server with date filtering. Ten minutes with inspect.getsource() on the running pod showed the feature already existed, configured by an environment variable that appears in no example I could find. The tool I did not write is the best code I shipped that day.
A shell timeout is not a deployment failure. Two helmfile apply runs died at 120 seconds and left a release stuck in pending-install. The chart was fine; my command was impatient. Helm was waiting on a PVC to bind. Check the resource, not the exit code — and if you cut a deployment off at the knees, clean up the half-installed release before retrying.
The free tier rate-limits harder than you plan for. Langfuse allows 15 requests per minute, and this job needs one per trace. Unpaced, it spent its entire allowance in about a second and then sat in backoff. Pacing at 4.5 seconds per request and honouring Retry-After is slower per request and faster overall.
glibc or bust on arm64. FastEmbed pulls ONNX Runtime, which publishes manylinux_2_28_aarch64 wheels and nothing for musl. On an Alpine base, pip quietly starts building it from source and never finishes on a Pi. python:3.12-slim and the wheel drops in.
Ask, don't guess, when the date is missing. Resolving "today" needs today's date, and nothing guarantees the model was told it. A wrong day returns the wrong window silently — the query succeeds and the answer is confidently incomplete. The agent is instructed to ask, or to fall back to unfiltered search and say so, rather than assume.
Closing
Sixty-one conversations indexed, filtered by date, answered through a tool the agents already knew how to call. Ask what you requested on a given day and you get the exchange back, quoted, with its timestamp.
The vector database was the easy part — an endpoint, a key, an afternoon. Everything between a running Qdrant and a working recall system was a schema that was not documented, a payload path that was not obvious, an index nobody said you needed, and a feature I nearly rebuilt because I did not read the source first.
Top comments (0)