Building Memory for an AI Career Advisor Was Harder Than Building the AI Itself

https://youtu.be/II6dkz-5QxI?si=2_Eipj3kjVIRXLvK

Building Memory for an AI Career Advisor Was Harder Than Building the Advisor Itself
I thought memory would be easy

While building an AI career advisor, I assumed memory would be straightforward.

Store past conversations. Pass them back into the model.

Done.

That approach failed almost immediately.

What the system actually does

The system provides:

resume feedback

skill gap analysis

internship tracking

interview support

But all of it depends on memory.

Without context, every feature becomes generic.

First attempt: store everything

I started by storing everything:

await db.save({
userId,
query,
response,
timestamp
});

Then I retrieved it like this:

const history = await db.getAll(userId);

const response = await llm.generate({
input: userQuery,
context: history
});

It worked for small inputs.

Then it broke.

The problem: too much context

Issues:

irrelevant data polluted responses

repeated information confused the model

token limits became a bottleneck

I wasn’t building memory. I was building noise.

The fix: structured memory

I split memory into three layers:

const profile = await getProfile(userId);
const events = await getEvents(userId);
const session = await getSession(userId);

Then filtered:

const context = {
skills: profile.skills,
projects: profile.projects,
recentEvents: events.slice(-5)
};

This improved quality significantly.

Using Hindsight

Instead of maintaining everything manually, I integrated
👉 https://github.com/vectorize-io/hindsight

The flow became:

const memory = await hindsight.retrieve(userId);

const output = await llm.generate({
input: userQuery,
context: memory
});

await hindsight.store(userId, {
query: userQuery,
response: output
});

This gave a structured pipeline:

recall

reason

store

What worked

selective retrieval

structured storage

limiting context size

What didn’t

dumping full history

relying only on prompts

ignoring time-based relevance

Behavior after improvements

Before:

repeated questions

no personalization

After:

consistent context

adaptive responses

Lessons

Memory is not storage—it’s retrieval

Context should be minimal but relevant

Systems should evolve after each interaction

Final thought

Building the AI part was straightforward.

Making it remember—properly—was the real engineering challenge.