AI is getting better and better really fast. It can do amazing things now, like complex calculations, writing code, and figuring out data patterns. But it still struggles with a basic skill that humans learn naturally: remembering personal tastes and talking to people in a way that fits them.
Today, I want to share an illuminating experiment that compares how different AI systems handle user preferences. We'll examine the behavior of traditional AI assistants like ChatGPT and Letta (previously known as MemGPT), an open-source solution specifically designed to address the challenge of long-term memory in AI systems. Through this comparison, we'll explore how different approaches to AI memory can dramatically impact the user experience and potentially reshape how we think about AI assistants.
The Session Problem: A Modern Day Groundhog Day
Imagine having a highly capable assistant who, every morning, completely forgets everything they learned about your work style the day before. They're brilliant at their job, but you must repeatedly explain that you prefer detailed reports, need technical explanations broken down step-by-step, or want your data presented in specific formats. This is the current reality with most AI systems.
The root of this problem lies in how traditional AI systems operate: each conversation exists in isolation, a fresh start without any memory of past interactions. While this approach has its benefits for privacy and resource management, it creates a frustrating user experience that fails to mirror natural human interaction.
The Experiment: Testing AI Memory
To demonstrate this challenge, I conducted a simple yet revealing experiment with three different AI systems: ChatGPT, Claude, and Letta. The test was straightforward - ask each system about Joe Biden, express a preference for more detailed information, and then ask the same question in a new session to see if the preference was retained.
With ChatGPT, the initial interaction followed a predictable pattern. It provided a concise response: "Joe Biden is the 46th President of the United States, serving since January 20, 2021." When asked for more detail, it adapted beautifully, offering comprehensive information about Biden's career, policies, and background. However, in a new session, this learned preference vanished, and we were back to square one.
Having received a very brief reply to my question from Openai, I ask it not to write so briefly. If this were a human being, he would probably remember this piece of preference.
But as expected, when I start a new session and ask the same question, I get exactly the same short answer.
Custom instructions for ChatGPT
Top comments (0)