An Agent That Remembers What You Actually Need

Adbuth kumar Kota — Tue, 29 Sep 2026 14:17:35 +0000

A grocery assistant can answer “Where is the bread?” in a second.

That is not the interesting part.

The harder problem is answering the next question correctly when the customer has already told the system something important earlier.

For example:

“Naku brown bread ante chala istam, white bread vaddu.”

Later:

“Which bread should I buy?”

A normal chatbot can understand the second sentence. A useful grocery assistant should connect it to the first one.

That is what I built with the GrocerAI Customer Agent: an AI shopping assistant that combines real-time store data, tool-based actions, multilingual conversations, customer-specific memory, and a persistent learning loop.

The goal was not to build another chatbot.

The goal was to make the customer experience become more useful over time.

The Problem With a Normal Shopping Chatbot

A grocery assistant has to answer questions about things that change constantly:

Is a product available?
Where is it located?
What alternatives are available?
What is already in the cart?
What is still needed for the shopping mission?
Has the customer mentioned a preference before?

These are different kinds of information.

Some belong to the current store state.

Some belong to the customer's current session.

Some are experiences that may become useful later.

I therefore designed the Customer Agent around three layers:

Current Store State
        +
Current Customer State
        +
Relevant Past Experience
        ↓
Customer Agent
        ↓
Action / Response

The structured backend remains the source of truth for current facts. Hindsight provides the memory layer for experiences that can influence future reasoning.

The Customer Agent Is Tool-Driven

The LLM is responsible for understanding the customer's request and deciding what information or action is needed.

It does not directly invent inventory or shelf locations.

Instead, it uses backend tools such as:

searchProducts()
getProductDetails()
checkAvailability()
getShelfLocation()
getStoreMapRoute()
getCustomerMission()
updateMission()
getCart()
addToCart()
removeFromCart()
getRecommendations()
checkSubstitution()
sendNotification()

Consider:

“Do you have brown bread?”

The flow can become:

Customer
   ↓
Customer Agent
   ↓
searchProducts()
   ↓
checkAvailability()
   ↓
getShelfLocation()
   ↓
Answer

That distinction matters.

The model handles reasoning.

The application handles authoritative store state.

This makes the assistant much more useful than simply asking an LLM to generate an answer from its own knowledge.

Adding Persistent Customer Memory

The biggest change came when I added Hindsight.

The Customer Agent can retain useful experiences and recall them when a future interaction makes them relevant.

Conceptually:

Customer Interaction
        ↓
      retain()
        ↓
Customer Memory
        ↓
Future Interaction
        ↓
      recall()
        ↓
Personalized Reasoning
        ↓
Relevant Action

I kept customer memories isolated using customer-specific memory banks:

grocerai-customer-USER00001
grocerai-customer-USER00002
grocerai-customer-USER00003

This is important because personalization is only useful if the memory belongs to the correct customer.

A preference from USER00001 should never influence USER00002.

I explicitly tested this isolation rather than assuming it would work.

Memory Should Survive Language Changes

GrocerAI supports English, Telugu, Hindi, and mixed-language conversations.

That created an interesting test.

A customer can express a preference in Telugu:

"Naku brown bread ante chala istam,
white bread vaddu."

Later, they can ask in English:

"Which bread should I buy?"

The useful behavior is not literal sentence matching.

The agent needs to retrieve the underlying preference and connect it to the new request.

The same idea applies to other customer experiences, such as remembering lactose intolerance and later using that information when discussing alternatives such as oat milk.

That is where persistent semantic memory becomes more useful than keeping a giant conversation transcript.

The Customer Agent Also Understands Shopping Missions

Shopping is rarely just a sequence of independent questions.

A customer may have a mission:

Breakfast
 ├── Bread
 ├── Milk
 ├── Cereal
 └── Fruit

The Customer Agent can work with that mission while the customer browses the store.

It can identify what has already been completed, what is still needed, and what actions can help the customer finish the mission.

This turns the interaction from:

“Ask a question → get an answer”

into:

“Start a shopping goal → receive assistance throughout the journey.”

Voice Uses the Same Agent

I did not want voice to become a separate intelligence layer.

The voice flow is:

Voice Input
    ↓
Speech-to-Text
    ↓
Customer Agent
    ↓
Backend Tools
    ↓
Reasoning
    ↓
Response
    ↓
Text-to-Speech

The same customer state, tools, mission, and memory can therefore be used whether the customer interacts through text or voice.

That keeps the architecture consistent.

From Big Store Display to Personal Companion

GrocerAI also separates the physical store experience from personal customer assistance.

The main store display handles shopping interactions such as browsing, searching, product information, cart actions, navigation, and checkout.

A customer can scan a QR code to activate a personal companion session.

That session gets a customer identifier and can provide personalized information such as:

shopping mission progress
remaining items
recommendations
notifications
preferences
availability
personalized aisle guidance
synchronized cart information

The important architectural rule is that the mobile experience is not a second independent cart.

It shares the customer's active shopping session with the store display.

The Complete Customer Learning Loop

The final Customer Agent loop became:

OBSERVE
   ↓
RECALL
   ↓
REASON
   ↓
ACT
   ↓
RE-OBSERVE
   ↓
REMEMBER

For example:

Customer says preference
        ↓
retain experience
        ↓
Later bread request
        ↓
recall preference
        ↓
Reason about suitable products
        ↓
Recommend relevant option
        ↓
Observe customer action

The important part is that memory is inside the decision cycle.

It is not simply a page showing stored memories.

What I Learned

1. An LLM should not be the database

Current inventory, cart state, mission state, and orders need deterministic application data.

The model should reason over that information rather than becoming the source of truth.

2. Memory should be selective

Not every conversation sentence deserves to become a long-term memory.

Useful memory is information that can improve a future decision.

3. Personalization requires isolation

A memory system that accidentally mixes customers destroys trust.

Customer-specific memory boundaries therefore became a first-class architectural requirement.

4. Multilingual support becomes more valuable with memory

Understanding Telugu or Hindi is useful.

Remembering a preference expressed in one language and applying it later in another makes the interaction much more continuous.

5. The real test is future behavior

I did not want to measure success only by whether the agent could store a memory.

The stronger test was:

Did recalling that experience change what the agent did next?

That became the core test for the Customer Agent.

Final Architecture

The Customer Agent ended up as a combination of:

             Customer
                 ↓
        ┌─────────────────┐
        │ Customer Agent  │
        └────────┬────────┘
                 ↓
        ┌─────────────────┐
        │ Groq Reasoning  │
        └────────┬────────┘
                 ↓
      ┌──────────┴──────────┐
      ↓                     ↓
Backend Tools          Hindsight Memory
      ↓                     ↓
Current State       Past Experiences
      └──────────┬──────────┘
                 ↓
          Personalized Action

The result is not simply an AI that can talk about groceries.

It is an agent that can understand a shopper, use the actual store as its source of truth, remember useful experiences, and use those experiences when the customer returns.

That is the behavior I wanted from a personal grocery assistant:

not just answering the customer's next question, but becoming more useful because of what it learned from the previous one.

DEV Community: Adbuth kumar Kota