DEV Community

technonotes-hacker
technonotes-hacker

Posted on

LLM - Memory for Agent

#agents #ai #llm

Memory Systems

  • End Goal is to create AI agent.
  • Agent is like a orchestrator + Integration of LLM + Action of Another tool + Memory.
  • We need to give MEMORY to the Agent.
  • Memory means --> இதுக்கு முன்னாடி நான் இது கிட்ட என்னென்ன கேட்டு இருக்கேன் & அது கொடுத்த விடை. [ Query & Response/Answer ],--> This combination is going to sit in the MEMORY.

  • How to store ? We cant't store continuously.
  • What is the LIMIT ?
  • There are types , lets discuss one by one.

List of Memory 

1. Buffer 
2. Window
3. Token
4. Summary
5. Long Term
6. Semantic
Enter fullscreen mode Exit fullscreen mode

BUFFER MEMORY

  • For each chat , we are going to store complete conversation, like below.
  1. In-memory --> RAM
  2. Persistent Storage --> like --> SQLITE3 , POSTGRES

  • For each and every call , whole conversation history is going to be sent to LLM.

  • For new chat , its going to have a DIFFERENT IDENTIFIER.

  • What problem can be identified in these 3 above points ? ஒரு சின்ன பிரச்சனை இருக்கு , அது என்ன ?

  • Point 2 --> We need to send the complete HISTORY to the LLM --> Tokens gets affected , also CONTEXT Limit gets affected. But you will get the output.

  • In the code , InMemorySaver() --> Will help to store in the RAM Memory.

  • With the capacity of RAM storage only it can store.

WINDOW MEMORY

  • It keeps only a FIXED NUMBER OF CHATS / MESSAGES. Eg., First message + Last N messages.
  • In Langchain --> There is a provision like a MIDDLE-WARE --> Modelகு முன்னாடி , இந்த Middlewareஅ தாண்டி தான் போகணும் .
  • In Middleware we can configure --> Remove message , just save the first & last messages alone.
  • Request goes to Middleware and then only MODEL.

  • What will pass in State and Runtime ?
  • Reply will be also taken along with the question.

  • Now remove all messages and stores only 5 messages.

  • This memory scenario is used only when recent conversation are important then OLD/PREVIOUS conversation.

TOKEN MEMORY

  • Similar to WINDOW memory.
  • Will set the tokens. Eg., 2000 Tokens .
  • Not going to store more than 2000 Tokens .

  • Token definition will defer from Model to Model and framework to framework.
  • Eg., 1000 tokens --> each character is a token in the above text.
  • Old data's will be removed.

  • arised ……………………….Ipsum → இவ்ளோ words இந்த paragraph ஆயிரம் டோக்கன் வெச்சுக்குங்க ( 1000 tokens ) , but popularised word is broken , பாதியிலிருந்து அந்த word வந்து எடுத்துக்குது , நம்ம இதுல set பண்ணலாம் , பாதி word வெச்சிக்கிட்டு நம்ம என்ன பண்ண முடியும் . Even though 1000 tokens are used , to get the proper meaning , it will take ( that means we will set ) , take from the start of the sentence. “ It was popularised …… Ipsum .” --> This will be handled by the Human conversation message.

  • trim_messages() --> Function will do the honour in this.

  • In scenarios --> It will be useful only for recent messages. OLD messages will be deleted as per the token size.

  • Anything older than the given budget is lost completely. --> this is the DOWNSIDE.

SUMMARY MEMORY

  • Human & AI → இதுல AI response பண்ணுது and Human கேள்வி கேக்குறாங்க. இப்ப இதுல , memoryல 4 conversation தான் store பண்ண முடியும் but first ரெண்டு போயிடும் . அதுக்கு முன்னாடி , ஒரு summary மாதிரி நம்ம save பண்ணி வச்சுக்கலாம் . So that , even the data gets deleted but we will have the summary. This summary can be created by passing to a cheap LLM for cost effectiveness.
  • For collating the summary --> Use Cheap Model.
  • SummarizationMiddleware () --> This function will be used.

LONG TERM MEMORY

  • Agent will decide were to store and what to store .

SEMANTIC MEMORY

  • Entire conversation into a POINT.
  • Its like in RAG.
  • Store in Vector DB.

Questions & Synopsis

Model needs memory ? --> They remember nothing: every call is fresh. The app must resend earlier messages.

Buffer memory --> Keep every message. Simple, but it grows forever (cost and size limit).

Window Memory --> Keep the first message plus the last few. Forgets the middle.

Token budget --> Keep the newest messages that fit a token limit (trim_messages). Still forgets old things.

Summary memory --> Replace old messages with a short summary so facts survive. Costs an extra AI call.

Long-term memory --> Facts saved in a Store, shared across a user's conversations. The agent decides what to save.

Semantic memory --> Long-term facts searched by meaning with embeddings. Needs an embedding model.

Checkpointer --> Saves the conversation after each turn (InMemorySaver; SqliteSaver or a database for persistence).

thread_id --> The conversation's file number. Same id continues it; a new id starts fresh.

Middleware (@before_model) --> A function that runs right before each AI call and can edit the message list.

Store vs checkpointer --> Checkpointer = one conversation. Store = lasting facts about the user across conversations.

user_id in memory tools --> Comes from your app's context, not from the AI, so users can't read each other's memories.

Notes

  • What is In-memory --> its RAM.
  • What is Langchain ?
  • What is decorator ?
  • what is lorem ipsum ?
  • What is Key word ?

Top comments (0)