How I Built Memory for REXA Using PostgreSQL and pgvector
One thing I wanted REXA to eventually have was long-term memory.
The idea is simple: when I explicitly tell REXA something like:
“Remember that I prefer PostgreSQL for my backend projects.”
REXA should be able to save that information so it can be used later.
The important distinction is that REXA does not currently retrieve these memories yet. Today, the implemented part is the memory-saving pipeline. Retrieval is the next part I plan to build.
I also didn't put the memory database directly inside the REXA CLI.
Instead, I built the memory infrastructure behind the REXA website backend and let the CLI communicate with it through an authenticated HTTP API.
The architecture looks like this:
REXA CLI
│
│ POST + Bearer Token
│ { "text": "..." }
▼
REXA Website Backend
│
├── Verify Token
├── Identify User
├── Process Text
├── Generate Embeddings
└── Store Memory
│
▼
PostgreSQL + pgvector
The CLI Does Not Silently Remember Everything
One important design decision is that REXA does not automatically save every preference or every piece of conversation.
Instead, the model uses a tool called:
save_memory
When the user explicitly asks REXA to remember something, the model calls this tool.
For example:
User:
Remember that I use Bun for my backend projects.
REXA can decide that the appropriate action is to call:
save_memory(...)
So memory creation is currently explicit, rather than REXA silently storing everything the user says.
1. REXA Calls save_memory
The save_memory tool receives the text that the user wants to store.
Before sending it to the backend, the CLI performs some basic validation.
The CLI:
- trims the text
- rejects empty input
- limits the text to 8192 characters
This means extremely large documents may never leave the CLI in the first place.
For normal memory entries, however, the text can then be sent to the backend.
2. The CLI Sends an Authenticated HTTP Request
The REXA CLI does not directly access PostgreSQL or pgvector.
Instead, it sends a POST request to the REXA backend:
POST https://rexa-server.onrender.com/api/cli/memory
Authorization: Bearer <token>
Content-Type: application/json
The request body contains only the memory text:
{
"text": "I prefer PostgreSQL for my backend projects."
}
Notice that there is no userId in the request body.
The identity of the user comes from the authentication token.
3. Where Does the Token Come From?
The token comes from the REXA CLI login flow.
The CLI first authenticates through the REXA backend using:
POST /api/cli/verify
The same Bearer-token mechanism is then used when making authenticated CLI requests.
This means the CLI doesn't simply tell the backend:
{
"userId": "123",
"text": "..."
}
Instead, it says, effectively:
Here is my authentication token.
Here is the memory I want to save.
The backend is responsible for determining who that token belongs to.
4. The Backend Verifies the Token
When the memory request reaches:
/api/cli/memory
the backend first checks the Bearer token.
If the token is missing, invalid, or expired, the request is rejected.
The CLI can surface these authentication failures and provide a hint to run:
rexa login
This makes the authentication layer separate from the actual memory-processing logic.
5. The Backend Identifies the User
After successful token verification, the backend knows which authenticated user is making the request.
This is important because memory is per user.
The CLI doesn't provide the user identity manually.
Instead:
Bearer Token
↓
Token Verification
↓
Authenticated User
↓
Memory belongs to that user
This prevents the client from simply claiming that a memory belongs to some other user.
6. The Backend Processes the Memory
Once authentication succeeds, the backend receives the text.
For example:
"I prefer PostgreSQL for my backend projects."
From here, the rest of the pipeline happens on the backend.
This is where the database and embedding infrastructure come into play.
The CLI doesn't need to know how the backend implements this processing.
Conceptually, the pipeline is:
Text
↓
Chunking
↓
Batch Creation
↓
Embedding Generation
↓
Vector
↓
PostgreSQL + pgvector
These are backend responsibilities.
7. Chunking
The backend can split larger pieces of text into smaller chunks.
Conceptually:
Large Text
│
├── Chunk 1
├── Chunk 2
├── Chunk 3
└── ...
For a tiny memory such as:
"I prefer PostgreSQL."
there may not be much to split.
But chunking becomes useful as the amount of stored information grows.
8. Creating Batches
After chunking, the backend can group chunks into batches for embedding.
For example:
Chunks
│
├── Batch 1
│ ├── Chunk 1
│ ├── Chunk 2
│ └── Chunk 3
│
└── Batch 2
├── Chunk 4
├── Chunk 5
└── Chunk 6
These batches are then passed to the embedding stage.
9. Turning Text into Embeddings
The text is converted into a numerical vector using an embedding model.
For example:
"I prefer PostgreSQL for my backend projects."
│
▼
Embedding Model
│
▼
[0.12, -0.38, 0.74, ...]
The resulting vector represents the semantic information contained in the text.
This is what makes vector-based memory possible.
Instead of treating a memory as only a string of characters, the backend also stores a mathematical representation of its meaning.
10. Storing the Memory in PostgreSQL
The generated vector is stored using pgvector alongside the memory data.
The database layer uses:
PostgreSQL
+
pgvector
+
Prisma
A simplified representation might look like:
Memory
─────────────────────────────
user_id
text
embedding
created_at
...
The exact schema can evolve, but the important part is that the memory is associated with the authenticated user and has a vector representation that can later be used for semantic retrieval.
11. The Backend Sends a Response
Once the memory has been successfully processed and stored, the API returns a success response.
The actual response from the controller is:
{
"success": true,
"message": "Data saved in memory"
}
The CLI can then use that result to tell the user that the memory was saved successfully.
So the complete flow is:
User
│
│ "Remember this..."
▼
REXA model
│
│ calls save_memory
▼
REXA CLI
│
│ validate + trim text
│
│ POST /api/cli/memory
│ Authorization: Bearer <token>
│ { "text": "..." }
▼
REXA Backend
│
│ verify token
│ identify user
│
│ process memory
│ ├── chunk
│ ├── batch
│ ├── embed
│ └── store vector
▼
PostgreSQL + pgvector
│
│ success
▼
REXA Backend
│
│ { "success": true,
│ "message": "Data saved in memory" }
▼
REXA CLI
Why I Designed It This Way
The main architectural decision was to keep REXA itself separate from the memory infrastructure.
The CLI is responsible for interacting with the agent and invoking save_memory.
The backend handles authentication and memory processing.
PostgreSQL and pgvector provide the persistent storage and vector representation.
So the responsibilities are roughly separated like this:
REXA CLI
→ Agent interaction + save_memory
Backend
→ Authentication + memory processing
PostgreSQL + pgvector
→ Persistent memory storage
This also means the CLI doesn't need database credentials or direct database access.
It only needs an authenticated API connection.
What REXA Has Today vs. What's Next
At the moment, the implemented functionality is the save side of memory.
User
↓
save_memory
↓
Authenticated API
↓
Embedding
↓
Vector storage
The retrieval side is not implemented in the CLI yet.
The next step is to build the other half:
Current:
SAVE
User
↓
save_memory
↓
Backend
↓
Vector Database
Future:
RECALL
Current Task
↓
Memory Search
↓
Similarity / Relevance
↓
Relevant Memories
↓
REXA
That is where things become much more interesting.
The goal isn't simply to give REXA a database full of memories.
The real goal is to build a system where REXA can eventually find the right memory when it is actually useful.
And that's the part I'm building next.
Top comments (0)