Joung Park

Posted on Oct 7

Vector database

#pgvector #rag #vectordatabase #ai

Second-Memory stores things a user writes over time.

If a user later asks:

“What did I write about performance problems?”

A normal keyword search isn't always enough.

The memory might say:

“The database becomes slow when thousands of users query it at the same time.”

There may be no exact keyword match between the question and the memory.

This is where semantic search becomes useful.

1. From text to vectors

I can convert a piece of text into an embedding — a vector representation of its meaning.

For example:

"The database becomes slow when thousands of users query it at the same time."
         ↓
[Embedding model]
         ↓
[0.021, -0.183, ...]

The actual vector contains many dimensions, so it isn't meaningful to look at the individual numbers.

What matters is the relationship between vectors.

Texts with similar meanings tend to have vectors that are closer together.

So when a user asks a question, I can:

Question
Embedding
Vector similarity search
Relevant memories

This gives Second-Memory a way to retrieve memories based on meaning, rather than just matching words.

2. Why pgvector?

Once I decided to use embeddings, I needed somewhere to store and search them.

I considered:

	pgvector	Pinecone
Vector search	Yes	Yes
Relational data	Yes	No
Existing PostgreSQL	Yes	No
Separate infrastructure	No	Yes
Operational complexity	Lower	Higher
Good fit for V1	Yes	Yes

I chose pgvector.

The main reason wasn't that pgvector is necessarily better than Pinecone.

It was that Second-Memory already had a natural place for the vectors:

the Memory Service's database.

With pgvector, I could keep the memory and its embedding together.

erDiagram
    "Memory Service" ||--|| "PostgreSQL + pgvector" : utilizes
    "PostgreSQL + pgvector" ||--|{ Memory : contains

    Memory {
        uuid user_id
        text content
        vector embedding
    }

That kept the architecture simple.

3. Why not introduce a vector database?

A dedicated vector database could make sense at larger scale.

But introducing one also creates another system to operate and another boundary to manage.

For V1, I didn't see enough benefit to justify that complexity.

The Memory Service could own:

memory data
embeddings
vector search

all within the same data boundary.

That also reinforced one of the architectural principles from the previous post:

The service that owns the data should own access to it.

The Ask Service doesn't need to know whether semantic search is implemented with pgvector, Pinecone, or something else.

It simply asks the Memory Service for relevant memories.

4. What this gives me

With pgvector, the basic retrieval flow becomes:

User question
      ↓
Generate query embedding
      ↓
Memory Service
      ↓
pgvector similarity search
      ↓
Relevant memories
      ↓
Ask Service
      ↓
     LLM

This is the first important piece of the AI architecture.

The LLM doesn't need to know everything the user has ever written.

Instead, the system retrieves the memories that are most relevant to the current question and uses those as context.

The decision

For Second-Memory V1, I chose:

PostgreSQL + pgvector

because it gave me semantic search without introducing another database.

It was a pragmatic choice:

vector search
relational data
one data boundary
less infrastructure
simpler development

DEV Community