RAG Is Not a Vector Database Problem. It’s a Data Problem.

I built a RAG application, and the more I worked on it, the clearer one thing became: the vector database was not the difficult part.

It's easy to focus on embeddings, similarity search, pgvector, and which LLM to use. But when a RAG application gives a wrong answer, the cause is usually much earlier in the pipeline. The document may have been parsed incorrectly. A chunk may have lost important context. Metadata may be missing. Or we may simply be sending the wrong pieces of information to the model.

That changed how I think about RAG. I now see it less as an "AI problem" and more as a data pipeline problem, and that view makes it much easier to reason about and debug.

Quick definitions: RAG (Retrieval-Augmented Generation) means searching your own documents for relevant passages and giving them to an LLM along with the user's question, so it answers from your data. An embedding is a list of numbers that represents the meaning of a piece of text, so texts with similar meanings get similar numbers. A chunk is a small piece of a document that gets its own embedding.

The RAG pipeline is longer than it looks

RAG is usually explained as a short chain: question, embedding, vector database, LLM, answer. That makes it sound almost trivial. In a real application, the pipeline looks more like this:

flowchart TD
    A[Documents] --> B[Parsing]
    B --> C[Cleaning]
    C --> D[Chunking]
    D --> E[Metadata]
    E --> F[Embedding]
    F --> G[(Vector store)]
    G --> H[Retrieval]
    H --> I[Filtering / Reranking]
    I --> J[Context construction]
    J --> K[LLM]
    K --> L[Answer]

Every arrow is a place where information can be lost. And once information is lost, no vector database can bring it back.

The vector database gets blamed for problems it didn't create

Imagine you upload a 200-page company policy and ask: "What is the maximum amount an employee can claim for business travel?" The app answers: "Employees can claim up to $500."

The actual policy says: "Employees can claim up to $500 per trip, excluding accommodation."

Nothing crashed. The vector search worked, and the LLM worked. The answer even looks plausible. But it's incomplete, and the cause could be as simple as the chunker splitting that sentence in two:

Chunk 12: "...Employees can claim up to $500 per trip,"
Chunk 13: "excluding accommodation. Meal expenses are..."

If chunk 12 is retrieved and chunk 13 isn't, the model never sees the full rule, and it can't answer correctly with information it never received. That leads to the mental model I now use:

RAG quality is limited by the quality of the data that reaches retrieval.

1. Parsing is already a RAG problem

Before you create any embeddings, you have to extract text from the document. With real-world files, and especially PDFs, that's harder than it sounds. A table that looks like this on screen:

Benefit	Limit
Travel	$500
Accommodation	$1,000
Meals	$100

can come out of a PDF text extractor like this:

Benefit Limit Travel Accommodation Meals $500 $1,000 $100

A human instantly sees which amount belongs to which benefit. The extracted text no longer says that. You can embed it with an excellent model, store it in Postgres, and send it to the newest LLM, and you'll still get unreliable answers, because the damage happened before the vector database was involved.

The practical lesson: look at your extracted text before blaming anything later in the pipeline. For table-heavy documents, it's worth converting tables into a form that keeps each row together, such as Travel: $500.

2. Chunking is not just "split every 500 characters"

A document isn't just a stream of characters. It has headings, paragraphs, lists, tables, and sentences that refer back to earlier sentences. Consider this section:

Refund Policy
Customers may request a refund within 30 days.
For enterprise customers, this period is extended to 60 days.

A naive chunker might put each sentence in a separate chunk. Now a user asks: "How long do enterprise customers have to request a refund?" The second chunk is the best match, but on its own it says "this period" without saying which period, and it doesn't even mention refunds.

A common fix is chunk overlap, where neighboring chunks share a few sentences so text cut at a boundary also appears whole in the next chunk. Overlap helps, but it doesn't guarantee that a chunk carries its full meaning. Two further approaches often help more:

• Split on the document's structure, meaning headings and paragraphs, rather than a fixed character count.
• Add the section heading to each chunk's text before embedding it, so "this period" arrives together with "Refund Policy".

3. More chunks don't automatically mean better retrieval

When retrieval is weak, it's tempting to just fetch more chunks. In RAG this setting is often called top K, meaning "return the K most similar chunks." Raising K sometimes helps, but it can also hurt.

If you retrieve 20 chunks and only 4 are relevant, the model now has to reason through old policy versions, duplicates, and loosely related sections. That costs more tokens, adds latency, and increases the chance of conflicting or outdated information in the answer.

So the goal of retrieval isn't to fetch as much as possible. It's to fetch the smallest set of information that is sufficient to answer the question. A reranker, which is a second model that re-scores the retrieved chunks by how well they actually answer the question, can help you retrieve broadly and then keep only the best few.

4. Metadata can be as important as embeddings

Each chunk has an embedding, but you usually know much more about it: its department, document type, year, region, and version. Throwing that information away is a mistake.

If a user asks about "the 2026 travel policy," you don't want a 2022 policy ranking highly just because its wording is similar. Metadata lets you combine semantic similarity with ordinary filters and business rules. With pgvector, both fit in one SQL query:

SELECT c.content, d.title, c.chunk_index
FROM document_chunks c
JOIN documents d ON d.id = c.document_id
WHERE d.tenant_id = $1              -- only this customer's data
  AND d.document_type = 'Policy'
  AND d.is_current_version = true   -- skip outdated versions
ORDER BY c.embedding <=> $2         -- <=> is cosine distance: smaller = more similar
LIMIT 5;

One pgvector detail is worth knowing here. With an HNSW index (an index that finds approximately nearest vectors quickly), Postgres finds nearby vectors first and applies the WHERE filter afterwards, so a strict filter can return fewer rows than your LIMIT. pgvector 0.8 and later can keep scanning until enough rows match:

SET hnsw.iterative_scan = relaxed_order;

This is the point where RAG starts to feel very familiar to backend engineers. It's not just AI anymore. It's data modeling.

5. Your schema matters

A chunk is not the document. Its embedding is only one property of it: an indexable representation of part of the document. A schema along these lines keeps the rest of the information that retrieval depends on:

CREATE TABLE documents (
    id                  UUID PRIMARY KEY,
    tenant_id           UUID NOT NULL,
    title               TEXT NOT NULL,
    document_type       TEXT NOT NULL,
    version             INT  NOT NULL,
    is_current_version  BOOLEAN NOT NULL,
    created_at          TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE document_chunks (
    id           BIGSERIAL PRIMARY KEY,
    document_id  UUID NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
    chunk_index  INT  NOT NULL,
    section      TEXT,
    content      TEXT NOT NULL,
    embedding    vector(1024) NOT NULL
);

The original content, the relationships, the version, and the permissions all still matter, and they belong in your schema just as much as the vector does.

6. Access control matters even more in enterprise RAG

Imagine one system holding HR, finance, customer, and engineering documents. A user asks: "What is our salary adjustment policy?" Semantic search may find exactly the right document, but should this user be allowed to see it? That's a separate question, and similarity search doesn't answer it.

Retrieved chunks are application data, and they need the same protection as any other data. Authentication, authorization, tenancy (keeping each customer's data separate), auditing, and data retention all still apply. The safest place to enforce them is inside the retrieval query itself, as in the tenant_id filter above, so unauthorized content never reaches the model at all. AI doesn't remove these backend responsibilities. It makes them more important.

7. Measure retrieval quality instead of guessing

The easiest mistake is testing RAG by hand. You ask one question, the answer looks good, and you conclude that RAG works. That tells you very little.

A better approach is a small evaluation set: a list of questions where you already know which chunk should be found. Even 20 to 30 cases are enough to start. Then you can measure the hit rate, which is the share of questions where the expected chunk appears in the top K results:

public record EvalCase(string Question, long ExpectedChunkId);

public class RetrievalEvaluator(IRetriever retriever)
{
    public async Task<double> HitRateAsync(IReadOnlyList<EvalCase> cases, int topK = 5)
    {
        var hits = 0;

        foreach (var evalCase in cases)
        {
            var results = await retriever.SearchAsync(evalCase.Question, topK);

            if (results.Any(r => r.ChunkId == evalCase.ExpectedChunkId))
                hits++;
        }

        return (double)hits / cases.Count;
    }
}

Now, when you change the chunk size or the embedding model, you get a number instead of a feeling. It also changes the debugging conversation from "the LLM gave a bad answer" to "did retrieval return the right evidence?", which is a much more useful engineering question.

8. Debug RAG from the bottom up

When an answer is wrong, don't start by swapping the LLM. Walk backwards through the pipeline instead:

1. The source document: was the information actually there?
2. Extraction: was the text extracted correctly?
3. Chunking: did the relevant information stay together?
4. Metadata: were the document, version, and tenant attached correctly?
5. Embedding: was the chunk embedded at all?
6. Retrieval: was the right chunk returned?
7. Context construction: did anything get cut off, removed, or reordered?
8. Prompt: did we clearly tell the model how to use the context?
9. The LLM: only now is it time to suspect the model.

This works much better when you can see what the model actually received, so log it on every request:

logger.LogInformation(
    "RAG query {QueryId} retrieved chunks {ChunkIds} with distances {Distances}",
    queryId,
    results.Select(r => r.ChunkId),
    results.Select(r => r.Distance));

Without this, it's easy to keep changing the embedding model, then the chunk size, then the LLM, then the prompt, without ever knowing which change fixed the problem.

The backend engineer's view of RAG

RAG is usually presented as an AI architecture, but much of it is really a data architecture. All the familiar backend concerns are still here: ingestion, validation, data modeling, indexing, caching, authorization, versioning, observability, performance, cost, and testing. What's new is that they now sit next to embeddings, semantic retrieval, context construction, and LLM calls. The interesting engineering happens where those two worlds meet.

So, should you use a vector database?

Yes. Whether it's a dedicated product or pgvector inside Postgres, you need a way to search by meaning. But I wouldn't start a RAG project by asking "Which vector database should I use?" I'd start by asking:

"What does my data look like, and what does a correct retrieval result look like?"

From there, work backwards: which questions the system must answer, what evidence each answer needs, where that evidence lives, how documents should be split, which metadata is required, and only then which storage technology fits. The technology should follow the retrieval requirements, not the other way around.

One final thought

In a RAG system, the LLM is often the easiest part to replace. You can switch models, vector databases, embedding providers, and frameworks. But if the data pipeline is poor, every combination will keep producing poor results.

A powerful model can't use information that was never indexed correctly. A vector database can't recover context that chunking destroyed. And an embedding model can't fix a badly extracted document.

So the next time your RAG application gives a strange answer, don't start with "Which LLM should we use?" Start with:

"What data did the model actually see?"

That question will usually take you much closer to the real problem, because RAG isn't just about giving an LLM access to your data. It's about building a reliable pipeline that gets the right data to the model at the right time. And that's a backend engineering problem.

Have you debugged a RAG system that gave confidently wrong answers? I'd love to hear where the problem turned out to be. Share it in the comments. 👇