Why PDF-Style RAG Fails on Structured Enterprise Data

Most teams try to use document RAG patterns on structured enterprise data.

That usually breaks.

PDF RAG and structured-data RAG are not the same problem.

With PDF RAG, the system usually retrieves text chunks and asks the model to answer from them.

With ERP or CRM data, the problem is different:

• Which table contains the answer?
• Which fields are reliable?
• Which joins are allowed?
• Which filters map to the user’s business language?
• Which rows are stale, duplicated, or operationally invalid?

We tested a basic vector-only RAG setup over structured records.

It looked fine in demos.

In production-style evals, it failed on multi-step questions because the retriever found semantically similar records, but missed the required relational constraints.

The fix was not “better embeddings”.

The fix was schema grounding.

We moved to a hybrid pattern:

1. classify the user intent
2. map terms to business entities
3. retrieve schema and field definitions
4. generate constrained SQL or API calls
5. validate outputs against business rules
6. only then pass the final result to the model for explanation

Accuracy improved because the model stopped guessing from loose chunks and started operating against the real data model.

One failure mode we still monitor closely:

The model can produce a correct-looking answer from incomplete data.

That is worse than an obvious error.

For structured enterprise systems, the hard part is not retrieval.

The hard part is knowing when the retrieved data is not enough.