TL;DR
most teams rush to add synthesis (a fancy generation layer) hoping to fix poor answers. but if your intake → embedding → retrieval steps aren’t stable, synthesis only polishes garbage. this is the bootstrap ordering mistake.
🚨 What developers usually do wrong
- normalize nothing, embed everything → embeddings scatter, retrieval misfires.
- top-k hops every run, yet synthesis still writes confident essays.
- citations vanish mid-answer because the input text was malformed.
- users report: “the model is fluent, but it cites things that don’t exist.”
Adding synthesis too early creates a dangerous illusion: the output looks polished, but the foundation is unstable.
🧭 The correct pipeline order
- Intake – clean, normalize, validate casing, diacritics, unicode.
- Embedding – verify metric matches store; ensure vector dimensions align.
- Retrieval – test consistency across paraphrases; coverage ≥ 0.7 before moving on.
- Synthesis – only after the first three are stable.
Think of it like building a house: you don’t start with the roof.
🔍 60-second self-diagnosis
- run your pipeline without synthesis (stop at retrieval).
- check if retrieval-only answers are more grounded than full pipeline.
- feed malformed input (wrong casing, schema errors). if synthesis tries to “smooth it over,” you’ve confirmed the ordering bug.
🛠 Minimal fix
- enforce pipeline logs that explicitly show:
intake → embedding → retrieval → synthesis. - block synthesis if intake validation fails.
- add an acceptance gate: retrieval coverage must hit 70% before synthesis runs.
🧩 Hard fixes
- rebuild indexes with normalized intake.
- add ingestion validators (reject malformed or duplicate entries).
- use multi-retriever voting to cut blind spots before synthesis.
🛡 Guardrails with WFGY
The WFGY framework calls this ProblemMap No.14. Guardrails include:
- ingestion checks (normalize before embedding),
- vectorstore metric validator,
- retrieval playbook (acceptance thresholds),
- ordering log (audit trail of pipeline sequence).
📌 Why this matters
This mistake is everywhere in RAG pipelines, vector database apps, and production LLM deployments. Teams polish synthesis instead of fixing intake, which only makes hallucinations harder to detect.
The fix isn’t glamorous — but if you care about stability, you must get the order right.
✅ Acceptance checks
- pipeline trace shows correct order every run
- retrieval coverage ≥ 0.7 before synthesis
- citations map to corpus spans, not filler
- no synthesis allowed if intake validation fails
Bottom line:
if you jump straight to synthesis, you’re building castles on sand. fix intake, embeddings, and retrieval first. synthesis comes last.
That’s Bootstrap Ordering Mistake (ProblemMap No.14).
Top comments (0)