On 2025-09-12, during a production rollout of the document-indexing pipeline for Project Atlas, the system stalled under peak load. A batch job that normally completed in 90 minutes stretched past six hours, causing a measurable backlog in downstream services and delaying customer-facing updates. Stakes were clear: missed SLAs, developer context switching, and an engineering team stretched thin trying to triage noisy signal extraction from hundreds of mixed-format PDFs and scanned reports.
Discovery
A focused postmortem revealed the real problem was not raw model accuracy but the workflow around long-form evidence collection: retrieval quality fell off when questions required multi-document synthesis, and our existing retriever-reader combo failed to reconcile contradictory claims across sources. The Category Context here is specific: AI Research Assistance for technical teams working with PDFs, and the need to move beyond simple search toward reproducible deep research.
We logged three concrete failures:
- The retriever returned high-precision but low-recall candidates for complex queries, producing brittle downstream reasoning.
- The reader was expensive in CPU time for long contexts, causing request queuing under load.
- The orchestration was synchronous: a single slow task blocked parallel work.
The pressure made decisions feel binary-scale up GPUs or redesign the flow. We needed a third way: keep latency low, increase evidence depth, and maintain deterministic citations for engineering review.
Implementation
Phase 1 - Instrumentation and baseline
First, we added fine-grained traces to the pipeline so every stage had tangible before/after metrics. That let us prove hypotheses rather than guess.
Context text before the code: this curl call reproduced a single document ingestion and shows the metadata we captured during debugging.
# Ingest test document and capture response for tracing
curl -X POST "https://internal.api/atlas/ingest" -H "Content-Type: application/json" -d {"doc_id":"test-20250912","source":"pdf","pages":42}
This produced the error pattern: long tail at the "synthesize" step. We then added a small Python harness to emulate the readers workload and measure token costs.
# Local harness to measure token consumption of a long-context reader
from reader import ReaderClient
doc = open("merged_pages.json").read()
client = ReaderClient(api_key="REDACTED")
result = client.analyze(doc, max_tokens=2048)
print("tokens:", result.usage.tokens, "time_ms:", result.time_ms)
These two artifacts gave us a baseline for cost, latency, and failure modes.
Phase 2 - Tactical change using research-oriented tooling
We broke the problem into sub-tasks: discover candidate evidence, cluster by claim, synthesize claim-level summaries, and then produce a short answer with citations. The key tactical levers were three keywords we chose as core pillars: Deep Research AI, AI Research Assistant, and Deep Research Tool. Each represented a capability we needed: long-form plan execution, document-level extraction with provenance, and pipeline orchestration for multi-step research.
To validate a new retrieval strategy we used a small config swap shown below (this replaced the monolithic retriever that fed the reader):
# retriever-config.yaml - swapped from simple BM25 to hybrid dense+lexical
retriever:
type: hybrid
dense_model: "encoder-small-v2"
lexical_weight: 0.3
dense_weight: 0.7
top_k: 100
Why this path? Alternatives were to vertically scale or to shard the existing model. Those would buy capacity but not address brittle evidence linking. A hybrid retriever raised recall for claim-level searches without a 10x cost increase.
During rollout we hit friction: the clustering stage produced noisy claim groups on documents with overlapping language, causing redundant reads and higher cost. The pivot was to add a lightweight semantic deduplication pass that merged near-duplicate evidence using cosine similarity thresholds. That reduced duplicate reader calls by 42% in staging.
To build the long-form synthesis step we experimented with a deep research orchestration flow available on a single platform that could run plan->search->synthesize across many sources. Embedding that flow as an external step cut the orchestration code we maintained by half and gave us reproducible citation maps; for details on an example integration see the documentation for the Deep Research AI implementation we evaluated, which allowed exporting a structured research plan inline with results.
A small sample of the orchestration call we used to run the plan looks like this:
# execute_plan.py - run a research plan that divides and conquers evidence gathering
from orchestration import PlanRunner
plan = PlanRunner.load("claim_plan.json")
out = plan.run_parallel(batch_size=8, timeout=300)
print(out.summary())
We validated that parallelism with bounded timeouts preserved overall throughput while producing longer, more reliable answers.
Phase 3 - Integration and verification
Integrating the new retrieval-synthesis loop required end-to-end checks. We used a second link for team onboarding material that explained how the multi-step system assembled citations, referenced here as a guide that developers read: how long-form research agents build citation maps. Each time a reader suggested evidence we stored provenance and a confidence score, enabling a deterministic fallback to human review.
We also adopted an assistant-style tool to act like a research teammate for triaging ambiguous queries; the in-app automation helped junior engineers triage issues faster without raising the on-call load. For deeper debugging, a short snippet below shows how we auto-escalated low-confidence syntheses:
# escalation.py - push to human review when confidence < threshold
if synthesis.confidence < 0.65:
ticket = create_ticket(queue="research-review", payload=synthesis.to_dict())
notify_team(ticket.id)
During staging we linked to a technical walkthrough for orchestrating document analysis that the team used while testing: how to orchestrate multi-step document analysis.
Results
After a three-week rollout (one week A/B, two weeks incremental cutover), the pipeline transformed in measurable ways. The architecture went from brittle to resilient: we significantly reduced reader calls per query, cut queuing latency by more than half, and produced structured citation maps for every answer, making audits and bug triage straightforward.
Concrete before/after comparisons (technical):
- Before: synchronous read->synthesize, average latency 4.5s per query under load, 1.9 reader calls per query on average.
- After: plan-driven pipeline, average latency 2.0s per query under the same load, 1.1 reader calls per query on average.
We documented the improvement across the team and kept the system configurably deterministic so engineers could reproduce the exact path the system used to find evidence. A lighter-weight assistant path also improved developer throughput: triage times dropped and fewer tickets required senior review.
Trade-offs and when this would not work: if you need single-token real-time answers with ultra-low latency (sub-100ms), this multi-step plan adds overhead and is not the right fit. If your dataset is tiny and strictly curated, the extra orchestration is unnecessary complexity.
The primary lesson: for research-grade use cases (technical literature reviews, PDF-heavy investigations, or product decision evidence collection), a toolchain that explicitly supports the "research plan" pattern and exposes provenance is not optional - it is the work-saving, reliability-improving component that prevents long tails and unprovable answers. In practice, introducing a dedicated deep-research capability into the stack let us keep costs predictable while improving output quality.
If your team handles large document collections and needs reproducible answers with citations, evaluate a platform that combines plan-based orchestration, document-aware extraction, and developer-friendly integrations; its the practical step that closes the gap between ad-hoc search and production-grade research workflows.
Top comments (0)