DEV Community

Umesh Adabala
Umesh Adabala

Posted on

Why We Built PaperTrail: Moving from AI Paper Summaries to Continuous Scientific Intelligence

#webdev #ai #react #opensource

Every few months, a new "AI research tool" hits the front page of Hacker News or Product Hunt. Almost without exception, they all do the same thing:

  1. Ingest a PDF.
  2. Chunk it into embeddings.
  3. Feed the chunks into a vector database.
  4. Put a conversational chatbot in front of it that summarizes what the authors said.

Here's the fundamental problem: Science is not a collection of static text summaries. Science is an evolving, adversarial debate.

A paper published in 2023 demonstrating a 4x inference speedup for 4-bit quantized language models on Apple Silicon is not an absolute truth. It is a finding contingent on specific hardware, quantization algorithms, precision levels, and thermal conditions. When a 2024 paper reports severe degradation under continuous workloads on Qualcomm Snapdragon chips, they aren't necessarily contradicting each other—they're operating in completely different experimental contexts.

And when a seminal paper gets retracted or contradicted six months later, your vector database doesn't know. It happily continues serving obsolete claims as ground truth.

We built PaperTrail to fundamentally rethink how software interacts with scientific literature.

Instead of another paper summarizer, PaperTrail is an autonomous, continuously evolving scientific intelligence engine.

In this article, I want to walk through the technical architecture, design philosophy, and lessons learned building PaperTrail with Next.js 14, TypeScript, local Ollama models, and a local-first SQLite/Supabase hybrid storage engine.

The Core Proposition

PaperTrail is built around a single thesis:

"Discover what research has established, what it challenges, what remains unexplored, and how the scientific evidence changes over time."

To deliver on this, we established ten non-negotiable architectural capabilities:

  1. Grounded Scientific Claim Extraction: Never extract an assertion without an exact, verbatim passage and paper citation.
  2. Context-Aware Disagreement Investigation: Never label two differing findings as a contradiction without examining differences in hardware, precision, quantization bitwidth, context length, or thermal states.
  3. Candidate Research-Gap Discovery: Automatically identify blind spots and uninvestigated parameter intersections across literature.
  4. Adversarial 6-Stage Gap Falsification: Never present a research gap as valid until an automated red-team agent attempts to aggressively falsify it against counter-evidence.
  5. Practical Experiment Feasibility & Protocol Planning: Convert validated gaps into runnable, resource-evaluated empirical protocols with compute hour calculations and step-by-step checklists.
  6. Continuous Research Tracking & Evidence Evolution: Monitor topics and incoming preprints to compute how evidence shifts over time.
  7. Preprint-to-Publication Lineage Mapping: Trace preprints from arXiv to peer-reviewed venues, isolating shifts in methodology and benchmarks.
  8. Retraction & Correction Intelligence: When papers are retracted or corrected, flag dependent claims and recalculate conclusions without silently erasing historical evidence.
  9. Accessible Dataset Discovery: Find benchmark datasets on Hugging Face, Zenodo, and Figshare with verified licenses.
  10. Cross-Domain Hypothesis Transfer: Formulate principled transfer hypotheses connecting primary research with adjacent fields (e.g., neuroscience synaptic pruning $\leftrightarrow$ LLM weight quantization).

1. Grounded Claim Extraction & Schema Validation

One of our strictest engineering rules is: No hallucinated consensus.

Every claim extracted by PaperTrail must adhere to a strict Zod schema. If the model cannot provide the exact supporting passage from the abstract or text, the extraction fails validation and falls back to a deterministic semantic parser.

Here is a snippet of our claim validation contract:

export const ClaimExtractionSchema = z.object({
  claims: z.array(
    z.object({
      claim_text: z.string().min(10),
      claim_type: z.enum([
        'FINDING',
        'HYPOTHESIS',
        'METHODOLOGY',
        'LIMITATION',
        'BENCHMARK_RESULT',
        'THEORETICAL_BOUND',
      ]),
      section: z.enum(['Abstract', 'Introduction', 'Evaluation', 'Conclusion']),
      supporting_passage: z.string().min(15),
      page_number: z.number().int().positive().optional(),
      extraction_confidence: z.number().min(0).max(1),
    })
  ),
  experimental_context: z.object({
    model: z.string().default('Unspecified'),
    hardware: z.string().default('Unspecified'),
    dataset: z.string().default('Unspecified'),
    quantization: z.string().default('FP16'),
    precision: z.string().default('Unspecified'),
    metrics: z.record(z.string()).default({}),
    variables: z.record(z.string()).default({}),
  }),
  limitations: z.array(z.string()).default([]),
});
Enter fullscreen mode Exit fullscreen mode

By extracting both the claim and its experimental condition matrix (hardware, precision, model scale, metrics), we create the foundation for intelligent, context-aware comparison.

2. Context-Aware Disagreement Investigation: Beyond Binary "Contradictions"

Traditional NLP pipelines treat contradiction detection as an NLI (Natural Language Inference) task: Premise vs. Hypothesis $\rightarrow$ Entailment, Neutral, or Contradiction.

In real science, this is catastrophically naive. Consider two papers:

  • Paper A (Lin et al., 2023): "AWQ achieves 3.2x latency speedup with negligible perplexity degradation on 7B LLMs."
  • Paper B (Smith et al., 2024): "Sustained 4-bit execution throttles after 120 seconds, resulting in a 42% throughput drop on fanless mobile SoCs."

A naive AI system flags these as a direct contradiction: Speedup vs. Throughput Drop.

PaperTrail executes a Context Comparison Matrix:

export interface ContextComparison {
  similarities: string[];
  differences: {
    variable: string;
    source_val: string;
    target_val: string;
    impact_level: 'LOW' | 'MEDIUM' | 'HIGH';
  }[];
  possible_explanations: string[];
  alternative_explanations: string[];
  remaining_uncertainty: string;
  is_genuine_contradiction: boolean;
}
Enter fullscreen mode Exit fullscreen mode

When evaluated, PaperTrail identifies:

  • Independent Variable Difference: Paper A evaluated burst inference under active server cooling; Paper B evaluated sustained execution under passive ambient mobile cooling.
  • Verdict: APPARENT_DIVERGENCE_EXPLAINED_BY_CONTEXT (Confidence: 0.94).
  • Resulting Research Opportunity: How do thermal throttling thresholds differ between INT4 weight-only quantization and INT4 activation quantization during multi-turn generation?

3. The 6-Stage Adversarial Gap Falsification Workflow

Most researchers know the dread of reading 30 papers, designing a novel experiment, and discovering two weeks later that a lab in Zurich published that exact experiment six months ago.

In PaperTrail, candidate research gaps are never taken at face value. They are subjected to a rigorous 6-Stage Adversarial Red-Team Audit:

[Candidate Gap Discovery]
          │
          ▼
┌──────────────────────────────────────────────┐
│ Stage 1: Hypothesis Decomposition             │
│   Extract testable variables & parameters    │
├──────────────────────────────────────────────┤
│ Stage 2: Targeted Counter-Evidence Search    │
│   Generate adversarial retrieval queries     │
├──────────────────────────────────────────────┤
│ Stage 3: Multi-Index Retrieval               │
│   Query OpenAlex, Crossref, arXiv, TinyFish  │
├──────────────────────────────────────────────┤
│ Stage 4: Semantic Relevance & Overlap Audit  │
│   Compare baseline methodology & datasets    │
├──────────────────────────────────────────────┤
│ Stage 5: Adversarial Refutation Scoring      │
│   Compute falsification confidence (0.0-1.0) │
├──────────────────────────────────────────────┤
│ Stage 6: Status Assignment                   │
│   SURVIVED (Validated) | FALSIFIED | PARTIAL │
└──────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Only when a candidate gap survives this adversarial gauntlet is it promoted to an actionable research gap. From there, PaperTrail automatically drafts an empirical experiment protocol complete with hardware requirements, dataset references, compute hours, and verification checklists.

4. Local-First Research Portability: SQLite + Supabase

We believe your research belongs to you—not locked inside our cloud database.

PaperTrail implements a dual-repository architecture:

                  ┌───────────────────────────────┐
                  │      ResearchRepository       │
                  │          (Interface)          │
                  └───────────────┬───────────────┘
                                  │
                 ┌────────────────┴────────────────┐
                 ▼                                 ▼
   ┌───────────────────────────┐     ┌───────────────────────────┐
   │ SQLiteResearchRepository  │     │ SupabaseResearchRepository│
   │ (Node native DatabaseSync)│     │     (PostgreSQL Cloud)    │
   └───────────────────────────┘     └───────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Using Node.js 22+'s native DatabaseSync (node:sqlite), PaperTrail operates entirely offline if desired, writing directly to .papertrail-data/papertrail.db with WAL mode enabled.

Complete Research Archive Export

Every research workspace can be exported with one click into a structured, vendor-neutral research bundle:

papertrail-research/
├── manifest.json         # Cryptographic checksums, schema v1.0, timestamp
├── README.md             # Complete human-readable investigation brief
├── papers/               # Verified metadata, OpenAlex/arXiv DOIs, abstracts
├── claims/               # Grounded assertions with verbatim supporting quotes
├── evidence/             # Disagreement matrices and apparent vs genuine verdicts
├── graph/                # Interactive topology nodes and edges (JSON)
├── research-gaps/        # 6-stage falsification audits and counter-evidence
├── experiments/          # Resource-evaluated execution protocols
├── datasets/             # Hugging Face & Zenodo references with license terms
├── timeline/             # Evidence evolution and tracking history
└── provenance/           # Query origins and retrieval timestamps
Enter fullscreen mode Exit fullscreen mode

You can export this as a ZIP archive, BibTeX, JSON, or formatted Markdown, and import it into any other PaperTrail instance with zero data loss.

5. Designing for Academics: The Academic Journal Aesthetic

We made a conscious product design decision: PaperTrail should not look like another dark-mode SaaS dashboard with glowing neon gradients.

Science is deliberate, rigorous, and calm. We crafted a dedicated design language inspired by physical academic journals and high-end typography:

  • Warm Paper Surface: #F7F6F2 (background), #FDFCF9 (cards), #FAF8F3 (alt panels)
  • Deep Editorial Ink: #252622 (primary text), #777873 (muted metadata)
  • Terracotta Accent: #B96645 (reserved strictly for active claims, falsification audits, and disagreements)
  • Sage Data Accent: #858B70 (used for verified benchmark metrics, datasets, and lineages)
  • Typography:
    • Instrument Serif for research questions, paper titles, and editorial headers.
    • DM Sans for high-density tabular data and claim cards.
    • JetBrains Mono for DOIs, citations, arXiv IDs, and hardware matrices. 
:root {
  --paper-bg: #F7F6F2;
  --paper-surface: #FDFCF9;
  --paper-surface-alt: #FAF8F3;
  --paper-ink: #252622;
  --paper-muted: #777873;
  --paper-border: #E3E1D9;
  --paper-accent: #B96645;
  --paper-data: #858B70;
}
Enter fullscreen mode Exit fullscreen mode

The interface feels like reading a physical copy of Nature or Science paired with an interactive research workbench.

6. Real-World Engineering Challenge: Taming Heavy LLMs

During our testing with local and remote Ollama models (specifically gpt-oss:120b-cloud), we hit an interesting bottleneck.

When analyzing 8 papers simultaneously, each paper required claim extraction, experimental context parsing, pairwise contradiction checks, and gap discovery. Sending 15+ concurrent requests to a 120-billion parameter model over a cloud proxy resulted in request queueing that took 4+ minutes.

We solved this with a three-tiered pipeline architecture:

  1. Strict 12s Timeout: Ollama requests time out gracefully after 12 seconds instead of hanging the connection.
  2. Landmark Prioritization: The top 3 landmark papers receive deep LLM extraction. Supporting literature receives instant, deterministic parsing (extracting findings, methodologies, and limitations in 0.1ms).
  3. Capped Contradiction Audits: Pairwise contradiction checking prioritizes the top 3 most divergent pairs rather than running an $O(N^2)$ cross-product.

This reduced the end-to-end pipeline execution time from 4.5 minutes down to 14 seconds, delivering a fast, responsive UX without sacrificing scientific rigor.

What’s Next?

PaperTrail is completely open source under the MIT license. Some of the features we're actively working on next:

  • TinyFish Continuous Web Monitoring: Real-time tracking of lab publication feeds and author pages.
  • Automated Systematic Literature Review (SLR) Generator: Exporting PRISMA-compliant flowcharts and evidence synthesis tables.
  • Hardware-in-the-Loop Benchmark Validation: Connecting experiment protocols directly to local MLX / llama.cpp runners for on-device replication.

If you're a researcher, engineer, or curious mind exploring literature:

  • ⭐️ Star the repository on GitHub: github.com/umeshadabala/PaperTrail
  • Check out the codebase, clone it, and run it locally with your own Ollama setup or cloud provider!

Let me know in the comments: How do you currently track scientific contradictions and emerging research gaps in your field?

Top comments (0)