DEV Community: VLSiddarth

I built a localhost proxy that stops LangChain from hallucinating on dead RAG data (and cuts token burn by 50%)

VLSiddarth — Sat, 04 Jul 2026 10:30:58 +0000

The problem nobody's proxy was solving

Every RAG pipeline I'd worked with had the same blind spot: once a document made it into the retrieval index, the LLM treated it as gospel forever. A 2022 blog post about a deprecated API, a GitHub issue that got resolved eight months ago, or an arXiv paper that has since been contradicted does not carry a "this might be stale" flag by the time it lands in your prompt. The model just reasons over it, confidently, as if it were current.

That isn't a retrieval problem. It's a freshness problem, and nothing in the standard LLM-app stack owns it. So I built KU-Gateway. It is an OpenAI-compatible proxy that sits in front of your model calls. It scores every piece of your embedded context for staleness and strips out anything that is too decayed to trust before a single token ever gets sent upstream.

The shape of it

KU-Gateway is deliberately boring to integrate. It's a FastAPI service that speaks the same /v1/chat/completions contract as OpenAI, so adopting it is a one-line change in any existing client:

from openai import OpenAI

client = OpenAI(
    api_key="sk-your-openai-key",
    base_url="http://localhost:8000/v1",   # ← the only line that changes
)

Behind that familiar interface, every request goes through a four-stage pipeline:

Your App  ──▶  KU-Gateway (:8000)  ──▶  LLM Provider (OpenAI / Anthropic / Gemini / local)
                    │
                    ├─▶ extract <context> blocks from the request
                    ├─▶ score each block via the Knowledge Universe API
                    ├─▶ drop anything past the decay threshold
                    └─▶ forward the trimmed request + print live telemetry

The scoring itself is delegated to the Knowledge Universe API, which returns a decay_score (0 = fresh, 1 = fully decayed). It also provides a knowledge_velocity label (like frozen, slow, moderate, fast, or hypersonic ) and a conflict_detected flag for content that actively contradicts more current knowledge. KU-Gateway's entire job is turning that score into a clean keep or drop decision so you do not have to think about it.

Extraction: finding the context in the first place

Requests don't arrive pre-tagged as "this part is retrieved context, this part is the user's actual question." KU-Gateway assumes anything wrapped in <context>...</context> tags inside a message is fair game for scoring:

def extract_context_chunks(messages: List[Dict[str, Any]]) -> List[ContextChunk]:
    chunks = []
    context_pattern = r"<context>(.*?)</context>"

    for msg in messages:
        content = msg.get("content", "")
        matches = re.findall(context_pattern, content, re.DOTALL)

        for i, match in enumerate(matches):
            chunk = ContextChunk(
                id=f"chunk_{i+1}_{hash(match[:100])}",
                content=match.strip(),
                source=None,
                url=None,
                title=None
            )
            chunks.append(chunk)

    return chunks

If there is nothing to extract, the request passes straight through. This setup completely avoids wasting API calls on plain conversational turns. This part of the codebase is definitely a v1 implementation. It currently assumes flat, unnested context blocks. If you want to use more elaborate embedding schemes like nested tags or structured JSON blobs, you will need a more capable parser. That upgrade is already on the roadmap.

Filtering: the actual gate

Once chunks come back scored, the decision logic is refreshingly simple. A chunk survives if its decay score is lower than the specific threshold set for its source:

def filter_chunks(self, chunks, results):
    fresh_chunks, fresh_results, blocked_chunks = [], [], []
    result_map = {r.chunk_id: r for r in results}

    for chunk in chunks:
        result = result_map.get(chunk.id)
        if result is None:
            # No evaluation result — fail open, keep the chunk
            fresh_chunks.append(chunk)
            continue

        threshold = self.get_threshold_for_source(chunk.source)

        if result.decay_score < threshold:
            fresh_chunks.append(chunk)
            fresh_results.append(result)
        else:
            blocked_chunks.append((chunk, result))

Two design decisions worth calling out:

Per-source thresholds. A single global cutoff doesn't make sense across domains because arxiv content ages differently than a Stack Overflow answer. KU_SOURCE_THRESHOLDS lets you set {"arxiv": 0.7, "github": 0.3} and the gateway will respect it per chunk.
Fail-open, deliberately. If the evaluator can't produce a result for a chunk at all (an uncaught exception, not a scoring failure), that chunk is kept rather than silently dropped. A gateway that fails closed on infrastructure hiccups is a gateway that quietly breaks your app in production. If the KU API itself is reachable but errors, it instead returns a neutral decay_score=0.5, confidence=0.0. This just goes through normal threshold comparison rather than getting special-cased.

What it actually looks like in the terminal

This isn't a hypothetical pipeline. Here is a real request against a running gateway, unmodified terminal output:

)

That request came in with 5 context chunks and 42 original tokens. Four chunks got blocked with decay scores of 0.80, 0.88, 0.73, and 0.54. leaving one fresh chunk and a clean payload of 21 tokens. 50% of the tokens in that request were pure staleness, stripped before the model ever saw them. Average decay across the blocked chunks: 0.74. That's the whole value proposition in one log line.

Every request gets this treatment automatically, and the same numbers are queryable over HTTP:

curl http://localhost:8000/v1/telemetry

{
  "requests": 12,
  "total_tokens_saved": 340,
  "total_cost_saved": 0.00068,
  "total_conflicts": 2,
  "recent": [ ... ]
}

A confession: the startup banner was dead code

Here's the kind of thing that only surfaces when you actually trace the call graph instead of trusting your own memory of what you built. telemetry.py has a fully-implemented print_startup() method that covers API key tier detection, decay thresholds, supported source counts. It even uses a Nice Rich-rendered panel. Except main.py never called it. There was no @app.on_event("startup") handler at all, only a shutdown hook that printed the session summary. The banner had been sitting there, fully written, entirely unreachable, since the initial commit.

The fix was small:

@app.on_event("startup")
async def startup_event():
    if settings.telemetry_enabled:
        telemetry.print_startup(settings.ku_api_key, settings.decay_threshold, settings.port)

I'm including this not because it's impressive, since it's a two-line fix, but because "does the thing I documented actually run" turned out to be a more useful question than "does the code compile." Worth asking of your own projects before you write the README.

Configuration, in brief

Everything is environment-variable driven via pydantic-settings, loaded from .env. The essentials:

Variable	Default	What it does
`KU_API_KEY`	(required)	Your Knowledge Universe key, must start with `ku_`
`KU_DECAY_THRESHOLD`	`0.5`	Global cutoff — chunks at or above this are blocked
`KU_SOURCE_THRESHOLDS`	`{}`	Per-source overrides as JSON
`UPSTREAM_LLM_BASE_URL`	`https://api.openai.com`	Where cleaned requests get forwarded
`KU_REDIS_ENABLED`	`false`	Cache decay scores instead of re-scoring identical chunks every call

Full reference is in the repo README.

Shipping it

There's a Dockerfile and a docker-compose.yml that spins up the gateway alongside a mock KU API and a mock LLM echo server. This setup is genuinely useful for kicking the tires without live credentials:

make docker-up

And for a real cluster, kubernetes/ has Deployment, Service, Ingress, ConfigMap, and Secret manifests ready to kubectl apply -f.

What's not done yet

I'd rather tell you this than have you find out the hard way:

Rate limiting and auth middleware are stubs. KU_RATE_LIMIT doesn't do anything yet. The middleware classes exist but pass every request through.
BYOK vault support is scaffolded, not built. KU_VAULT_ENABLED has no effect.
YAML config isn't wired up. There's a .ku-gateway.yaml.example in the repo describing a config format the code doesn't actually read yet. Environment variables are the only real input right now.
Message reconstruction is intentionally simple. It handles flat <context> blocks well; deeply nested or structured context embedding will need more work.

None of this blocks the core use case of request in, stale context stripped, clean request out. However, I'd rather be upfront about the beta-ness than have you discover a stub in production.

Try it

git clone https://github.com/VLSiddarth/KU-Gateway.git
cd KU-Gateway
python -m venv venv && source venv/bin/activate
make install
cp .env.example .env # Mint your free KU_API_KEY at https://api.knowledgeuniverse.tech/dashboard.html
make run

It's MIT-licensed and I'd genuinely welcome issues, PRs, or just someone telling me the decay thresholds are wrong for their domain. If you're building anything that hands retrieved documents to an LLM and you've never once asked "how old is this, actually?" then that is exactly the gap this fills.

Repo: https://github.com/VLSiddarth/KU-Gateway

Why RAG Pipelines Silently Hallucinate — And The Decay Score That Catches It Before The LLM Does

VLSiddarth — Sun, 24 May 2026 08:09:01 +0000

Your RAG pipeline has a blind spot. It is not your embeddings. It is not your retrieval algorithm. It is time.

Vector databases return results ranked by semantic similarity. A document from 18 months ago and a document from last week score identically if the wording is similar. The LLM receives both with equal confidence. When the older document contains superseded information — a deprecated API method, an outdated compliance rule, a retracted clinical guideline — the model hallucinates with full conviction. No exception is raised. No warning appears.

I ran a live query against our production endpoint this week: "multi-agentic ai orchestration" at difficulty 4.

Here is what came back:

arxiv:2505.02861v2   decay: 0.214  label: fresh   age: 381 days
github:harmonist     decay: 0.015  label: fresh   age: 4 days
arxiv:2601.14652v4   decay: 0.072  label: fresh   age: 118 days
github:win4r/tasks   decay: 0.317  label: aging   age: 99 days ⚠️
arxiv:2601.10560v1   decay: 0.075  label: fresh   age: 123 days
github:builderz      decay: 0.306  label: aging   age: 95 days ⚠️

Two sources flagged as aging. Not stale enough to block — but enough to warn before synthesis. Knowledge velocity: STABLE. Recommended refresh: quarterly.

The math behind the score is straightforward:

decay = 1 - 0.5^(age_days / half_life_days)

Half-life varies by source type. A GitHub repository has a 180-day half-life — code goes stale as dependencies update. An arXiv paper has a 1,095-day half-life — research ages more slowly. A Stack Overflow answer sits at 365 days.

This is the layer I built: a post-retrieval decay gate that stamps every retrieved document with a deterministic freshness score before it enters the LLM context window. It sits between your vector database and your generation step. It requires zero changes to your existing pipeline.

Try it yourself — free tier, 500 calls/month, no credit card:

Step 1 — get your key

bash
curl -X POST "https://api.knowledgeuniverse.tech/v1/signup?email=you@example.com"

Step 2 — run a query

curl -X POST https://api.knowledgeuniverse.tech/v1/discover \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"topic":"your domain here","difficulty":4,"formats":["pdf","github"]}'

The max_decay_detected field in the response tells you the worst freshness score in your retrieved set. Gate on that. Block anything above 0.4 in fast-moving domains.

How are you handling temporal staleness in your RAG pipelines today?

Andrej Karpathy said manual data ingest for AI agents is too slow. I built the fix.

VLSiddarth — Tue, 07 Apr 2026 09:00:05 +0000

Andrej Karpathy said manual data ingest for AI agents is too slow. I built the fix.

Last week Andrej Karpathy posted about building personal knowledge
bases for LLM agents. He described his workflow: manually indexing
source documents into a raw/ directory, writing custom search tools,
building a naive search engine over his wiki.

Then he wrote this:

"I think there is room here for an incredible new product
instead of a hacky collection of scripts."

He was right. So I built it.

The Problem He Identified

Karpathy's workflow is brilliant but it requires him to manually
curate every source. He clips articles with Obsidian Web Clipper,
downloads images locally, feeds them one-by-one to his LLM agent.

For a researcher at his level that works. For a developer building
production AI agents for clients, it doesn't scale.

Here's the specific failure mode I kept hitting:

You build a RAG pipeline. It works. A user asks about a Python library.
Your retriever finds a Stack Overflow answer with cosine similarity 0.94.
The LLM answers confidently. The user follows the advice. It breaks their project.

The Stack Overflow answer was from 2021. The library changed its API in 2023.

Your retriever did its job perfectly. Your vector store had no concept
of when that document was written. No exception was raised. No warning
was shown. The cosine similarity score told you nothing about whether
the knowledge was still true.

This is the silent failure mode of every RAG pipeline in production.
Tavily, Exa, and SerpAPI don't tell you when their results are stale.

So I built a retrieval API that does.

What I Built

Knowledge Universe is an open-source retrieval API that gives
Karpathy's LLM wiki agents something they currently don't have:
a production-grade data ingestion layer that crawls 18 knowledge
sources simultaneously, scores every result for freshness, and
returns structured documents in 3 seconds.

# Install the CLI
pip install knowledge-universe

# Get a free API key
ku signup you@email.com

# Run your first query
ku discover "transformer architecture" --difficulty 3

Output:
Found 8 sources [2980ms]

🟢 [arxiv] Learning Novel Transformer Architecture for Time-series

https://arxiv.org/abs/2502.13721v1
decay=0.23 (fresh) quality=8.5/10
⚪ [kaggle] In-depth guide to Transformer architecture

https://www.kaggle.com/code/tientd95/in-depth-guide-to-transformer-arc
decay=0.40 (unknown) quality=4.5/10
🟢 [paperswithcode] Gradient Boosting within a Single Attention Layer

https://arxiv.org/abs/2604.03190
decay=0.01 (fresh) quality=7.6/10
🟠 [github] An-Jhon/Hand-Drawn-Transformer

https://github.com/An-Jhon/Hand-Drawn-Transformer
decay=0.68 (stale) quality=2.6/10
🟡 [semantic_scholar] Transformer+transformer architecture for image captioni

https://doi.org/10.11591/ijai.v14.i3.pp2338-2346
decay=0.44 (aging) quality=5.1/10
🟢 [arxiv] A Survey of Graph Transformers: Architectures, Theories

https://arxiv.org/abs/2502.16533v2
decay=0.23 (fresh) quality=8.9/10
⚪ [kaggle] LB 0.73 single fold transformer architecture

https://www.kaggle.com/code/hengck23/lb-0-73-single-fold-transformer-a
decay=0.40 (unknown) quality=4.5/10
🟠 [github] tum-pbs/pde-transformer

https://github.com/tum-pbs/pde-transformer
decay=0.70 (stale) quality=2.5/10

Cache hit: False | Time: 2980ms

Every result tells you not just what it found, but how much to trust it.

Live API: https://vlsiddarth-knowledge-universe.hf.space

GitHub: https://github.com/VLSiddarth/Knowledge-Universe

The Architecture

The core idea: run 18 crawlers in parallel, score everything,
return the best 8-10 results with freshness metadata attached.

Your App / Agent
          │
          ▼  POST /v1/discover
┌─────────────────────────────────────────────────┐
│            Knowledge Universe API               │
│                                                 │
│  1. Cache check (Redis) ──── HIT → 200ms return │
│        │ MISS                                   │
│  2. asyncio.gather(18 crawlers, per-timeouts)   │
│        ├── arXiv          (25s timeout)         │
│        ├── CrossRef       (8s) ← [Academic]     │
│        ├── PapersWithCode (8s) ← [SOTA Models]  │
│        ├── Documentation  (3s) ← [Fast-Fail]    │
│        ├── GitHub         (8s)                  │
│        ├── StackOverflow  (6s)                  │
│        ├── HuggingFace    (8s)                  │
│        ├── Kaggle         (6s)                  │
│        ├── YouTube        (8s)                  │
│        ├── Sketchfab      (5s) ← [3D Spatial]   │
│        ├── Freesound      (5s) ← [Audio]        │
│        ├── Wikipedia      (5s)                  │
│        ├── MIT OCW        (5s)                  │
│        ├── OpenLibrary    (5s)                  │
│        ├── Podcast Index  (5s)                  │
│        ├── Libgen         (4s)                  │
│        ├── CommonCrawl    (2s) ← [Fast-Fail]    │
│        └── GH Archive     (2s) ← [Fast-Fail]    │
│        │                                        │
│  3. Semantic pre-filter (cosine sim > 0.25)     │
│  4. Quality ranker (5-dimension scoring)        │
│  5. Knowledge Decay Engine                      │
│  6. LLM reranker (all-MiniLM-L6-v2)             │
│  7. Coverage Confidence Score                   │
│  8. Cache result (Redis, 4h TTL)                │
│        │                                        │
│  Return: sources + decay_scores + confidence    │
└─────────────────────────────────────────────────┘

Why asyncio.gather and not threads?

Each crawler is an async HTTP call. asyncio.gather runs all 18
simultaneously, so the total wall time equals the slowest crawler,
not the sum. The parallel ceiling is arXiv at ~2.5s for complex queries.

One lesson that cost me 5 seconds of latency: one blocking crawler
kills everything. My original Kaggle integration used the official
SDK which runs synchronous urllib3 under the hood. I wrapped it in
run_in_executor thinking that was fine. It held a thread pool slot
for 2.5 seconds on every query and pushed cold latency from 3s to 8s.

The fix: replace the SDK with direct async HTTP calls using httpx:

# Before — blocks the thread pool
async def crawl(self, topic, difficulty):
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(None, self._crawl_sync, topic, difficulty)

# After — true async, 300ms for same results
async def crawl(self, topic, difficulty):
    async with httpx.AsyncClient() as client:
        datasets = await client.get(
            "https://www.kaggle.com/api/v1/datasets/list",
            params={"search": topic, "sortBy": "usability"},
            headers={"Authorization": f"Basic {self._encoded_creds}"}
        )
        return self._parse(datasets.json(), difficulty)

Cold latency dropped from 8.8s to 3.1s with that one change.

The Knowledge Decay Formula

This is the part that doesn't exist in any other retrieval API.

Every result gets a decay score computed from its age and source type:
decay = 1 - 0.5 ^ (age_days / half_life)
freshness = 1 - decay

Half-lives are tuned per platform based on how fast knowledge
in that domain becomes outdated:

Platform	Half-life	Why
HuggingFace	120 days	ML model landscape changes monthly
GitHub	180 days	Dependencies update constantly
YouTube	270 days	Library tutorials date quickly
Stack Overflow	365 days	API answers age with framework versions
arXiv	1,095 days	Research papers have longer shelf life
Wikipedia	1,460 days	Actively maintained, slow decay
Open Library	1,825 days	Books revised infrequently

The output on every result:

"stackoverflow:59523557": {
  "decay_score": 0.986,
  "freshness": 0.014,
  "label": "decayed",
  "age_days": 2263,
  "half_life_days": 365
}

That Stack Overflow answer from 2021 with cosine similarity 0.94?
Its freshness score is 0.014. It gets downweighted before it
reaches your LLM. The silent failure mode is no longer silent.

Fast-moving topics get an additional volatility multiplier.
Topics like "LLMs", "React", "Docker", "Claude" use a ×1.1
multiplier on the decay rate — knowledge in those domains
goes stale faster than the platform average.

The 5-Dimension Quality Ranker

Before decay is applied, each source gets a base quality score:

WEIGHTS = {
    "authority":            0.35,  # platform trust + content type
    "difficulty_alignment": 0.30,  # how well difficulty matches request
    "completeness":         0.20,  # metadata richness
    "social_proof":         0.10,  # stars, citations, views (log-scaled)
    "accessibility":        0.05,  # open access bonus
}

final_score = base_quality * decay_penalty_multiplier

Difficulty alignment is the most impactful dimension for
Karpathy's use case specifically. When you're feeding an LLM wiki
agent, you want sources matched to the agent's context level.
A research synthesis agent should get arXiv papers, not YouTube
explainers. A learning tool for beginners should get the opposite.

# difficulty gap penalty
# gap=0: score 10.0  (perfect match)
# gap=1: score 8.5   (acceptable)  
# gap=2: score 6.0   (marginal)
# gap=3: score 2.0   (nearly blocked)

The Coverage Confidence Score

This is the feature that surprised me most when I built it.

After reranking, the API computes the average cosine similarity
between your query and the top results. If the average falls below
0.45, it warns you and suggests better queries:

"coverage_intelligence": {
  "confidence": 0.36,
  "confidence_label": "low",
  "coverage_warning": true,
  "warning_message": "Low confidence — results may not match intent",
  "suggested_queries": [
    "attention mechanism self-attention explained",
    "transformer encoder decoder architecture tutorial",
    "attention is all you need paper walkthrough"
  ]
}

This matters for LLM wiki agents specifically. When the agent asks
an ambiguous question that doesn't match how sources are indexed,
instead of silently returning mediocre results, the API tells the
agent to rephrase. The agent can use the suggested queries directly.

Karpathy mentioned he runs "health checks" over his wiki to find
inconsistent data and impute missing data. Coverage confidence is
essentially that health check, automated and running on every query.

Performance vs Competitors

Tested against Tavily, Exa, and SerpAPI using identical queries:

Metric	Knowledge Universe	Tavily	Exa	SerpAPI
Cold latency	3.1s	5.4s	1.5s	3.5s
Cache hit	220ms	N/A	N/A	N/A
Decay scoring	✅	✗	✗	✗
Confidence score	✅	✗	✗	✗
Difficulty ranking	✅	✗	✗	✗
Source diversity	18 typed platforms (incl. 3D & Audio)	Web only	Web only	Google only

KU is faster than Tavily on cold queries despite hitting 18 typed
sources vs Tavily's general web index. The parallel architecture
is what makes this possible — the wall clock time equals the
slowest single crawler, not the sum of all crawlers.

Note on Exa: Exa is faster (1.5s) because it uses a single unified
search index rather than parallel crawling. The tradeoff is no decay
scoring and no source type diversity — you get whatever their index
decided to rank.

LangChain Integration — Drop-in Ready

import requests
from langchain_core.documents import Document

def get_knowledge_universe_docs(
    topic: str,
    difficulty: int = 3,
    formats: list = None,
    api_key: str = "your_key_here",
    min_freshness: float = 0.3,  # filter sources below 30% fresh
) -> list[Document]:

    formats = formats or ["pdf", "github", "html", "video", "stackoverflow"]

    resp = requests.post(
        "https://vlsiddarth-knowledge-universe.hf.space/v1/discover",
        headers={"X-API-Key": api_key},
        json={
            "topic": topic,
            "difficulty": difficulty,
            "formats": formats,
            "max_results": 10,
        },
        timeout=30,
    ).json()

    # Check coverage confidence
    cov = resp.get("coverage_intelligence", {})
    if cov.get("coverage_warning"):
        print(f"⚠️  Low confidence. Try: {cov.get('suggested_queries', [])}")

    docs = []
    decay_map = resp.get("decay_scores", {})

    for source in resp.get("sources", []):
        decay = decay_map.get(source["id"], {})
        freshness = decay.get("freshness", 0.5)

        # Filter stale sources before they reach your LLM
        if freshness < min_freshness:
            continue

        docs.append(Document(
            page_content=source.get("summary", ""),
            metadata={
                "title":         source["title"],
                "url":           source["url"],
                "platform":      source["source_platform"],
                "freshness":     freshness,
                "decay_label":   decay.get("label"),
                "quality_score": source.get("quality_score"),
                "difficulty":    source.get("difficulty"),
            }
        ))

    return docs


# Usage — drop into any existing LangChain RAG chain
docs = get_knowledge_universe_docs(
    topic="transformer architecture",
    difficulty=3,
    api_key="ku_test_...",
)

for doc in docs:
    print(f"[{doc.metadata['decay_label']}] {doc.metadata['title']}")
    print(f"  freshness={doc.metadata['freshness']:.2f}")
    print(f"  url={doc.metadata['url']}")

For Karpathy's LLM wiki use case specifically:

# Give your LLM wiki agent a tool that does the manual ingest
# he described — automatically, with freshness scoring

wiki_sources = get_knowledge_universe_docs(
    topic="mixture of experts routing algorithms",
    difficulty=5,           # researcher-level sources only
    min_freshness=0.5,      # only recent sources go into the wiki
    formats=["pdf", "github"]  # papers and implementations
)

# Feed directly to your wiki agent
for source in wiki_sources:
    agent.ingest_to_wiki(source)

The manual raw/ directory collection Karpathy describes is now
three lines of code.

Things That Didn't Work

MinHash LSH deduplication misses near-identical titles.
Wikipedia returns both "Neural network" and "Neural network
(machine learning)" as separate articles. After normalization
they differ, so both pass deduplication. Fixed with a
parenthetical-stripping step before the hash comparison.

Per-crawler timeouts were the wrong abstraction initially.
I started with a global 8s timeout for all crawlers.
CommonCrawl and GHArchive always time out at 8s with 0 results,
wasting the full 8s on every query. Setting them to 2s fast-fail
dropped the parallel ceiling from 8s to 3s.

Semantic Scholar blocked Hugging Face IP addresses.
I originally used Semantic Scholar for academic papers. In late 2024, they changed their policy and started throwing 403 Forbidden errors for server-to-server requests from Hugging Face Spaces free tiers.

The Fix: I ripped it out and integrated CrossRef. They index 150M+ scholarly works, have a fully open API (CC0 metadata), and actively encourage programmatic access via their "polite pool" (just pass your email in the User-Agent). It gave me access to IEEE, ACM, and Nature papers that arXiv misses, with zero rate-limit blocks.

Lesson: profile each crawler individually before setting any
global timeout.

_is_stale() was silently killing cache hit rate.
The function checked if a cached result was older than 80% of
the TTL (14,400s × 0.8 = 11,520s). Any query between 3.2 and
4 hours old triggered a full cold re-crawl even though Redis
still had the result. Cache hit rate was 25%. One-line fix:
use the full TTL. Hit rate went to 50%+ immediately.

The shared model singleton took too long to notice.
Both LocalLLMReranker and CoverageConfidenceScorer were
independently loading all-MiniLM-L6-v2 (90MB, ~300MB RAM).
Loading twice pushed HuggingFace Spaces free tier (2GB limit)
near the ceiling and added ~500ms to first requests. Fixed with
a module-level singleton:

# src/integrations/shared_model.py
_model = None
_model_lock = threading.Lock()

def get_shared_model():
    global _model
    if _model is not None:
        return _model
    with _model_lock:
        if _model is None:
            from sentence_transformers import SentenceTransformer
            _model = SentenceTransformer("all-MiniLM-L6-v2")
    return _model

Both classes now call get_shared_model(). Model loads once.
Shared embeddings from the reranker pass directly to the
confidence scorer — zero extra encode() calls per request.

Try It Now

# Install the Python SDK
pip install knowledge-universe

# Or Node
npm install knowledge-universe

# Get a free API key (500 calls/month, no credit card)
ku signup you@email.com

# Query the live API directly
curl -X POST https://vlsiddarth-knowledge-universe.hf.space/v1/discover \
  -H "X-API-Key: ku_test_..." \
  -H "Content-Type: application/json" \
  -d '{
    "topic": "transformer architecture",
    "difficulty": 3,
    "formats": ["pdf", "github", "html", "video"],
    "max_results": 10
  }'

Live API + Swagger docs:

https://vlsiddarth-knowledge-universe.hf.space

GitHub (MIT licensed):

https://github.com/VLSiddarth/Knowledge-Universe

What's Next

Two things I'm actively building:

1. Streaming results — return sources as they arrive from
each crawler rather than waiting for all 18. The first 3 results
could be in your agent pipeline within 800ms. You see something
immediately; the pipeline enriches as more crawlers complete.

2. /v1/monitor webhook alerts — register a topic and a
webhook URL. Knowledge Universe checks that topic daily. When
freshness drops below a threshold or a significantly better
source appears, it pushes an update to your endpoint. Your
RAG pipeline stays current without polling.

If you're building LLM agents that need external knowledge —
whether it's Karpathy's wiki pattern, a production RAG pipeline,
or something in between — I'd genuinely like to hear what breaks.

What's your current approach to handling source freshness in
retrieval? Drop it in the comments.

Tags: python, ai, machinelearning, webdev