Turning public-apis into a Cache-Friendly RAG Retrieval Layer

JJ657 — Tue, 08 Sep 2026 13:41:10 +0000

Turning a Public API Catalog into a Cache-Friendly RAG Retrieval Layer

At 2:00 AM, your internal API discovery assistant trips its monthly token quota. The culprit isn't an unexpected traffic spike—it is a steady trickle of routine developer queries like "Which free weather API supports HTTPS and requires no auth?" repeatedly dumping 15,000 tokens of raw Markdown headers, repetitive table formatting, and near-identical catalog rows straight into context. Naive RAG pipelines rarely fail with dramatic 500 errors; they silently bleed infrastructure budgets through redundant input tokens and destroyed cache hit rates.

The core production pain point is repetitive retrieval context cost.

A toy RAG implementation typically splits the public-apis/public-apis README every N characters, embeds every fragmented chunk, pulls the top five nearest neighbors, and dumps them blindly into a prompt. In production, this naive pattern triggers three operational failures:

Category headers decouple from API rows, destroying semantic filtering once chunks cross arbitrary character boundaries.
Near-identical descriptions trigger duplicate retrieval, packing context windows with redundant endpoints.
Static instructions and catalog metadata are resent on every query, blowing past model KV-caches and multiplying token bills.

When integrating an external catalog like public-apis/public-apis, retrieval must be record-oriented rather than document-oriented. Each API entry must become an isolated, searchable record with category context bound directly to its text payload. You then decouple long-lived, cache-eligible system instructions from query-specific volatile context.

Architecture: Treat the Repository as an Ingestion Source, Not a Live Database

Treat upstream Git repositories as immutable batch sources. Querying the raw upstream README at runtime invites upstream rate limits, format drift, and transient GitHub outages directly into your critical user path.

A resilient retrieval layer enforces four architectural boundaries:

GitHub repository snapshot
        |
        v
Markdown parser -> normalized API records -> content hash / deduplication
        |                                      |
        v                                      v
Embedding index <------------------------ versioned JSONL archive
        |
        v
Retriever -> context assembler -> cache-friendly model prompt -> answer

The repository snapshot is immutable for a given ingestion run. Do not fetch the README at request time. A transient GitHub outage, branch update, or formatting change should not alter live answers halfway through a user session.

The parser should retain fields that users actually ask about:

category
API name
description
authentication requirement
HTTPS support
CORS status
canonical URL
source revision or snapshot timestamp

Crucially: embed the category directly into each chunk’s text payload, not just as external metadata. While vector store metadata filters are effective when supported, baking categories into the text ensures semantic survival when assembling context outside proprietary index filters.

A Runnable Chunking and Deduplication Script

This script parses the table-style records typically used by public API catalogs, produces one record per API, and removes duplicates based on normalized semantic content. It intentionally does not guess whether an endpoint is still available; catalog ingestion and runtime endpoint health checks are separate jobs.

# build_catalog.py
import hashlib
import json
import re
from pathlib import Path

ROW = re.compile(r"^\|\s*(.*?)\s*\|\s*(.*?)\s*\|\s*(.*?)\s*\|\s*(.*?)\s*\|\s*(.*?)\s*\|$")
CATEGORY = re.compile(r"^#{2,3}\s+(.+?)\s*$")

def clean(value: str) -> str:
    value = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", value)
    return re.sub(r"\s+", " ", value).strip()

def fingerprint(record: dict) -> str:
    stable = "|".join(
        record[key].lower()
        for key in ("category", "name", "description", "auth", "https")
    )
    return hashlib.sha256(stable.encode()).hexdigest()[:16]

def parse(readme: Path):
    category = "Uncategorized"
    seen = set()

    for line in readme.read_text(encoding="utf-8").splitlines():
        heading = CATEGORY.match(line)
        if heading:
            category = clean(heading.group(1))
            continue

        match = ROW.match(line)
        if not match or match.group(1).lower() == "api":
            continue

        name, description, auth, https, cors = map(clean, match.groups())
        if not name or name.startswith("-"):
            continue

        record = {
            "category": category,
            "name": name,
            "description": description,
            "auth": auth,
            "https": https,
            "cors": cors,
        }
        record["chunk"] = (
            f"Category: {category}\n"
            f"API: {name}\n"
            f"Description: {description}\n"
            f"Authentication: {auth}\n"
            f"HTTPS: {https}\n"
            f"CORS: {cors}"
        )
        record["id"] = fingerprint(record)

        if record["id"] not in seen:
            seen.add(record["id"])
            yield record

records = list(parse(Path("README.md")))
Path("catalog.jsonl").write_text(
    "\n".join(json.dumps(r, ensure_ascii=False) for r in records) + "\n",
    encoding="utf-8",
)
print(f"Wrote {len(records)} unique API records")

Run it against a pinned checkout:

git clone --depth 1 https://github.com/public-apis/public-apis.git
cd public-apis
python ../build_catalog.py

A production pipeline should store the Git commit SHA next to catalog.jsonl. That gives an operator a reproducible answer trail: which catalog state informed the response, which records were retrieved, and which model generated the result.

Cache-Friendly Context Assembly

Prompt caching only works when the prompt prefix is strictly byte-identical. Prepending timestamps, random request IDs, user locale, or the user's question before your reusable catalog policy immediately destroys cacheability across requests.

A practical assembly order enforces clear prefix boundaries:

prompt_layout:
  cached_prefix:
    - response policy
    - field definitions
    - catalog snapshot revision
    - compact category guide
  retrieved_context:
    max_records: 6
    max_chars_per_record: 900
    deduplicate_by: id
  volatile_suffix:
    - user_question
    - session-specific constraints

The catalog itself is usually too large to keep entirely in every prompt. Cache the stable answer policy and schema; retrieve only records that match the query. If a workload repeatedly asks within a narrow domain, such as weather, finance, or geocoding APIs, a category summary can also live in the cached prefix while individual entries remain retrieved context.

The trade-off is freshness. A stable prefix improves cache reuse but delays catalog updates until the next indexed snapshot is activated. That is acceptable for discovery guidance, but not for claims that an API is currently operational, free, rate-limit-free, or safe for regulated data. Those require a separate endpoint validation process.

Failure Modes That Matter

The most common failure is treating catalog metadata as verified operational truth. “No authentication” in a list entry does not mean unlimited use, no abuse controls, or no later registration requirement. The assistant should phrase results as catalog-derived leads and point users to the provider documentation before implementation.

Another failure is duplicate retrieval. API names may be repeated across categories or descriptions may differ only in punctuation. Hashing normalized records removes exact semantic duplicates before embedding; retrieval-time deduplication by record ID prevents the same entry from occupying several context slots.

Finally, do not let the model invent missing fields. If a record lacks a clear CORS or auth value, return “not specified in the indexed catalog snapshot,” not an inference.

When the prompt structure is stable, native prompt caching can substantially reduce the billed input associated with repetitive system instructions and catalog guidance. The actual reduction depends on provider cache rules, cache hit rate, prompt length, and how often the catalog prefix changes; it must be measured from request-level usage data rather than assumed. Combining cache-friendly RAG prompt structures with B-Lost’s native prompt caching is intended to reduce repetitive query-context costs, potentially by up to 90% in high-reuse workloads.

The hardest operational dilemma in catalog-driven RAG isn't indexing scale—it is drawing the boundary between static, cacheable catalog metadata and volatile, live endpoint telemetry. How does your engineering team structure prompt boundaries to keep KV-caches warm without serving stale API schemas? Do you isolate dynamic uptime status into a secondary lookup pass or run inline synthetic probes? Drop your architecture or battle scars in the comments below.