DEV Community

DataChainHub Engineering
DataChainHub Engineering

Posted on

Designing a Zero-Synthetic-Data Crypto Analytics Portal with Multi-RPC Gas Aggregation & LLMs.txt

When building real-time blockchain analytics dashboards, two engineering challenges surface quickly: avoiding synthetic fallback numbers during upstream RPC rate limits, and making structured domain data cleanly readable by both traditional search crawlers and AI retrieval engines (OAI-SearchBot, ClaudeBot, PerplexityBot).

Here is a concise breakdown of how we designed the real-time data pipeline and AI-friendly indexing architecture for DataChainHub.

1. Concurrent Multi-Chain Gas Aggregation (curl_multi + LKG SWR)

Querying gas metrics across 9+ heterogeneous networks sequentially introduces unacceptable tail latency (2.5s–4.0s per uncached request). Instead of serial HTTP fetches or static hardcoded estimates, our Real-Time Multi-Chain Gas & Fee Tracker batches upstream queries into a single concurrent pool:

  • Ethereum Mainnet: Gas oracle tiers (SafeGasPrice, ProposeGasPrice, FastGasPrice) with sub-second caching.
  • EVM Layer-2s & Alt-L1s (Arbitrum, Optimism, Base, BSC, Polygon, Avalanche): Direct JSON-RPC eth_gasPrice calls under a strict 3s timeout budget.
  • Non-EVM Networks: suix_getReferenceGasPrice for Sui and recommended sat/vB mempool fee tiers for Bitcoin.
  • Stale-While-Revalidate (SWR) & Last-Known-Good (LKG): Snapshots are prewarmed every 60 seconds in rate-limited 4-concurrency batches. If an upstream node times out, the endpoint serves the timestamped LKG snapshot rather than fabricating synthetic numbers.

We apply the same zero-synthetic-data architecture to derivative telemetry on our Perpetual Futures Funding Rate Board, aggregating live 8-hour settlement rates in parallel without client-side CORS bottlenecks.

2. Implementing the llmstxt.org Standard & Bot-Aware Hydration

Modern search renderers and AI answer engines frequently truncate oversized HTML documents beyond 125–150 KB or waste crawl budget on high-frequency XHR polling. Three small optimizations solved both issues:

  1. Externalized Minified Core CSS: Moving inline styles into a fingerprinted static bundle shifted <body> semantic content from 152 KB down to < 55 KB.
  2. Bot-Aware Polling Guards: Detecting headless crawler user-agents (such as Google WRS and Bingbot) on live boards so initial SSR data renders cleanly without spawning 3s/5s setInterval polling loops.
  3. Structured /llms.txt Index: Serving a clean Markdown directory at shulianku.com/llms.txt that maps 11 live quantitative tools, 72 tokenomics profiles, 10 sector hubs, and 100 Web3 glossary definitions for deterministic retrieval.

Top comments (0)