Title: Building a Zero-Backend AI Stock Bubble Tracker: Architecture, Web Scraping, and Performance Lessons

Every time I open X (formerly Twitter) or Hacker News, I see developers arguing about the AI ecosystem. One camp claims we’re in a permanent super-cycle, while the other insists that massive GPU Capex without proportional software revenue will trigger a dot-com-style market crash.

Instead of taking sides on social media, I decided to do what any sane software engineer would do: build a lightweight, real-time metrics dashboard to track the data myself.

The result is AI Stock Bubble Index.

In this post, I want to walk you through the architectural decisions, client-side performance tricks, data pipeline challenges (like crawling Google News RSS via local proxies without getting rate-limited), and why I chose a completely zero-backend / JAMstack approach for a data-heavy utility site.

---

The Goal: Low Latency, Zero Server Costs, Max Scannability

When I set out to build this dashboard, I had a strict set of architectural rules:

1. Zero Database Overheads: I didn't want to maintain a Postgres or MongoDB instance just to serve static or semi-static analytical metrics.
2. Sub-second Page Loads: Financial dashboards are notoriously bloated. Most stock portals load 40MB of tracking scripts and take five seconds to render a single chart. My site needed to feel instant.
3. Flawless Core Web Vitals: Cumulative Layout Shift (CLS) needed to be zero, and First Contentful Paint (FCP) had to stay under 500ms on low-end mobile devices.
4. Automated Data Discovery: News headlines, P/E ratio shifts, and sentiment metrics needed to be updated asynchronously via simple, build-time or client-side JSON hydration.

Here’s how the architecture looks and the engineering lessons learned along the way.

---

Architectural Overview

The stack is deceptively simple:

• Frontend: React (SPA) with component-driven state management.
• Component Library: Ant Design, custom-shaken to ensure minimal CSS and JS bundle footprints.
• Data Layer: Asynchronously generated, statically served JSON schemas hosted via CDN endpoints.
• Scrapers & Aggregators: Node.js (ESM) automation scripts that fetch RSS discovery candidate streams, parse XML, score relevance via term-matching algorithms, and generate structured data payloads.

┌───────────────────────────────┐
│ Google News / Financial Feeds │
└───────────────┬───────────────┘
                │
                ▼ (Node.js ESM Scraper + Proxy)
┌───────────────────────────────┐
│ Term Matching & Weighting     │
└───────────────┬───────────────┘
                │
                ▼
┌───────────────────────────────┐
│ JSON Static Assets (Data CDN) │
└───────────────┬───────────────┘
                │
                ▼ (React Hydration)
┌───────────────────────────────┐
│  AI Stock Bubble UI Dashboard │
└───────────────────────────────┘

---

Technical Challenge 1: Fetching External Feeds Without Bottlenecks or Proxy Failures

One of the first features of the platform was an automated news discovery tool that aggregates headlines regarding AI Capex, valuations, revenue, and model releases.

If you’ve ever tried scripting Node.js to fetch Google News RSS feeds ([https://news.google.com/rss/search?q=](https://news.google.com/rss/search?q=)...), you know it is a minefield.

Lesson A: Node.js fetch (Undici) Ignores Native Proxy Environment Variables

If you run Node 18+ and try using the native fetch() method with standard export HTTPS_PROXY variables in your local environment, it will fail with `UND_ERR_CONNECT_TIMEOUT`.

Why? Because Node's global fetch is built on top of undici, which intentionally bypasses standard system environment proxy flags to maintain spec compliance with Web APIs.

To fix this, you have to explicitly pass a dispatcher or use ProxyAgent directly inside your request logic:

import fs from "node:fs";
import { XMLParser } from "fast-xml-parser";
import { ProxyAgent } from "undici";

// Explicitly route requests through local proxy when testing from restrictive networks
const proxyAgent = new ProxyAgent({ uri: "http://127.0.0.1:7890" });

async function fetchFeed(feedUrl) {
  const response = await fetch(feedUrl, {
    dispatcher: proxyAgent, // Mandatory for undici to route through local tunnels
    headers: {
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/120.0.0.0 Safari/537.36",
      "Accept": "application/rss+xml, application/xml;q=0.9, */*;q=0.8",
      "Accept-Language": "en-US,en;q=0.9"
    }
  });

  if (!response.ok) throw new Error(`HTTP ${response.status}`);
  return await response.text();
}

Lesson B: Custom User-Agents Will Get You Instantly Blocked

Do not set custom User-Agents like "My-Cool-Crawler/1.0". Google’s edge security will identify the non-browser headers and instantly drop your connections with a 403 Forbidden or 503 Service Unavailable. Always mirror standard modern Chrome/Safari User-Agents and include Accept-Language headers.

---

Technical Challenge 2: Client-Side Scoring & Data Normalization

Instead of sending unstructured text to expensive LLM endpoints on every scrape iteration, I implemented a lightweight, term-matching relevance engine directly in the ingestion script.

We define two primary arrays:

1. Company Entities: Hardware, cloud, and foundation model providers (Nvidia, Microsoft, TSMC, OpenAI, DeepSeek, etc.).
2. Relevance Keywords: Financial & risk terms (bubble, valuation, Capex, data center, debt, inference cost, ROI, guidance).

const companyTerms = ["Nvidia", "Microsoft", "TSMC", "OpenAI", "DeepSeek", "Broadcom"];
const relevanceTerms = ["bubble", "valuation", "capex", "debt", "inference cost", "margin"];

export function evaluateHeadline(headline) {
  const normalized = headline.toLowerCase();

  const matchedCompanies = companyTerms.filter((c) => 
    normalized.includes(c.toLowerCase())
  );

  const matchedTerms = relevanceTerms.filter((t) => 
    normalized.includes(t.toLowerCase())
  );

  // Weighted scoring logic: Companies are weighted higher than generic financial buzzwords
  const relevanceScore = (matchedCompanies.length * 2) + matchedTerms.length;

  return {
    matchedCompanies,
    matchedTerms,
    relevanceScore
  };
}

This simple algorithm filters out noisy headlines and retains high-signal candidates without requiring a heavy Python NLP server or paying $50/month in LLM token fees.

---

Technical Challenge 3: Eliminating Cumulative Layout Shift (CLS) on Financial Charts

If you are rendering interactive line charts or data tables on client devices, layout thrashing is your biggest enemy. Users hate clicking a button only for the layout to jump 200 pixels down because a chart canvas finished rendering a second late.

Here is how I kept CLS down to 0.00 on aistockbubbleindex.com:

1. Explicit Aspect-Ratio Wrappers

Never let your charting component dictate container height dynamically upon data arrival. Always wrap canvases in CSS grid/flex containers with fixed or aspect-ratio locked placeholders:

.chart-container {
  width: 100%;
  min-height: 380px;
  aspect-ratio: 16 / 9;
  contain: layout intrinsic-size; /* Prevents browser reflow during dynamic canvas mounting */
}

2. Decoupled UI State from Canvas Rendering

When users toggle between different metrics (e.g., P/E ratios vs. Capex spending trends), the DOM metadata (headers, index numbers, legend text) updates immediately in React state, while heavy canvas re-draws are deferred using standard requestIdleCallback or microtask queuing.

---

Technical Challenge 4: SPA SEO for Niche Keywords

Single Page Applications are historically terrible at ranking for specific search terms like "AI stock bubble index" unless you handle DOM structure carefully.

Since I opted against a full server-side Node instance (like Next.js) to keep hosting costs at absolute zero, I focused heavily on semantic HTML optimization:

• Semantic Landmarks: No generic <div> soup for critical data blocks. I used <main>, <article>, <section>, and <aside> strictly for structured content.
• Structured JSON-LD: Injecting Schema.org data schemas directly into the head tag during static generation so search engine bots can parse metric definitions instantly.
• Automated Build-time Sitemaps: Every time news or index candidate files update, a build script updates the sitemap.xml and pings Search Console endpoints automatically.

---

Results & Performance Benchmarks

By eliminating backend database queries and keeping the frontend architecture razor-thin:

• Lighthouse Performance Score: 98/100
• First Contentful Paint (FCP): ~0.4s
• Cumulative Layout Shift (CLS): 0.00
• Monthly Server Costs: $0 (served directly via CDN static edge networks)

Most importantly, I have a functional, noise-free utility that I actually use every week to check the temperature of the AI market without wading through social media hype.

---

Takeaways for Developers

If you want to build data-driven tools or side projects in 2026, you don't always need a complex server infrastructure.

Before you launch a Docker cluster or pay for database instances:

1. Ask if your data can be pre-parsed and served as static, edge-cached JSON.
2. Keep your frontend component bundles minimal.
3. Optimize your scraping logic with simple term-matching algorithms before reaching for heavy AI models.

Check out the live platform here: https://aistockbubbleindex.com

I’d love to hear your thoughts! How are you handling external API feeds in Node.js ESM setups? How do you keep client-side dashboard rendering fast? Let’s chat in the comments below!

---