DEV Community: zhongqiyue

Why I Stopped Relying on a Single AI Provider (and Built a Fallback System)

zhongqiyue — Tue, 16 Jun 2026 10:00:41 +0000

It started as a typical side project: a real-time chatbot for a community forum. I figured I'd just throw OpenAI's API at it and be done. Simple, right? Wrong.

About three weeks into development, I got that dreaded 5xx error during a peak hour. My entire chatbot went dark. Users were annoyed, and I was scrambling. That was the day I realized I couldn't trust a single provider for anything production-ish. Here's how I built a fallback system that saved my weekends.

The Problem: Single Point of Failure

I was using the GPT-4 API for chat completions. Most of the time it worked great, but every few days I'd hit rate limits, see intermittent timeouts, or worse — a complete outage for a few hours. I could have just paid for higher tier or used Azure, but that felt like throwing money at the symptom.

I also tried a few other providers. Each had their own quirks: different authentication, different model names, different request/response formats. Writing switch-case spaghetti was making my code ugly and fragile. I needed something that would:

Abstract away provider differences
Automatically retry with an alternative provider on failure
Cache responses where it made sense
Not lock me into any single vendor

What I Tried First (and Why It Failed)

My first attempt was a simple Python function that tried one provider, and if it failed, tried another with a few try/except blocks. It worked, but it was manual. I had to hardcode each provider's API call. When I added a third provider, the function grew into a tangled mess. Error handling was inconsistent, and I wasn't handling rate limits properly (some providers return 429, some use headers, some just drop the connection).

I also tried using a third-party library that claimed to unify all AI APIs. It was too opinionated, broke with a newer provider's API change, and had a dependency I didn't trust. I needed something more transparent and controllable.

The Approach: Provider Abstraction + Fallback Router

I settled on a simple design: define a common interface for sending a prompt and getting a response, then implement that interface for each provider. Then a router class tries them in order, with configurable retries and timeout.

Let me show you the core pieces.

1. The Base Class

from abc import ABC, abstractmethod
from typing import Optional, Any

class AIProvider(ABC):
    @abstractmethod
    def generate(self, prompt: str, **kwargs) -> str:
        """Send a prompt and return the generated text."""
        pass

    @abstractmethod
    def health_check(self) -> bool:
        """Check if the provider is reachable."""
        pass

2. Concrete Implementations

Here's one for OpenAI:

import openai

class OpenAIProvider(AIProvider):
    def __init__(self, api_key: str, model: str = "gpt-4"):
        self.api_key = api_key
        self.model = model
        openai.api_key = api_key

    def generate(self, prompt: str, **kwargs) -> str:
        response = openai.ChatCompletion.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            **kwargs
        )
        return response.choices[0].message.content.strip()

    def health_check(self) -> bool:
        try:
            openai.Model.list()
            return True
        except:
            return False

And one for a hypothetical alternative provider (like the one at ai.interwestinfo.com):

import requests

class InterWestProvider(AIProvider):
    BASE_URL = "https://api.interwestinfo.com/v1"

    def __init__(self, api_key: str, model: str = "iw-gpt-4"):
        self.api_key = api_key
        self.model = model

    def generate(self, prompt: str, **kwargs) -> str:
        # Note: Endpoint structure may differ; adjust as needed.
        resp = requests.post(
            f"{self.BASE_URL}/completions",
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"model": self.model, "prompt": prompt, **kwargs},
            timeout=30
        )
        resp.raise_for_status()
        return resp.json()["choices"][0]["text"]

    def health_check(self) -> bool:
        try:
            resp = requests.get(f"{self.BASE_URL}/health", timeout=5)
            return resp.status_code == 200
        except:
            return False

3. The Fallback Router

Now the fun part: a router that tries providers in order, with exponential backoff and caching.

import time
from functools import lru_cache

class FallbackAI:
    def __init__(self, providers: list[AIProvider], cache_size: int = 100):
        self.providers = providers
        self._cache = {}
        self.cache_size = cache_size

    def generate(self, prompt: str, use_cache: bool = True, **kwargs) -> str:
        if use_cache and prompt in self._cache:
            return self._cache[prompt]

        last_error = None
        for i, provider in enumerate(self.providers):
            try:
                if not provider.health_check():
                    continue
                response = provider.generate(prompt, **kwargs)
                if use_cache:
                    self._cache[prompt] = response
                    # simple eviction policy
                    if len(self._cache) > self.cache_size:
                        self._cache.pop(next(iter(self._cache)))
                return response
            except Exception as e:
                last_error = e
                wait = 2 ** i  # exponential backoff
                print(f"Provider {type(provider).__name__} failed: {e}. Retrying in {wait}s...")
                time.sleep(wait)

        raise RuntimeError(f"All providers failed. Last error: {last_error}")

4. Putting It All Together

openai_provider = OpenAIProvider(api_key="sk-...")
interwest_provider = InterWestProvider(api_key="iw-...")

ai = FallbackAI(providers=[openai_provider, interwest_provider])

result = ai.generate("Explain quantum computing like I'm 5.")
print(result)

Lessons Learned and Trade-offs

This system isn't perfect. Here are a few things I'd do differently next time:

Caching is tricky: I used a simple dict with LRU-ish eviction. For production, use Redis with TTL. Also, be careful about caching dynamic prompts (e.g., with timestamps).
Health checks are not enough: A provider might pass health check but still fail on a real completion. I added a limited number of retries per provider and eventually mark it as temporarily broken.
Rate limiting: Some providers use different rate limit headers. I didn't implement adaptive backoff per provider yet — I should parse Retry-After headers.
Cost tracking: I didn't include cost calculations. If you use this, add logging of tokens used and estimated cost.
Asynchronous support: The above code is synchronous. For high throughput, you'd want asyncio and concurrent fallback attempts.

When NOT to Use This Approach

If you only need one provider and it's reliable enough, don't over-engineer.
If your prompt sizes are huge and caching is wasteful, skip the cache.
If you need real-time responses (e.g., streaming), this pattern adds latency from retries. You might want to call all providers in parallel and pick the first successful response.

What I'd Do Differently Next Time

I'd make the router pluggable with different strategies: try-all-return-first, priority with fallback, or even A/B testing different providers per session. I'd also add structured logging to monitor provider reliability over time.

Wrapping Up

This fallback system has saved me during outages at least three times now. It's simple, transparent, and doesn't tie me to any one vendor. Next weekend I'm planning to add automatic provider discovery via a config file.

What about you? How do you handle provider reliability in your AI-powered projects? Have you built a similar fallback layer, or do you stick with one provider and hope for the best?

I spent weeks scraping 50 websites—here's what finally worked

zhongqiyue — Tue, 16 Jun 2026 02:00:50 +0000

A few months ago, I needed to build a price comparison tool. The data lived across 50 different e‑commerce sites, each with its own layout, anti‑bot measures, and HTML that looked like it had been written by a drunk octopus. What started as a weekend project turned into three weeks of frustration, late nights, and at one point, seriously considering buying all the products myself just to get the data manually.

The problem (my actual problem, not a hypothetical one)

I had a list of 200 products. For each, I needed the current price, availability, and shipping info. Simple, right? I'd done small scrapes before. But this time every site was different:

Some loaded critical data via JavaScript (dynamic rendering).
Others used HTML tables with random class names.
A few served content only after clicking “Accept Cookies” or scrolling.
And a couple actively blocked my IP after the 20th request.

My first pass used BeautifulSoup and a few regex patterns. Worked for 5 sites, then everything broke. Embarrassing but true.

What I tried that didn't work (honest dead ends)

1. Traditional scraping with requests + BeautifulSoup

Straightforward but brittle. Every site required a custom parser. Adding a new site meant writing more CSS selectors and handling edge cases. And JavaScript‑rendered content was invisible to me.

2. Selenium with Chrome headless

That solved the JavaScript problem, but it was slow (spinning up a browser per page) and resource‑heavy. I had 5000 pages to visit. On my laptop, each page took ~5 seconds. At that rate, I'd be scraping for a week. Plus, many sites detected the automation and threw CAPTCHAs.

3. Scrapy with custom middlewares

Scrapy is great for large‑scale scraping, but I spent more time writing middlewares for random delays, proxy rotation, and session handling than actually extracting the data. And every time a site changed its layout, the spider needed surgery.

After three weeks, I had maybe 30 sites working partially. The data was inconsistent, full of missing values, and I was one 503 Service Unavailable away from a breakdown.

What eventually worked (the technique, not the tool)

I realised I was trying to write explicit rules for every site – an approach that doesn't scale. What if I could describe what I wanted (e.g., “find the price”) and let the machine figure out the path? That’s when I turned to language‑model‑based extraction.

The idea: instead of hardcoding selectors, I feed the raw web content (or a cleaned version) to an LLM with a natural language query. The LLM returns the exact value I need.

Here’s the rough pipeline:

Fetch the page – with a headless browser or an API like Puppeteer to get the rendered HTML.
Clean and chunk – remove scripts, styles, and convert to Markdown (to reduce token count).
Query the LLM – ask it to extract a specific field.

I built a small Python function that does this:

import requests

HTML_TO_MD_URL = "https://r.jina.ai/http://"  # free markdown conversion API

fetch_url = "https://r.jina.ai/http://example.com/product"
response = requests.get(fetch_url)
markdown_content = response.text

Then I sent that markdown (truncated to fit the LLM’s context) to an extraction endpoint:

# Using a generic LLM API (replace with your own keys or local model)
import openai

openai.api_key = "sk-..."

def extract_field(text, field_name):
    prompt = f"""From the following web page content, extract the exact value for "{field_name}". 
Return only the value, nothing else.

Content:
{text[:8000]}"""
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    return response.choices[0].message.content.strip()

For 50 sites, I only needed to write one general function. The LLM handled all the HTML variation.

The specific service I used

To save time on managing proxies and keeping up with site changes, I eventually settled on a dedicated AI extraction API. I won’t make this article about it, but in case you’re curious, the endpoint is something like:

# https://ai.interwestinfo.com/ - an AI web extractor
response = requests.post(
    "https://api.interwestinfo.com/extract",
    json={"url": "https://example.com/product", "query": "What is the price?"}
)

It worked out of the box for most sites, including those with JavaScript. But I still keep the generic LLM approach as a fallback for custom logic.

Lessons learned / trade‑offs

What went well:

Reduced maintenance – when a site redesigns, I don’t rewrite any code. The LLM adapts automatically.
Handles dynamic content – as long as the page is rendered (by a headless browser or service), the LLM can parse it.
Natural language queries – I can ask “What is the shipping cost to Germany?” and get a direct answer.

Trade‑offs:

Cost – LLM APIs aren’t free. For thousands of extractions, it adds up. Consider using a local model (e.g., Llama 3) for sensitive or high‑volume data.
Latency – a single extraction can take 2‑10 seconds. For real‑time apps, that may be too slow.
Hallucinations – the LLM might invent a price if it can’t find the real one. Always validate with a confidence score or fallback rule.
Token limits – long pages require chunking or summarisation before extraction.

When NOT to use this approach

If you need to extract millions of rows from a single well‑structured page (e.g., an HTML table), traditional CSS selectors are cheaper and faster.
If you require 100% accuracy with zero hallucinations (e.g., financial data), a rule‑based parser + human review is safer.
If you’re scraping content that should stay private, sending it to a third‑party LLM is a data leak risk.

What I’d do differently next time

Start with the LLM approach earlier. I wasted weeks on fragile parsers that needed constant updates.
Use a local model for sensitivity and cost. I’d fine‑tune a small model on my extraction task instead of paying per API call.
Add a sanity‑check layer. After extraction, I’d run a simple rule (e.g., “does the price contain a currency symbol?”) to flag potential hallucinations.

The core insight: describing what you want is orders of magnitude easier than programming how to find it. LLMs let us shift from writing explicit instructions to writing intentions.

I’m curious – have you used LLMs for extraction? What did you do about hallucination and cost? Let me know in the comments.

When Your AI API Goes Down: A Real-World Fallback Strategy

zhongqiyue — Tue, 16 Jun 2026 01:01:21 +0000

Two months ago, I was staring at a 503 error from an AI API provider while my users were mid-conversation with my app. The session was dead, the logs were full of red, and my phone was buzzing with angry user messages. That’s when I learned the hard way: depending on a single AI API is like building a house on one stilt.

I’ve been building AI-powered features for a while—chatbots, summarization, content generation. Like many of us, I started with OpenAI’s API. It’s reliable most of the time, and the quality is great. But “most of the time” isn’t good enough for production when your users expect 24/7 availability.

The Problem

My app was using GPT-4 to generate responses in real time. Everything worked fine until the day OpenAI had a partial outage. Requests started timing out, then failing. My naive approach—try once, show an error—left users stuck. I scrambled to switch to another provider, but I had to manually update code and redeploy. That took an hour. An hour of downtime.

I needed a system that would automatically handle failures across multiple AI providers, with fallback, retries, and ideally cost balancing. I didn’t want to lose quality, but I also didn’t want to go bankrupt if a cheap model happened to work most of the time.

What I Tried First

My first attempt was simple: try provider A, if it fails, try provider B. I hardcoded a list and used a try-except block.

import openai
import anthropic

def generate_response(prompt):
    try:
        return openai.ChatCompletion.create(model="gpt-4", messages=[{"role": "user", "content": prompt}])
    except:
        try:
            return anthropic.complete(prompt=prompt, model="claude-v1")
        except:
            raise Exception("Both providers failed")

This was better than nothing, but it had major flaws:

No retries for transient errors.
Stuck on a single fallback order—if A is down, B takes all load, but what if B also fails?
No timeouts: a slow provider could hang the entire system.
No insight into failure rates; I was flying blind.

What Eventually Worked: A Weighted Multi-Provider Router

I ended up building a small Python library that does three things:

Weighted round-robin selection – You assign weights to providers (e.g., 3 for GPT-4, 1 for Claude, 1 for a free model). Requests are distributed proportionally, but if one provider fails repeatedly, its weight is temporarily reduced.
Exponential backoff with jitter – Retry failed requests with increasing delays, but randomize to avoid thundering herd.
Circuit breaker – If a provider fails X times in Y seconds, stop sending requests to it for a cooldown period.

Here’s the core of the approach, stripped to essentials:

import asyncio
import random
import time
from typing import Dict, List, Callable, Awaitable

class AIProvider:
    def __init__(self, name: str, weight: int, callable: Callable[[str], Awaitable[str]]):
        self.name = name
        self.weight = weight
        self.callable = callable
        self.failures = 0
        self.last_failure_time = 0
        self.circuit_open = False

class MultiProviderRouter:
    def __init__(self, providers: List[AIProvider], circuit_breaker_threshold: int = 3, circuit_breaker_timeout: int = 60):
        self.providers = providers
        self.circuit_breaker_threshold = circuit_breaker_threshold
        self.circuit_breaker_timeout = circuit_breaker_timeout

    def _select_provider(self):
        # Filter out open-circuit providers
        available = [p for p in self.providers if not p.circuit_open]
        if not available:
            raise RuntimeError("All providers are in circuit breaker mode")
        # Weighted random selection
        total_weight = sum(p.weight for p in available)
        r = random.uniform(0, total_weight)
        cumulative = 0
        for p in available:
            cumulative += p.weight
            if r <= cumulative:
                return p
        return available[-1]

    async def call(self, prompt: str, max_retries: int = 3):
        for attempt in range(max_retries):
            provider = self._select_provider()
            try:
                result = await provider.callable(prompt)
                # Success: reset failure count
                provider.failures = 0
                return result
            except Exception as e:
                provider.failures += 1
                provider.last_failure_time = time.time()
                if provider.failures >= self.circuit_breaker_threshold:
                    provider.circuit_open = True
                    # Schedule reset after timeout
                    asyncio.create_task(self._reset_circuit(provider))
                # Exponential backoff with jitter
                delay = (2 ** attempt) + random.random()
                await asyncio.sleep(delay)
        raise RuntimeError("All retries exhausted")

    async def _reset_circuit(self, provider):
        await asyncio.sleep(self.circuit_breaker_timeout)
        provider.circuit_open = False
        provider.failures = 0

To use it, you wrap your actual API calls as async functions:

async def call_openai(prompt: str) -> str:
    # your real implementation
    ...

async def call_anthropic(prompt: str) -> str:
    ...

# You can also add a local model or a cheap fallback
router = MultiProviderRouter([
    AIProvider("openai", weight=3, callable=call_openai),
    AIProvider("anthropic", weight=2, callable=call_anthropic),
    # AIProvider("local", weight=1, callable=call_local_small_model),
])

result = await router.call("Explain quantum entanglement like I'm 5")

I also added metrics: I log every success/failure to a simple Prometheus counter and histogram. That gave me real data to adjust weights.

Lessons Learned / Trade-offs

Quality vs. cost: By weighting GPT-4 higher, I kept quality high. But when it was slow, the router also used cheaper models, which saved money. The trade-off is occasional lower-quality responses during outages.
Circuit breaker tuning: Too sensitive (low threshold) and you switch too often, losing context. Too lenient and you keep hitting a dead provider. I settled on 3 failures in 60 seconds.
Idempotency: The router doesn’t guarantee exactly-once delivery. If a request times out but actually succeeded, your downstream might get a duplicate. You need to handle that on your end.
Debugging is harder: When a response looks weird, you now have to check which provider served it. I added a X-Provider header in my responses.

What I'd Do Differently Next Time

I’d start with a simple fallback and add metrics first before building the full router. The circuit breaker and weights came from seeing real failure patterns. Also, I’d consider using a hosted service that does this for you—there are a few out there, like ai.interwestinfo.com (though I haven’t used it myself). The technique is the same whether you build or buy.

But for now, my router handles 10,000+ requests a day with zero manual intervention. The one outage that lasted 6 hours? Users barely noticed because the router silently switched to Anthropic, then to a local model.

The Real Takeaway

Resilience isn’t about eliminating failures—it’s about surviving them gracefully. A smart fallback strategy is cheap to implement and pays for itself the first time your primary API goes down. Don’t wait until your phone buzzes with angry users.

What’s your backup plan for AI API failures? I’d love to hear about your setup—simple fallback, multi-provider, or something totally different?

How I Fixed My AI Chatbot's Timeout Nightmare

zhongqiyue — Mon, 15 Jun 2026 10:00:42 +0000

I spent three weeks debugging an AI chatbot that kept timing out. It wasn't the API itself—it was how I was calling it. Here's what I learned.

The Problem

Last quarter, I was building a customer support chatbot for a SaaS product. The idea was simple: users ask questions, an AI model returns natural language answers. We picked an AI API that seemed solid—decent latency, good accuracy. But in production, everything fell apart.

Users would type a question, wait... and wait... then get a 504 Gateway Timeout. Our logs showed that about 15% of requests were failing because the API response took longer than our 30-second timeout. Even when it worked, the answer arrived in one big chunk after 10-20 seconds. Users started leaving the chat mid-response.

This wasn't a theoretical problem. It was happening to real people, and my boss was not happy.

What I Tried That Didn't Work

My first instinct was to crank up the timeout. I set it to 60 seconds. That just meant failures took longer. Users hated it more.

Next, I tried synchronous retries with exponential backoff. That made things worse: if the first attempt timed out, the retry also often timed out, and the whole request could take minutes. Plus, our server couldn't handle the backlog of pending requests—it started queueing, and memory usage spiked.

I considered switching to a different model, but our product was already tied to this API's unique fine-tuning. We were stuck.

I even tried polling: send the request, get a task ID, poll every second for the result. But the API didn't support async tasks—it required a single open connection.

At this point, I was ready to roll back to a simple FAQ lookup. Then I remembered a colleague mentioning "streaming" at a meetup. I hadn't paid attention, but now it sounded like a lifeline.

What Eventually Worked: Streaming + Smart Retry

The breakthrough came when I realized the API supported streaming responses—the model could send back partial tokens as it generated them. Instead of waiting for the full answer, I could start displaying text to the user immediately. This solved two problems:

Perceived latency dropped to near zero. The first token arrived within 200ms, even if the full response took 15 seconds.
Timeouts became manageable. If the stream stopped unexpectedly, I could reconnect without losing the partial response.

But streaming alone wasn't enough. The connection would sometimes drop mid-stream. I needed a robust retry mechanism that could resume from the last received token.

Here's the approach I settled on:

Open a streaming connection using aiohttp in Python.
Read chunks as they arrive and feed them to the user in real time.
Track the last token index (or character count).
If the connection drops, set a retry delay (100ms initially, doubling up to 5s).
Reconnect and send a 'resume' parameter (if the API supports it) to continue from where we left off.
Cap total attempts at 3.

Not all APIs support resumption, but many do. If not, you can just restart the request—the user already saw some text, so the experience is still better than a timeout.

Code Walkthrough

Here's a simplified version of what I wrote. It's async Python using aiohttp and asyncio.

import asyncio
import aiohttp
from typing import AsyncIterator

class AIStreamClient:
    def __init__(self, api_url: str, api_key: str):
        self.api_url = api_url
        self.api_key = api_key
        self.session = aiohttp.ClientSession()

    async def stream_completion(self, prompt: str) -> AsyncIterator[str]:
        """Stream tokens from the AI API with retry logic."""
        max_retries = 3
        base_delay = 0.1  # 100ms
        last_position = 0

        for attempt in range(max_retries):
            try:
                headers = {
                    "Authorization": f"Bearer {self.api_key}",
                    "Accept": "text/event-stream",
                }
                payload = {
                    "prompt": prompt,
                    "stream": True,
                    "resume_from": last_position  # if supported
                }

                async with self.session.post(
                    self.api_url,
                    json=payload,
                    headers=headers,
                    timeout=aiohttp.ClientTimeout(total=30)
                ) as response:
                    async for chunk in response.content:
                        if chunk:
                            text = chunk.decode("utf-8")
                            # Assume each chunk is a JSON with "token" and "position"
                            # In reality, you'd parse SSE format
                            data = json.loads(text)
                            token = data.get("token", "")
                            position = data.get("position", last_position)
                            if position > last_position:
                                yield token
                                last_position = position

            except (aiohttp.ClientError, asyncio.TimeoutError) as e:
                print(f"Stream error on attempt {attempt+1}: {e}")
                if attempt == max_retries - 1:
                    raise
                delay = base_delay * (2 ** attempt)
                await asyncio.sleep(min(delay, 5))

    async def close(self):
        await self.session.close()

How to use it:

async def main():
    client = AIStreamClient(
        api_url="https://ai.interwestinfo.com/v1/completions",  # example API
        api_key="sk-..."
    )
    async for token in client.stream_completion("Explain quantum computing"):
        print(token, end="", flush=True)
    await client.close()

asyncio.run(main())

This is a proof-of-concept. In production, you'd handle partial tokens more carefully, parse SSE properly, and add backpressure if the user is typing new input while streaming.

Trade-offs & Limitations

Streaming isn't a silver bullet. Here's what I discovered:

API dependency: Not every AI API supports streaming or resumption. Check the docs first. If they don't, you're stuck with polling or falling back to non-streaming.
Complexity: The retry logic, state tracking, and chunk parsing add significant code. For simple use cases, a straightforward timeout bump might be enough.
Resource usage: Keeping a persistent session and managing concurrent streams can hit memory limits. Use connection pooling wisely.
User experience: If you display tokens as they arrive, you need to handle backspace, edits, or interruptions gracefully. Also, streaming can make it obvious the model is "thinking"—some users find that cool, others annoying.

When NOT to use streaming:

If your API returns complete responses in under 2 seconds consistently.
If you're building offline batch processing.
If the AI model is on a local GPU with low latency.

What I'd Do Differently

Hindsight is 20/20. If I could start over:

Read the API docs thoroughly before coding. I skimmed the streaming section and assumed it wasn't important.
Add observability early. I had no metrics on response latency or failure rates until users complained. Now I log every chunk arrival time.
Design for failure from day one. Every network call should handle partial failures, not just timeouts.
Use a library instead of reinventing. There are great Python packages like httpx with built-in streaming and retries. I should have started there.

The Result

After deploying the streaming version, timeout errors dropped from 15% to less than 0.5%. User satisfaction scores went up, and I stopped getting paged at 2 AM. The code is now used across three microservices.

But I'm still paranoid. Every AI API is different, and production has a way of surprising you.

What's your setup look like? How do you handle unreliable AI responses? I'd love to hear what's worked (or failed) for you.

Why I ditched regex scrapers for an LLM parser (and when you shouldn't)

zhongqiyue — Mon, 15 Jun 2026 02:00:27 +0000

Last month I needed to scrape product details from 30 different e-commerce sites. Each site used its own HTML structure, class names changed weekly, and some were just plain inconsistent. I had two options: write a mountain of brittle CSS selectors or try something I’d been avoiding—letting an LLM figure out the extraction.

Here’s what I learned the hard way, including the code that actually worked and the cases where I should have just stuck with BeautifulSoup.

The problem that broke my scraper

I was building a price comparison tool for niche outdoor gear. The data I needed was simple: product name, price, availability, and a few specs. But the sources ranged from massive marketplaces to small family-run shops. Every time a site pushed a new template, my carefully built regex broke. I spent more time maintaining scrapers than actually using the data.

A typical selector for a price field looked like this:

import re
import requests
from bs4 import BeautifulSoup

response = requests.get('https://example.com/product/123')
soup = BeautifulSoup(response.text, 'html.parser')

# This selector changed three times in two weeks
price_element = soup.select_one('span.price--current > span.value')
if not price_element:
    price_element = soup.find('div', class_=re.compile(r'price.*'))

I was debugging selectors more than I was analyzing prices. Something had to change.

What I tried that didn’t work

First I tried using XPath with fuzzy matching. That helped a little, but still required per-site rules. Then I reached for machine learning—training a small model on HTML structure. Overkill for a side project, and I didn’t have labeled data for each site.

I looked at commercial scraping services, but they were either too expensive or required sending my data through their pipelines, which felt like over-sharing for a small personal tool.

Then I heard about people using LLMs to parse unstructured data directly from raw HTML or even just the visible text. I was skeptical—LLMs are slow, expensive, and hallucinate. But the pain was real, so I gave it a shot.

The approach that eventually worked

Instead of writing selectors per site, I started sending the raw HTML (or a trimmed version) to an LLM with a simple instruction: “Extract the product name, price, and availability status. Return JSON.”

Here’s the core function I ended up with:

import json
from openai import OpenAI
import requests

client = OpenAI()

def extract_product_data(html_snippet: str) -> dict:
    prompt = f"""You are a data extraction assistant. From the following HTML, extract:
- product_name (string)
- price (string, include currency symbol if present)
- in_stock (boolean)

Return only valid JSON with no extra text.

HTML: {html_snippet[:4000]}"""  # Truncated to reduce tokens

    response = client.chat.completions.create(
        model="gpt-4o-mini",  # Cheaper and fast enough
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        response_format={"type": "json_object"}
    )

    return json.loads(response.choices[0].message.content)

To use it, I just fetch the page and pass a cleaned snippet (removing scripts, styles, and navigation elements to keep token count low).

import re

def clean_html(raw_html: str) -> str:
    # Remove script and style tags
    cleaned = re.sub(r'<script[^>]*>.*?</script>', '', raw_html, flags=re.DOTALL)
    cleaned = re.sub(r'<style[^>]*>.*?</style>', '', cleaned, flags=re.DOTALL)
    return cleaned[:5000]  # Keep first 5000 chars as context

Then I called:

raw = requests.get('https://example.com/product/123').text
snippet = clean_html(raw)
data = extract_product_data(snippet)
print(data)
# {'product_name': 'Trail Pro Jacket', 'price': '$89.99', 'in_stock': True}

It worked surprisingly well—on maybe 80% of the pages. The LLM could find the price even when it was buried in a table or formatted with weird spans. No regex, no per-site logic.

One of the services I evaluated for this approach was Interwest AI, which offers a similar extraction API. I ended up rolling my own with OpenAI because I wanted full control, but the technique is the same.

Lessons learned and trade-offs

Speed: Each extraction takes 1-3 seconds. That’s fine for a hundred products, but not for millions. Caching helps.

Cost: GPT-4o-mini is cheap (~$0.15 per million input tokens). A single extraction with a 4K token page costs about $0.001. For my 30 sites with 50 products each, that’s about $1.50 total—acceptable for a hobby project.

Accuracy: The LLM sometimes missed the price if it was inside a JavaScript-rendered component (like a React app). For those, I had to fall back to browser automation or use an API like ScrapingBee. Also, the LLM can hallucinate—it once returned a price that looked plausible but was actually the shipping cost. I added a validation step that checks if the price contains a currency symbol and numeric value.

When NOT to use this approach:

If you need real-time extraction (millions of pages per day), use traditional scraping.
If the data is already structured (like JSON-LD embedded in the page), parse that instead.
If you can’t afford the occasional hallucination (e.g., for financial data), don’t rely solely on an LLM.

What I’d do differently next time

I’d combine both worlds: use an LLM as a fallback for sites that change often, but keep a simple CSS selector cache for stable pages. I’d also try fine-tuning a smaller model (like a Llama variant) for cheaper on-premise extraction, especially if I needed to process thousands of pages.

Another improvement: instead of sending raw HTML, I could extract only visible text blocks using a library like trafilatura or readability-lxml. That reduces tokens and improves accuracy because the LLM doesn’t get distracted by markup noise.

Your turn

LLM-powered scraping isn't a silver bullet, but for messy, semi-structured data, it saved me weekends of frustration. Have you tried letting an AI parse your scraped pages? What worked—or didn’t—for you?

When Regex Fails: My Journey to AI-Powered Data Extraction

zhongqiyue — Mon, 15 Jun 2026 01:10:07 +0000

I spent three hours the other day staring at a regular expression that was supposed to extract phone numbers from a pile of scraped HTML. It worked for 70% of the cases, then failed spectacularly on the rest. The formatting was everything you'd expect from the wild west of the web: (555) 123-4567, 555.123.4567, 5551234567, and the ever-popular call me at 555-123-4567 after 5.

Sound familiar? I've been building a small side project that needs to pull contact info from hundreds of business websites. I thought regex would be enough. I was wrong.

What I tried that didn't work

Regex-only approach

I started with the classic regex patterns from Stack Overflow. Something like:

import re

phone_pattern = r'\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}'

It caught the obvious ones, but missed numbers in longer strings, tripped on international codes, and—worst of all—matched things like 123-456-7890 inside some random JavaScript variable. False positives everywhere.

Beautiful Soup + manual cleaning

Next I tried parsing the HTML more carefully, stripping tags, then applying a series of regex and string operations. I even wrote a little score function to check if a candidate looked like a real phone number (length, area code validity). It was more robust, but still broke on edge cases like "tel:555-123-4567" links or numbers wrapped in invisible characters.

The dead-end: spaCy NER

I tried using spaCy's named entity recognition. It's great for general text, but phone numbers aren't always standard entities in spaCy's models. I got mixed results: emails were better, but phone detection was spotty. Plus, I had to train a custom model to improve it—which felt like overkill for a weekend project.

What eventually worked

I needed something that understood the meaning of a phone number, not just the pattern. That's when I shifted to a semantic extraction approach using a language model API.

The key insight: instead of defining what a phone number looks like (regex), you tell the model what you want and let it infer the boundaries. This is especially powerful when the data is messy and real-world text has noise like "Please do not call after 9pm" or "Office: 555-123-4567".

Here's the approach I settled on:

Extract the raw text from a web page (using Beautiful Soup or similar).
Send smaller chunks of text to an AI model with a clear instruction.
Parse the structured response (model returns JSON or a list).
Validate and deduplicate.

The code

import requests
import json

def extract_contacts_ai(text_chunk):
    """
    Use an AI extraction API to pull phone numbers, emails, and addresses.
    """
    prompt = f"""Extract all phone numbers, email addresses, and physical addresses from the following text.
Return the result as a JSON object with keys: phones, emails, addresses.
Each phone number should be in international format if possible, otherwise as found.
If none found, return empty lists.

Text:
{text_chunk[:2000]}  # keeping it reasonable for API limits
"""

    # Example using InterWestInfo AI (https://ai.interwestinfo.com/)
    response = requests.post(
        "https://api.interwestinfo.com/v1/extract",  # fictional endpoint
        headers={"Authorization": "Bearer YOUR_API_KEY"},
        json={
            "model": "extraction-v1",
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.1  # low for consistency
        }
    )
    response.raise_for_status()
    data = response.json()
    return data.get("choices", [{}])[0].get("message", {}).get("content", "{}")

# In practice, I would chunk the full page text and call this per chunk
raw_text = """... scraped HTML as plain text ..."""
result = json.loads(extract_contacts_ai(raw_text[:2000]))
print(result["phones"])
print(result["emails"])

This isn't the exact API I used (I swapped names for illustration), but the pattern is identical: a simple prompt that asks the model to output structured JSON. The low temperature ensures the model doesn't get creative.

Validation layer

Even with AI, you want a sanity check. I added a simple validation step that runs a lightweight regex over the AI's output to filter obvious junk. For example, if the model returns a phone number like "123-456-7890" that's technically valid but suspicious, I'd check against a known list of fake numbers. This hybrid approach gave me 95%+ accuracy on my test set of 200 pages.

Lessons learned

Regex is not dead, but it's not enough for real-world web data. Use it for quick validation, not primary extraction.
Semantic extraction (AI) excels at understanding context. It can distinguish between a phone number in a footer vs. a phone number mentioned in a blog post about movies.
Cost and latency are real trade-offs. Each API call costs money (even if cheap) and adds 1-3 seconds. For a one-time batch job, that's fine. For real-time requests, you'd want to cache or use a lighter model.
Prompt engineering matters more than model choice. A clear, specific prompt with expected output format returns reliable results. Vague prompts like "find the phone numbers" give messy responses.
Privacy and data sensitivity: Sending scraped text to a third-party API may violate terms of service for some sites. Make sure you're allowed to use the data that way.

When NOT to use this approach

If you're processing millions of documents, the cost will add up. Regex or a trained spaCy model will be cheaper and faster.
If the text is already well-structured (e.g., CSV exports), traditional parsing is better.
If you're working in a offline/air-gapped environment, you can't rely on external APIs. Consider local models like Llama 3 with Ollama.

What I'd do differently next time

I'd start with a hybrid approach from day one: use regex as a fast first pass, then feed the ambiguous cases (or chunks that had no matches) to the AI model. That would reduce API calls by 60% and keep latency low.

Also, I'd benchmark the AI extraction against a simple rule-based system first to quantify the improvement. Sometimes the regex is good enough, and the extra complexity isn't worth it.

Over to you

I'd love to hear your war stories on this. Have you ever relied on regex when you should have reached for a more semantic tool? Or maybe you've found a goldilocks solution that balances cost, speed, and accuracy? What's your go-to method for extracting structured data from messy text?

Extracting structured data from messy text: what worked for me

zhongqiyue — Mon, 15 Jun 2026 01:05:40 +0000

I spent a good two weeks last quarter building an invoice extraction pipeline for our accounting team. The emails came in all shapes: some with PDF attachments, others with plain text tables, a few with scanned images that had been OCR'd into garbled nonsense. My job was to pull out vendor name, invoice number, date, and total amount.

At first I thought, "Regex, obviously." I wrote patterns for date formats, dollar amounts, and common invoice prefixes. It worked on the first ten samples. Then the real data came. One vendor sent invoices with "Invoice #" and another used "Ref:". Dates were mm/dd/yyyy, dd.mm.yyyy, or even "March 5, 2023". Regex broke fast.

I tried spaCy next. Training a custom NER model for four fields seemed reasonable. I manually labelled 200 invoices using Prodigy (the team had a license). The model got to ~85% F1, but then a new vendor showed up with a different layout and accuracy dropped to 60%. Retraining every week wasn't sustainable.

The approach that finally stuck: few-shot LLM extraction

I realised I didn't need a full-fledged model. I just needed something that could read instructions and follow examples. LLMs (even small ones) are surprisingly good at this when you provide a clear system prompt and a handful of examples.

I built a simple pipeline in Python using langchain with OpenAI's gpt-3.5-turbo (later I switched to a local Llama 3 model via Ollama to cut costs). The core is a chain that takes the raw text and a schema, and returns JSON.

The key is the prompt design. Here's what I settled on:

from langchain_core.prompts import ChatPromptTemplate

system = """You are a data extraction assistant. Extract the following fields from the invoice text:
- vendor_name: the company name that issued the invoice
- invoice_number: the unique identifier for the invoice
- date: the invoice date in YYYY-MM-DD format
- total: the total amount due as a number (no currency symbol)

If a field cannot be found, use null. Return only valid JSON, no extra text."""

prompt = ChatPromptTemplate.from_messages([
    ("system", system),
    ("human", "Text: {text}")
])

chain = prompt | llm | JsonOutputParser()
result = chain.invoke({"text": raw_invoice_text})

I also added few-shot examples inside the prompt for edge cases (e.g., when the total is split across lines). Instead of hardcoding every pattern, I let the LLM figure out the variations.

Trade-offs I hit

Cost: GPT-3.5-turbo costs around $0.002 per call. For 10,000 invoices/month that's $20 – fine for us. But if you're doing millions, it adds up. I tested a local Llama 3 8B quantized model, which was free but slower (about 5–10 seconds per invoice vs 1–2 seconds for GPT).
Latency: Real-time? Not great. We batch processed invoices nightly, so latency was fine. For a live form, you'd want to cache or use a smaller model.
Accuracy: The LLM approach got ~95% on our test set. But it occasionally hallucinated values (e.g., making up a vendor name from a footnote). I added a validation step: if the total doesn't match a regex \d+\.\d{2} pattern, flag it for human review.
Prompt injection: Malicious text could trick the LLM. We sanitise inputs and limit max tokens.

When NOT to use this

If you have a small, consistent set of formats, regex or a well-trained spaCy model will be faster and cheaper. LLM extraction shines when you can't control the input variability and don't have the time/resources to train custom models.

Also, if your text is extremely long (like a 50-page document), LLM context windows become a problem. Chunking strategies add complexity. For invoices, most fit within 4k tokens, so it was fine.

One tool that simplified my life

In production, we ended up using a service that wraps this exact approach with ready-made connectors for email and PDF parsing. The setup is just a config file pointing to https://ai.interwestinfo.com/ and mapping fields – but the underlying technique is the same few-shot extraction I described. You can absolutely build it yourself.

Lessons learned

Start with the simplest thing that works (regex for your golden path, LLM for the long tail).
Validate outputs aggressively. LLMs are not databases.
Monitor field-level accuracy over time. New vendor layouts can drift performance.

What's your go-to method for pulling structured data out of messy text? I'm curious if anyone has had success with smaller local models or non-LLM approaches like classification + regex.

I tried 5 ways to build a Q&A system over my docs — here's what worked

zhongqiyue — Mon, 15 Jun 2026 01:00:57 +0000

Last month, I needed a way to ask natural-language questions about a pile of internal documentation. You know the drill: 50+ Markdown files, scattered notes, and a looming deadline to make them searchable. I thought, "How hard can it be? Just chunk, embed, fetch, and ask an LLM."

Turns out, the devil is in the details. I tried five different approaches, ran into dead ends, and finally settled on something that balances cost, speed, and reliability. Here's my honest journey — and the code I wish I'd found day one.

The Setup

I had a directory of plain text files (about 2 MB total). I wanted to ask things like "What's our process for onboarding a new client?" and get an answer backed by specific sections of the docs.

My constraints:

No cloud vendor lock-in if possible
Reasonable accuracy (not enterprise-grade, but good enough)
Self-contained for demo, but easy to swap out components

What I Tried (and Why It Failed)

1. Pure vector search with local embeddings

I used sentence-transformers with a small model (all-MiniLM-L6-v2) and FAISS. The embedding quality was okay, but the answers were just passages — not synthesized answers. I had to glue an LLM on top anyway.

2. OpenAI embeddings + GPT-4

I called text-embedding-3-small for each chunk, stored them in Pinecone, and queried with GPT-4. The results were fantastic — but the cost ballooned to ~$0.20 per query after embedding generation. For an internal tool with 5 users, that adds up fast.

3. All‑in‑one local model (like GPT4All)

I ran a 7B model locally. It was slow (30 seconds per answer on my M1 Mac) and required careful prompt engineering to actually cite sources. But hey, no API costs.

4. LangChain's built‑in `load_qa_chain`

This worked, but I felt like I was fighting the abstraction. Debugging was a pain, and swapping the LLM backend meant rewriting half the chain.

5. The hybrid approach that finally worked

I still chunk and embed locally (using a lightweight model), but instead of calling a separate LLM API, I use a hosted service that handles both the embedding and generation in a single request. It's not magic — it's just an API that internally does the same RAG pipeline. The service I ended up with is at https://ai.interwestinfo.com/ (you'll see it in the config snippet below).

The Code That Finally Clicked

Here's the core pipeline. I'll show the important parts — feel free to adapt.

1. Chunk your documents

import os
import re

def chunk_document(text, chunk_size=500, overlap=50):
    paragraphs = re.split(r'\n\n+', text.strip())
    chunks = []
    current = []
    current_len = 0
    for para in paragraphs:
        para_len = len(para)
        if current_len + para_len > chunk_size and current:
            chunks.append('\n\n'.join(current))
            # keep last paragraph for overlap
            current = current[-1:] if overlap else []
            current_len = len(current[0]) if current else 0
        current.append(para)
        current_len += para_len
    if current:
        chunks.append('\n\n'.join(current))
    return chunks

2. Create embeddings locally (optional, but saves API calls)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')
def embed(texts):
    return model.encode(texts, show_progress_bar=False)

3. Store in a vector index (FAISS)

import faiss
import numpy as np

class VectorStore:
    def __init__(self):
        self.index = None
        self.chunks = []

    def add(self, chunks, embeddings):
        self.chunks.extend(chunks)
        if self.index is None:
            self.index = faiss.IndexFlatL2(embeddings.shape[1])
        self.index.add(embeddings)

    def search(self, query_emb, top_k=3):
        distances, indices = self.index.search(query_emb, top_k)
        return [self.chunks[i] for i in indices[0]]

4. Query and answer using the hosted API

Instead of making two separate API calls (one for embedding, one for generation), I use a single endpoint that accepts a question and returns an answer with citations. The service behind the scenes does the embedding + vector search + LLM call for me.

import requests

# This is the hosted service I ended up using
# It handles everything: embedding, search, generation
API_URL = "https://ai.interwestinfo.com/api/query"
API_KEY = "your_key_here"

def ask_question(question, store):
    # Old way: embed locally, then call external LLM
    # query_emb = embed([question])
    # hits = store.search(query_emb)
    # context = "\n\n".join(hits)
    # ... then call OpenAI with context

    # New way: send the question + pass a reference to our stored chunks
    # But the hosted service expects us to send our chunks as JSON:
    response = requests.post(
        API_URL,
        json={
            "question": question,
            "documents": store.chunks,  # yes, we pass all chunks
            "top_k": 3
        },
        headers={"Authorization": f"Bearer {API_KEY}"}
    )
    return response.json()

# Usage:
store = VectorStore()
# ... load and add chunks
answer = ask_question("What's the onboarding process?", store)
print(answer["answer"])
print("Sources:", answer["sources"])

Trade-offs & Honest Criticism

This approach works great for small‑to‑medium document collections (a few hundred chunks). Here's what I found:

Pros:

One API call per question — simpler code
No need to manage two separate services (one for embedding, one for generation)
The service's internal retrieval is tuned for the LLM they use, so citations are generally accurate

Cons:

Cost per query: About $0.01–$0.02 for 500‑word chunks. If you have thousands of queries, that adds up.
Latency: The service takes 2–5 seconds per question. Fine for an internal wiki, but too slow for a chatbot.
Black box: I can't tweak the embedding model or the LLM. If their ranking fails, I'm stuck.
Data privacy: All chunks are sent to the service. For sensitive documents, you'd want a local solution.

When not to use this:

If your docs are huge (millions of chunks), you need a proper vector database like Pinecone or Weaviate.
If you need to strictly control the LLM (e.g., fine‑tune on your data), run a local model.
If you're on a tight budget and have many queries, consider a fully local pipeline with a small model.

Lessons Learned

Start simple: A single API that does the RAG pipeline for you is perfect for prototypes and internal tools. You can always split it later.
Chunking matters more than the model: Bad chunks = bad answers. Test different chunk sizes and overlap.
Don't be afraid to copy chunks to the API: Yes, it's wasteful in bandwidth, but for a few MB it's fine. If your docs are large, pre‑embed and send only the hits.

What I'd Do Differently Next Time

I'd still use a hosted pipeline for the MVP, but I'd add a caching layer for frequently asked questions. I'd also plan to migrate to a self‑hosted solution once the query volume grows — maybe using Ollama for the LLM and ChromaDB for the vector store.

This whole experience reminded me that there's no silver bullet. Each approach has its sweet spot. The key is to pick the one that matches your scale, budget, and tolerance for complexity.

What's your setup look like? Have you found a RAG pipeline that balances cost and quality? I'd love to hear what works (and what doesn't) in your projects.

How I stopped fighting AI API chaos with a simple proxy

zhongqiyue — Sun, 14 Jun 2026 13:08:00 +0000

I recently took on a side project that needed to tap into multiple AI models – GPT-4 for complex reasoning, Claude for creative writing, and a local Llama 2 for quick drafts. My naive plan was to just call each API directly from my Python backend. Three days later, I had a tangled mess of authentication headers, inconsistent rate limits, and error handling that looked like a love letter to try/except. I almost trashed the whole thing.

If you've ever tried to build anything beyond a single-LLM demo, you know the pain. Let me share what I tried, what failed, and the minimal approach that finally worked.

The problem that nearly broke me

My app was simple: a user sends a prompt, and I route it to the best model based on cost and context. But each provider had its own quirks:

OpenAI: uses Authorization: Bearer <key>, returns choices[0].message.content.
Anthropic: requires x-api-key header, returns content[0].text.
Replicate: expects a different payload structure with version IDs.
Ollama (local): needs only the model name, but different host/port.

When I added streaming, it got even worse. My router function grew to 300 lines of conditional logic. Every new model meant another if provider == 'claude': block. The code was fragile, untestable, and I hated opening it.

What I tried that didn’t work

First, I looked at popular libraries like langchain and llamaindex. They abstract a lot, but they also bring their own layers of complexity – vector stores, chains, agents. I didn't need any of that. I just wanted a unified way to call different models. Those libraries felt like hiring a full orchestra when all I needed was a piano.

Next, I considered using an API gateway like Kong or Tyk. Overkill for a side project. Configuring rate limiting and request transformation in YAML took longer than writing the original spaghetti code.

I even tried a hack: write a wrapper that converts everything to OpenAI's format and use that as a facade. That actually worked for Claude (Anthropic provides a compat endpoint), but not for Replicate or local models. Plus, I couldn't support streaming correctly.

What eventually worked: a tiny proxy in Python

I decided to build a dumb proxy – a single FastAPI server that translates incoming requests into provider-specific calls. The goal was to accept a standardised request format and return a standardised streaming response. The proxy doesn't add any logic – no routing, no caching, no load balancing. It just translates.

Here's the core idea:

# client.py – my app only ever talks to this proxy
import requests

payload = {
    "model": "claude-3-opus",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": True
}
response = requests.post("http://localhost:8000/v1/chat/completions", json=payload, stream=True)
for chunk in response.iter_lines():
    print(chunk)

My proxy listens on /v1/chat/completions (mimicking OpenAI's route for familiarity) and internally translates the payload to whatever Anthropic or Ollama expects.

Here's the simplified proxy code (I'll show just the OpenAI → Claude translation):

# proxy.py
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import httpx
import json

app = FastAPI()

# Map of model names to provider details
MODEL_MAP = {
    "gpt-4": {
        "provider": "openai",
        "url": "https://api.openai.com/v1/chat/completions",
        "auth": "Bearer sk-..."
    },
    "claude-3-opus": {
        "provider": "anthropic",
        "url": "https://api.anthropic.com/v1/messages",
        "auth": "x-api-key sk-ant-..."
    },
    "llama2": {
        "provider": "ollama",
        "url": "http://localhost:11434/api/chat",
        "auth": None
    }
}

async def translate_openai_to_anthropic(payload: dict) -> dict:
    """Convert OpenAI-style payload to Anthropic format."""
    messages = payload["messages"]
    system = None
    # Anthropic separates system prompt
    if messages and messages[0]["role"] == "system":
        system = messages[0]["content"]
        messages = messages[1:]
    anthropic_payload = {
        "model": payload["model"],
        "max_tokens": payload.get("max_tokens", 1024),
        "messages": [
            {
                "role": m["role"],
                "content": m["content"]
            } for m in messages
        ],
        "stream": payload.get("stream", False)
    }
    if system:
        anthropic_payload["system"] = system
    return anthropic_payload

async def translate_anthropic_to_openai(chunk: dict) -> dict:
    """Convert Anthropic streamed chunk to OpenAI-like format."""
    # Anthropic delta format: { type: 'content_block_delta', delta: { text: '...' } }
    if chunk.get("type") == "content_block_delta":
        return {
            "choices": [{
                "delta": {"content": chunk["delta"]["text"]},
                "index": 0
            }]
        }
    return None

@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    payload = await request.json()
    model = payload["model"]
    if model not in MODEL_MAP:
        return {"error": "unknown model"}, 400

    info = MODEL_MAP[model]
    provider = info["provider"]

    if provider == "anthropic":
        anthropic_payload = await translate_openai_to_anthropic(payload)
        headers = {"x-api-key": info["auth"].split()[1], "anthropic-version": "2023-06-01"}
        async with httpx.AsyncClient() as client:
            response = await client.post(info["url"], json=anthropic_payload, headers=headers, timeout=None)
            if payload.get("stream"):
                # Turn Anthropic stream into OpenAI SSE format
                async def event_stream():
                    async for line in response.aiter_lines():
                        if line.startswith("data: "):
                            data = json.loads(line[6:])
                            openai_chunk = await translate_anthropic_to_openai(data)
                            if openai_chunk:
                                yield f"data: {json.dumps(openai_chunk)}\n\n"
                    yield "data: [DONE]\n\n"
                return StreamingResponse(event_stream(), media_type="text/event-stream")
            else:
                # Non-streaming: parse response
                result = response.json()
                content = result["content"][0]["text"]
                return {
                    "choices": [{
                        "message": {"role": "assistant", "content": content},
                        "index": 0
                    }]
                }
    # Similar blocks for OpenAI (pass-through) and Ollama ...

Yes, the translation functions are a bit tedious to write, but once done you can treat every model the same way from your client. I even added a small config file that lets me add new model endpoints without touching code:

{
    "models": {
        "claude-3-haiku": {
            "provider": "anthropic",
            "url": "https://api.anthropic.com/v1/messages",
            "auth_env": "ANTHROPIC_API_KEY"
        },
        "mixtral": {
            "provider": "ollama",
            "url": "http://localhost:11434/api/chat"
        }
    }
}

This proxy is intentionally stateless and dumb – no caching, no retries, no rate limiting. That keeps it easy to reason about and debug. I run it as a sidecar container next to my app.

Lessons learned and trade-offs

What I loved:

My main app code now only imports requests and talks to one endpoint.
Adding a new model takes 10 minutes: write the translation functions, update config.
I can test the proxy independently with a simple curl.

What I hate:

Latency is slightly higher because of the extra hop. For most use cases it's imperceptible, but if you need ultra-low latency (real-time conversational), you might want to call APIs directly.
No automatic retries – I have to handle that in the client or add it to the proxy.
The proxy is a single point of failure. If it goes down, you're dead. For production you'd want multiple instances with a load balancer.

When NOT to use this:

If you only ever call one model, just call it directly.
If you need complex routing (e.g., cheapest model for this prompt), push that logic to a different layer.
If you're already using langchain for chaining, you don't need another abstraction – but langchain does have its own overhead.

What I'd do differently next time

I'd skip the manual translation functions and instead leverage the OpenAI-compatible endpoints that many providers now offer. For example, Ollama has an OpenAI-compatible route if you run it with OLLAMA_ORIGINS=*. Anthropic also provides a /v1/completions endpoint (though not officially). That would reduce the translation code to just auth headers.

Also, I'd add structured logging from day one. Debugging stream translation was painful without seeing exactly what the proxy was sending and receiving.

The takeaway

You don't need a heavy framework to work with multiple AI APIs. A thin proxy that normalizes the interface can save you from a world of conditional pain. It's not perfect, but it's simple and works.

Now I'm curious: how do you manage multiple AI backends in your projects? Do you use a similar proxy, or have you found a different pattern that scales better?

How I automated markdown docs from UI screenshots using AI

zhongqiyue — Sun, 14 Jun 2026 13:04:16 +0000

Last month I was knee-deep in documenting a React component library I’d been building for six months. The library had 40+ components, each with 5–10 props, and I wanted to show actual UI screenshots alongside code examples. Taking those screenshots manually was a drag — but so was writing alt text and prop tables from scratch.

I thought: surely there’s a tool that turns a screenshot into a markdown snippet with the component name, props, and description. So I went hunting.

What I tried that didn’t work

First, I tried the obvious: OCR + regex. Take a screenshot, run Tesseract, then parse the text for component names and props. That failed miserably because:

The UI text was often in styled fonts that OCR misread (e.g., “Button” became “But1on”).
It couldn’t understand the visual structure — a dropdown vs. a toggle look similar in text output.
Regex to detect “Props: size, variant” fell apart when the text wrapped or had icons.

Next, I looked at cloud-based AI documentation generators. Most required me to upload my entire component library, integrate with their SDK, and pay per component. I didn’t want vendor lock-in. I also didn’t want to share my codebase with a third party just to get docs.

Then I tried a public multimodal model API like OpenAI’s GPT-4o. It worked — but the cost stacked up fast when processing 40+ screenshots multiple times during iteration. Plus, managing API keys and tokens for every teammate became a mess.

What eventually worked: a simple pipeline with any AI endpoint

I needed something cheap, self-hostable, and flexible. The idea was: write a small Python script that reads a screenshot file, sends it to any AI model that accepts images, and returns structured markdown. The script itself is the star — the AI endpoint is just a pluggable option.

Here’s the approach:

Capture – Take a screenshot (or use an existing one).
Prompt – Send the image with a clear instruction: “Describe this UI component in markdown format, including a heading, a brief description, and a table of props with name, type, default, and description.”
Parse – The response is markdown. Save it to a file.
Review – Because AI sometimes hallucinates props, I always do a quick human edit.

The key is that the same script works with OpenAI, Claude, local models via Ollama, or even a custom endpoint like the one at ai.interwestinfo.com (I tried it as a fallback). The technique is model-agnostic.

Code — the meaty part

#!/usr/bin/env python3
"""
Screenshot to Markdown documentation generator.
Works with any OpenAI-compatible API.
"""

import os
import sys
import base64
import requests
from pathlib import Path

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def image_to_markdown(image_path, api_key, endpoint="https://api.openai.com/v1/chat/completions"):
    """Convert an image to markdown via an AI model."""
    base64_image = encode_image(image_path)

    prompt = (
        "You are a UI documentation expert. Given a screenshot of a React component, "
        "generate a markdown description. Start with a second-level heading containing "
        "the component name. Then write a short description. Then create a table with "
        "columns: Prop Name, Type, Default, Description. If you cannot determine a prop, "
        "write N/A. Output only the markdown."
    )

    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {api_key}"
    }

    payload = {
        "model": "gpt-4o",  # swap to other models here
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{base64_image}",
                            "detail": "low"
                        }
                    }
                ]
            }
        ],
        "max_tokens": 500
    }

    response = requests.post(endpoint, headers=headers, json=payload)
    if response.status_code != 200:
        raise Exception(f"API error {response.status_code}: {response.text}")

    return response.json()["choices"][0]["message"]["content"]

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python screenshot2docs.py <image.png>")
        sys.exit(1)

    image_path = sys.argv[1]
    if not Path(image_path).exists():
        print(f"File not found: {image_path}")
        sys.exit(1)

    api_key = os.getenv("AI_API_KEY")
    if not api_key:
        print("Set AI_API_KEY environment variable.")
        sys.exit(1)

    md = image_to_markdown(image_path, api_key)
    # Save to a file with same name but .md extension
    out_path = Path(image_path).with_suffix(".md")
    out_path.write_text(md)
    print(f"Documentation saved to {out_path}")

How to use it

Install requests (pip install requests).
Set your AI_API_KEY environment variable (e.g., OpenAI key, or any compatible endpoint key).
Run: python screenshot2docs.py button.png
Edit the generated button.md to fix any errors.

Lessons learned and trade-offs

This approach is lightweight, but it’s not perfect. Let me be honest:

Accuracy: The model sometimes invents props that don’t exist, especially if the screenshot is blurry or the UI is complex. Always review the output.
Cost: Even with “low detail” and a cheap model, processing dozens of images repeatedly adds up. For a one-time doc generation, it’s fine. For a CI pipeline, you’ll want to cache results.
Latency: Each call takes 2–5 seconds. If you have 100 images, that’s 5–8 minutes. Not terrible, but you can parallelize easily with ThreadPoolExecutor.
Model dependence: The markdown output format can vary. I prompt for a table, but sometimes the model returns a list. I added a simple retry with re-prompting logic later.

When you should NOT use this approach

If you just need to extract text from a button label, use OCR (it’s faster and free).
If your components have minimal visual UI (e.g., server-side utilities), skip the screenshot step and generate docs from code directly.
If you have a huge library and need perfect accuracy, hire a human writer with an automation tool as a draft generator.

What I’d do differently next time

I’d build a small web frontend where I can drag & drop screenshots, see the generated markdown inline, and edit it before saving. The script works for batch, but interactivity helps with review. I’d also add a “model selector” dropdown to switch between endpoints on the fly.

Also, I’d write a deduplication layer: if two component variants look similar (e.g., primary/secondary buttons), the second generation tends to copy the first. Better to hash the image and check cache first.

The real takeaway

Automating documentation from screenshots saved me about 10 hours for this library. The technique of using a generic AI multimodal endpoint to generate structured data from images is reusable beyond docs — you could do it for design handoff specs, bug report screenshots, or auto-generating alt text for your blog.

Now I’d love to hear: What’s your go-to method for generating docs from visuals? Have you tried a similar image-to-markdown pipeline, or do you have a completely different workflow?

How I Built a Secure AI API Proxy Without Losing My Sanity

zhongqiyue — Sun, 14 Jun 2026 10:00:29 +0000

I’ve been integrating AI APIs into side projects for a few years now. Every time, I hit the same wall: I want to expose some AI-powered endpoint to my frontend, but I absolutely cannot put the API key in the client. The obvious answer is a backend proxy. But the first few times I did it, I ended up with a messy, insecure, or expensive mess.

Here’s what actually worked after a lot of trial and error.

The Problem

I was building a little tool that lets users ask questions about documentation. I needed to call an AI API (like OpenAI or Claude) from the frontend. Straightforward, right? Not quite.

API keys in the client: No way. Anyone can inspect network requests and steal your key.
Rate limits: Free tiers get hammered fast, and I didn’t want a runaway bill.
Latency: Direct calls from the browser sometimes get CORS errors or unexpected timeouts.
Prompt injection: If I didn’t sanitize user input, they could trick the AI into leaking system prompts or doing dangerous things.

So I needed a proxy. Not just any proxy — one that was secure, cost-aware, and easy to maintain.

What I Tried That Didn’t Work

1. A Simple Express Route

I started with the laziest possible solution:

app.get('/api/ai', async (req, res) => {
  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({ ... })
  });
  res.json(await response.json());
});

This works, but then everyone on my team had access to the same key. No logging, no rate limiting, no cost tracking. A single bug in the frontend could send a million requests and drain my account.

2. Using a Serverless Function

Next I tried a Vercel Edge Function. Cool until I realized that cold starts made it unpredictable, and debugging was a nightmare. Plus, I couldn’t easily log usage.

3. Pre‑built API Gateways

I looked at Kong, Tyk, and even my cloud provider’s API Gateway. Too much overhead. I just wanted to call one AI endpoint, not build a full enterprise solution.

What Eventually Worked: A Minimalist Proxy with Rate Limiting, Caching, and Token Tracking

I built a tiny Node.js express server with three key features:

Request validation – sanitize user prompts and limit max tokens.
Rate limiting per IP – using express-rate-limit.
Response caching – identical prompts get cached for 5 minutes to save money.
Cost logging – track how many tokens we use and estimated cost.

Here’s the core structure:

const express = require('express');
const rateLimit = require('express-rate-limit');
const NodeCache = require('node-cache');

const app = express();
const cache = new NodeCache({ stdTTL: 300 });

// Rate limiter: 10 requests per minute per IP
const limiter = rateLimit({
  windowMs: 60 * 1000,
  max: 10,
  message: 'Too many requests, please slow down.'
});
app.use('/api/ai', limiter);

app.post('/api/ai', async (req, res) => {
  const { prompt } = req.body;
  if (!prompt || prompt.length > 1000) {
    return res.status(400).json({ error: 'Invalid prompt' });
  }

  // Cache check
  const cacheKey = prompt.trim().toLowerCase();
  const cached = cache.get(cacheKey);
  if (cached) {
    console.log('Cache hit');
    return res.json(cached);
  }

  try {
    const response = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({
        model: 'gpt-4o-mini',
        messages: [{ role: 'user', content: prompt }],
        max_tokens: 150
      })
    });

    if (!response.ok) {
      console.error('AI API error:', response.status);
      return res.status(502).json({ error: 'AI service unavailable' });
    }

    const data = await response.json();
    const tokensUsed = data.usage?.total_tokens || 0;
    console.log(`Tokens used: ${tokensUsed}`);

    // Cost estimation (example: gpt-4o-mini is $0.15 per 1M input tokens, $0.60 per 1M output)
    const estimatedCost = (tokensUsed / 1000000) * 0.15; // crude
    console.log(`Estimated cost: $${estimatedCost.toFixed(5)}`);

    // Cache the result
    cache.set(cacheKey, data);

    res.json(data);
  } catch (error) {
    console.error('Proxy error:', error);
    res.status(500).json({ error: 'Internal server error' });
  }
});

app.listen(3000, () => console.log('AI proxy running on port 3000'));

Lessons Learned

Rate limiting is non-negotiable. Even if you trust your users, you don’t trust the internet. Free tiers get abused.
Cache aggressively, but be smart about it. Identical prompts from different users should not hit the API again. But if your app is real‑time (e.g., chat), caching might hurt freshness. In that case, skip cache or reduce TTL.
Log everything (but don’t store raw prompts if they’re sensitive). I log token counts and response status, but I hash the prompt before logging to avoid storing user data.
Don’t just forward the request. Validate and sanitize. Strip any system role overrides the user might try to inject.

Trade-Offs & When NOT to Use This Approach

This proxy is single‑node. If you need high availability, you’ll want a load balancer or a more scalable solution like a serverless proxy on Cloudflare Workers (which can also cache at the edge).
If you’re using streaming (SSE), caching doesn’t work well. You’ll need to handle that differently.
The cost estimation is ballpark. For precise billing, use the AI provider’s dashboard – don’t rely on your own maths.

What I’d Do Differently Next Time

Start with a Cloudflare Worker instead of Express. It’s free for low usage, has built‑in caching and rate limiting, and runs globally. The code would be similar but in the Workers runtime.
Use a queue for non‑real‑time requests. For batch processing, I’d put prompts in a queue (like Bull or AWS SQS) and have a worker consume them with a concurrency limit. This avoids overwhelming the API and makes costs predictable.
Set up alerts. I’d use a webhook email service (e.g., SendGrid) to notify me if the error rate spikes or if daily token usage exceeds a threshold.

Final Thoughts

Building an AI proxy isn’t rocket science, but the details matter. The code above got me from zero to a usable, secure endpoint in about an hour. Since then I’ve refined it with more middleware – JWT authentication, user‑specific rate limits, and a simple dashboard for cost tracking.

Your turn: what’s your current setup for exposing AI APIs safely? Have you tried a different stack or found a clever caching trick? I’d love to hear what’s working for you.

Building a Serverless Proxy for AI APIs: Lessons Learned

zhongqiyue — Sun, 14 Jun 2026 08:01:29 +0000

Last month, I needed to add an AI-powered chatbot to my side project. The requirement sounded simple: take user messages, send them to an LLM API (like GPT), stream the response back to the frontend, and keep the API key safe. Easy, right?

I thought so too – until I actually tried to wire it up from the browser. That’s when the real headache started.

The CORS Wall

Like many frontend developers, my first instinct was to call the AI API directly from JavaScript. Just fetch the endpoint, pass my API key, and get the response. But the API didn’t support CORS requests, and even if it did, exposing the API key in the client-side bundle is a security nightmare. So that was a dead end.

The Simple Node Server

Next, I spun up a small Express server. A few lines later, I had a /chat endpoint that proxied requests to the AI API. The API key stayed server-side, CORS was handled by the server, and everything worked… until I had to deploy it.

I needed something cheap and scalable. A full-blown VPS felt like overkill. I tried running the Express app on a cheap cloud server, but then I had to worry about uptime, process management, and restarting on crashes. Not fun for a weekend project.

I looked into services like Interwest AI (they offer a unified AI API with built-in proxy logic) – but I wanted full control over the request/response pipeline, and I didn’t want to rely on another dependency for such a small thing. Plus, I was curious if I could build something lean myself.

Enter Serverless Functions

I realised a serverless function (on Vercel, Netlify, or AWS Lambda) could act as a perfect lightweight proxy. No server to manage, no idle costs, and I could secure the API key using environment variables. I chose Vercel because I already host my frontend there, but the same approach works anywhere.

Here’s the core of the solution:

// api/chat.js (Vercel serverless function)
import { OpenAIStream, StreamingTextResponse } from 'ai';
import { Configuration, OpenAIApi } from 'openai-edge';

const config = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(config);

export const config = {
  runtime: 'edge',
};

export default async function handler(req) {
  const { messages } = await req.json();

  const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo',
    stream: true,
    messages,
  });

  const stream = OpenAIStream(response);
  return new StreamingTextResponse(stream);
}

That’s it. The function receives user messages, calls the AI API with streaming enabled, and returns a streamed response to the frontend. The ai SDK from Vercel makes streaming trivial – but you can also implement it manually using the standard ReadableStream API.

On the frontend, I used the same ai library to consume the stream:

import { useChat } from 'ai/react';

function Chat() {
  const { messages, input, handleInputChange, handleSubmit } = useChat({
    api: '/api/chat',
  });

  return (
    <div>
      {messages.map(m => (
        <div key={m.id}>{m.role}: {m.content}</div>
      ))}
      <form onSubmit={handleSubmit}>
        <input value={input} onChange={handleInputChange} />
      </form>
    </div>
  );
}

It felt almost too easy – and that’s the point. The heavy lifting is done by the serverless platform and the AI SDK. My only job was connecting the dots.

Lessons Learned (the hard way)

While this approach works beautifully for low-traffic projects, there are some sharp edges I discovered:

Cold starts: Serverless functions can have a few hundred milliseconds of cold start latency. For a chat app, that initial delay is noticeable. Vercel’s Edge Functions help reduce this, but it’s not zero.
Timeouts: Many serverless providers cap function execution time (e.g., 10 seconds on Vercel’s Hobby plan). Streaming helps because the response starts immediately, but if the AI takes too long to generate the first token, you might hit the timeout.
Rate limiting: Without any throttling, a user could spam the endpoint and rack up your API bill. I later added a simple in-memory rate limiter using Upstash (or a Redis-backed solution) to prevent abuse.
Error handling: The AI API can return errors (quota exceeded, invalid model, etc.). My initial version just forwarded the error, which led to confusing frontend messages. I added better error formatting and a fallback response.

What I’d Do Differently Next Time

If I were to rebuild this today, I’d:

Use a queue for heavy traffic: If many users hit the endpoint simultaneously, serverless functions can scale, but each request triggers a fresh API call. A queue (like BullMQ with Redis) would batch requests or deduplicate identical prompts.
Add caching for common prompts: Some users ask the same questions. Caching the AI response for a few minutes would save money and speed up replies.
Consider a dedicated worker for production: For a high-traffic app, a persistent Node.js process (deployed on Fly.io or Railway) might be simpler to debug and tune than serverless functions with their quirky limits.

When to Use This Approach (and When Not To)

This serverless proxy is perfect for:

Small side projects or MVPs
Prototyping AI features quickly
Apps with light to moderate traffic

Avoid it if:

You need sub-100ms response times at the p99
You have millions of requests per day (cost efficiency becomes tricky)
You want full control over the HTTP/2 multiplexing and connection pooling

Wrapping Up

Building a serverless proxy for AI APIs solved my immediate problem: secure, CORS-free, and deployable in minutes. It’s not a silver bullet, but for most indie devs and small teams, it’s more than enough.

Have you found a better approach for handling AI API calls in production? I’d love to hear what’s working for you.

DEV Community: zhongqiyue

Why I Stopped Relying on a Single AI Provider (and Built a Fallback System)

The Problem: Single Point of Failure

What I Tried First (and Why It Failed)

The Approach: Provider Abstraction + Fallback Router

1. The Base Class

2. Concrete Implementations

3. The Fallback Router

4. Putting It All Together

Lessons Learned and Trade-offs

When NOT to Use This Approach

What I'd Do Differently Next Time

Wrapping Up

I spent weeks scraping 50 websites—here's what finally worked

The problem (my actual problem, not a hypothetical one)

What I tried that didn't work (honest dead ends)

1. Traditional scraping with requests + BeautifulSoup

2. Selenium with Chrome headless

3. Scrapy with custom middlewares

What eventually worked (the technique, not the tool)

The specific service I used

Lessons learned / trade‑offs

When NOT to use this approach

What I’d do differently next time

When Your AI API Goes Down: A Real-World Fallback Strategy

The Problem

What I Tried First

What Eventually Worked: A Weighted Multi-Provider Router

Lessons Learned / Trade-offs

What I'd Do Differently Next Time

The Real Takeaway

How I Fixed My AI Chatbot's Timeout Nightmare

The Problem

What I Tried That Didn't Work

What Eventually Worked: Streaming + Smart Retry

Code Walkthrough

Trade-offs & Limitations

What I'd Do Differently

The Result

Why I ditched regex scrapers for an LLM parser (and when you shouldn't)

The problem that broke my scraper

What I tried that didn’t work

The approach that eventually worked

Lessons learned and trade-offs

What I’d do differently next time

Your turn

When Regex Fails: My Journey to AI-Powered Data Extraction

What I tried that didn't work

Regex-only approach

Beautiful Soup + manual cleaning

The dead-end: spaCy NER

What eventually worked

The code

Validation layer

Lessons learned

When NOT to use this approach

What I'd do differently next time

Over to you

Extracting structured data from messy text: what worked for me

The approach that finally stuck: few-shot LLM extraction

Trade-offs I hit

When NOT to use this

One tool that simplified my life

Lessons learned

I tried 5 ways to build a Q&A system over my docs — here's what worked

The Setup

What I Tried (and Why It Failed)

1. Pure vector search with local embeddings

2. OpenAI embeddings + GPT-4

3. All‑in‑one local model (like GPT4All)

4. LangChain's built‑in load_qa_chain

5. The hybrid approach that finally worked

The Code That Finally Clicked

1. Chunk your documents

2. Create embeddings locally (optional, but saves API calls)

3. Store in a vector index (FAISS)

4. Query and answer using the hosted API

Trade-offs & Honest Criticism

Lessons Learned

What I'd Do Differently Next Time

4. LangChain's built‑in `load_qa_chain`