DEV Community: mahuan2928

Don't Bolt LLM Calls Into Your App — Put Them Behind a Tiny API Instead

mahuan2928 — Mon, 26 Oct 2026 14:24:57 +0000

The problem: the five-line feature that metastasizes

Every team adds their first LLM feature the same way. Someone writes five lines inside the existing codebase:

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
summary = client.chat.completions.create(...)

It works. Then product asks for a second feature — chat. Then classification for the support inbox. Six months later, OpenAI calls are scattered across four modules, the API key lives in three config files, nobody knows which endpoint is burning tokens, and swapping models means a search-and-replace across the whole repo. The LLM stopped being a feature and became infrastructure — except nobody built it like infrastructure.

There's a boring pattern that prevents this: put LLM calls behind a tiny HTTP service of your own. Your app talks to one small FastAPI sidecar; the sidecar talks to the provider. Keys, prompts, and provider quirks live in exactly one place.

I built a free demo of this pattern — one endpoint, about 150 lines — so you can see the shape before committing to it.

The demo: one feature, done cleanly

The demo service has exactly two endpoints:

GET /health — no auth, just liveness
POST /features/summarize — takes {text, max_sentences}, returns a summary

The whole thing is a single app.py. Here's the interesting part — the auth dependency:

def require_api_key(x_api_key: str | None = Header(default=None)) -> None:
    expected = os.getenv("SERVICE_API_KEY")
    if not expected:
        raise HTTPException(status_code=503,
            detail="SERVICE_API_KEY is not set on the server.")
    if x_api_key != expected:
        raise HTTPException(status_code=401,
            detail="Invalid or missing X-API-Key.")

Your backend sends a shared X-API-Key header; browsers and mobile apps never see it, because they talk to your backend, not to this service. One key, one check, every feature endpoint protected by adding dependencies=[Depends(require_api_key)].

And here's the detail most demos skip — what happens when the OpenAI key is missing:

openai_key = os.getenv("OPENAI_API_KEY")
if not openai_key:
    return JSONResponse(status_code=503, content={
        "detail": "OPENAI_API_KEY is not set. Copy .env.example "
                  "to .env and add your OpenAI API key."
    })

Not a stack trace. Not a service that crashes on boot at 2 a.m. A clean 503 that tells the next developer exactly what to fix. Small thing, but it's the difference between a template and a toy.

The summarize endpoint itself is deliberately unremarkable:

@app.post("/features/summarize", dependencies=[Depends(require_api_key)])
def summarize(req: SummarizeRequest):
    # ... missing-key check above ...
    completion = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": f"Summarize in at most "
             f"{req.max_sentences} sentence(s). Keep key facts and numbers."},
            {"role": "user", "content": req.text},
        ],
    )
    return {"summary": completion.choices[0].message.content, ...}

Run it:

pip install -r requirements.txt
cp .env.example .env   # set SERVICE_API_KEY + OPENAI_API_KEY
uvicorn app:app --port 8000

curl -X POST localhost:8000/features/summarize \
  -H 'X-API-Key: YOUR_KEY' -H 'Content-Type: application/json' \
  -d '{"text": "Long ticket text...", "max_sentences": 2}'

That's the pattern. Your Rails, Django, Laravel, or Node backend doesn't care that Python and OpenAI are on the other side of the HTTP call — and that's the point.

Where the demo stops (on purpose)

The demo is honest about being a demo. The moment this serves real users, three production concerns show up:

1. Who spent what? A summarize call costs fractions of a cent — a runaway loop or one enthusiastic client costs real money. Production needs per-client token budgets, checked before the provider call: estimate the request, and if the client is over their daily cap, return 402 without spending anything. Settling with the provider's real usage numbers afterwards keeps the accounting honest.

2. What happens when the provider hiccups? OpenAI returns 429s and 5xx like everyone else. Without retries with backoff and a sane timeout, your users see raw failures for what was a two-second blip.

3. Can you prove what happened? When finance asks "why did the bill triple in March?", you need one JSON log line per call — endpoint, client, input/output tokens, latency — not a forensic expedition through application logs.

Add streaming chat (SSE) and a classify endpoint, and you've described the full version of this service — same request/response shape, with budgets, rate limits, retries, logging, and a provider abstraction so "switch to Claude" is a config change, not a refactor.

Try it

The demo is free and MIT-licensed — clone it, run the three smoke tests (/health → 200, no key → 401, no OpenAI key → clean 503), and wire summarize into your app this afternoon:

GitHub: https://github.com/mahuan2928/llm-feature-demo

If you want the production version — /features/chat with SSE streaming, /features/classify, per-client daily token budgets with HTTP 402 enforcement, rate limiting, retries with backoff, and JSON call logging — it's here:

Full version ($49): https://mahuan220.gumroad.com/l/gwwrrn

Start with the demo. You'll know within one integration whether the pattern fits — and if it does, the upgrade path is a drop-in, not a rewrite.

Built by Marty Ma — Senior AI & Data Engineer (Tokyo), 11+ years building production systems for enterprise clients.

Stop Rewriting Your Web Scraper for Every New Site

mahuan2928 — Mon, 26 Oct 2026 14:23:55 +0000

Every web scraper I've ever written started the same way: a quick script for one site, hardcoded selectors, done in an afternoon. Then someone asks for "the same thing, but for this other site" — and you copy the file, change the URL, re-hunt every selector, rediscover the pagination quirks, and ship a second script that shares 90% of its DNA with the first. By the fourth site you're maintaining a small zoo of near-identical programs, each of which breaks independently whenever a site redesigns.

This post builds the honest version of that first script — small, readable, actually working — and then shows where it hits its ceiling, and the config-driven pattern I now use instead.

The demo: one file, one site, twenty books

The target is books.toscrape.com, a practice site built for exactly this. Goal: every book on the first catalogue page — title, price, rating, availability — into a CSV. The whole thing is one file, scrape.py, using only requests and BeautifulSoup:

URL = "https://books.toscrape.com/catalogue/page-1.html"
HEADERS = {"User-Agent": "Mozilla/5.0 ... web-scraper-demo/1.0"}

response = requests.get(URL, headers=HEADERS, timeout=30)
response.raise_for_status()
response.encoding = "utf-8"  # don't let requests guess — see below

Two details in those few lines matter more than the rest of the script combined. First, send a real User-Agent and keep your request rate polite; scraping that hammers a server is how IPs get banned and side projects get legal letters. Second, the encoding line: on my first run, every price came out as Â£51.77 instead of £51.77, because requests trusted the response headers over the page's actual UTF-8 content. One explicit line fixed all twenty rows. Small bug, but it's the kind that silently poisons a dataset if you're not eyeballing your output.

Parsing is one function. Each book is an article.product_pod card, and everything we want hangs off it:

for pod in soup.select("article.product_pod"):
    title = pod.select_one("h3 a")["title"]
    price = pod.select_one("p.price_color").get_text(strip=True)
    rating_el = pod.select_one("p.star-rating")
    rating = next(c for c in rating_el["class"] if c in RATING_WORDS)
    availability = " ".join(
        pod.select_one("p.availability").get_text(" ", strip=True).split()
    )

The rating is a nice example of real-world markup: it isn't text at all, it's a CSS class (class="star-rating Three"), so you extract it from the class list rather than the element's contents. The availability text comes with stray whitespace and a nested icon element, so it gets normalized with a split-and-join. Then csv.DictWriter writes the rows out, and you have books.csv with 20 clean records. Total: about 60 lines. It works. Ship it?

Where the one-file scraper breaks

It breaks at the second site, and the failures are structural, not incidental:

Selectors live in code. A new site means editing Python, re-testing, and redeploying — for what is really just data about the site.
Pagination is a snowflake. This demo deliberately stops at page 1. Real sites paginate with next-links, page-number URLs, or infinite scroll, and each style needs different driving logic.
Not everything is static HTML. The moment a site renders with JavaScript, requests gets an empty shell and your selectors match nothing — silently.
Nobody counts the bad rows. Real pages have missing prices, duplicate listings, layout variants. A quick script mixes them into your CSV and you find out downstream, in a spreadsheet, at the worst time.

None of these are hard problems individually. Together they mean the "quick script" quietly becomes the job.

The fix: the site is data, not code

The pattern I use in production flips the script inside out. Everything site-specific — start URLs, the item container, per-field selectors and types, pagination style, rate limits, dedup keys — moves into a YAML config. The engine stays fixed. A new site looks like this:

site:
  name: books
  start_urls: ["https://books.toscrape.com/catalogue/page-1.html"]
  rate_limit_seconds: 1.0
items:
  container: "article.product_pod"
  fields:
    title:  { css: "h3 a", attr: "title", type: string, required: true }
    price:  { css: "p.price_color", type: float, required: true }
pagination:
  type: next_link
  next_selector: "li.next a"
dedup:
  key_fields: [title]

Same target, same fields — but now adding a site is writing twenty lines of YAML instead of forking a program. The engine adds the things the demo lacks: all three pagination styles, a Playwright mode for JS-rendered pages, type coercion and required-field validation, deduplication with a count of dropped rows (so bad data is a number in your log, not a surprise in your spreadsheet), Excel output, and retries with backoff so it can run unattended from cron.

Try it, then skip the rewrite

The demo from this post is free and MIT-licensed — clone it, run it, take it apart:

👉 GitHub: https://github.com/mahuan2928/web-scraper-demo

And if you're at the "second site" stage — or you can see it coming — the full config-driven engine is available here, with example configs (including this same book site, all 50 pages of it), the Playwright mode, Sheets export, and the validation/dedup pipeline described above:

👉 Config-Driven Web Scraper Engine ($39): https://mahuan220.gumroad.com/l/zknkka

Build the demo version once to learn the mechanics. Just don't build it five times.

— Marty Ma, Senior AI & Data Engineer (Tokyo)

Stop Copy-Pasting Invoices: Extract PDF Data to CSV with 100 Lines of Python

mahuan2928 — Mon, 26 Oct 2026 14:22:47 +0000

Stop Copy-Pasting Invoices: Extract PDF Data to CSV with 100 Lines of Python

Every month, somebody on every finance team does the same soul-crushing job: open an invoice PDF, copy the invoice number into a spreadsheet, copy the date, copy the total, repeat forty times. It's slow, it's boring, and it's exactly the kind of work where humans make quiet mistakes — a transposed digit in a total doesn't announce itself until the books don't balance.

The good news: if your invoices are text-based PDFs (the kind exported by accounting software, not scanned paper), this is a solved problem with embarrassingly little code. In this post we'll build a batch extractor in Python: point it at a folder of PDFs, get a CSV out. Full working demo, MIT licensed, is on GitHub — link at the end.

Step 1: Get the text out of the PDF

We'll use pdfplumber, which wraps pdfminer and gives you clean page text:

import pdfplumber

def pdf_text(path):
    with pdfplumber.open(path) as pdf:
        return "\n".join(page.extract_text() or "" for page in pdf.pages)

That's the whole "PDF parsing engine." For born-digital invoices, the text layer is already there — we just need to mine it.

Step 2: Extract fields with anchored regexes

Invoice layouts vary, but the labels are remarkably stable. Three patterns cover our three fields:

FIELDS = {
    "invoice_number": re.compile(
        r"Invoice\s*(?:No\.?|Number|#)\s*[:#]?\s*([A-Za-z0-9\-]+)", re.IGNORECASE),
    "invoice_date": re.compile(
        r"\bDate\s*[:\-]?\s*(\d{4}-\d{2}-\d{2})", re.IGNORECASE),
    "total_amount": re.compile(
        r"^\s*Total\s*:\s*\$?\s*([\d,]+\.\d{2})", re.IGNORECASE | re.MULTILINE),
}

One detail matters here: the total_amount pattern is anchored to the start of a line. Without that anchor, Subtotal: $1,000.00 happily matches a search for Total: — and your CSV fills up with subtotals wearing a fake moustache. Ask me how I know.

Amounts get a light normalization (1,100.00 → 1100.00) so the CSV is spreadsheet- and diff-friendly.

Step 3: Never drop a row silently

Here's the design decision I'd defend in a code review: when a field isn't found, we still write the row, leave the field empty, and name the gap in a status column:

filename,invoice_number,invoice_date,total_amount,status
partial.pdf,INV-2026-099,2026-03-01,,missing: total_amount

A batch tool that quietly skips files is a batch tool that creates reconciliation mysteries. Forty invoices in, thirty-nine rows out, and no clue which one vanished? No thanks. The status column turns "the numbers don't add up" into a ten-second lookup.

Step 4: Batch it and test it

The main loop is unglamorous: glob *.pdf, extract each, write rows with csv.DictWriter, print a per-file status. The part most tutorials skip is the test: the demo repo generates two synthetic invoices with ReportLab, ships an expected_output.csv with the known-correct answers, and the smoke test is literally:

python make_sample.py
python extract.py --input ./samples --out results.csv
diff results.csv expected_output.csv   # silence means success

Synthetic samples matter more than they look. You can't put real client invoices in a public repo, and "it worked on my machine once" isn't a test. Generate your fixtures, check them in, and let the diff be the judge.

Where the demo stops — and production starts

This demo extracts three hardcoded fields from well-behaved PDFs. Real invoice work is messier, and three problems show up immediately:

Every vendor has a different layout. Hardcoded regexes become a maintenance hobby. You want the fields — names, patterns, types — defined in a config file, so a new vendor is a new YAML block, not a code change.
Extraction without validation is a liability. Regex found a number after Total: — but is it right? The check that catches real money errors is cross-field arithmetic: does total == subtotal + tax? I once generated a test invoice with a deliberately wrong total (500 + 50 ≠ 560) specifically to watch a validator flag it. That's the feature that pays for itself the first time it catches a fat-fingered invoice before payment, not after.
Someone has to review the weird ones. In production you want a validation report and a "needs human review" list — the 5% of files that deserve eyes — instead of a human re-checking 100% of them.

Add per-field confidence scores, Excel output, and an OCR path for scanned PDFs, and the demo grows into the tool I actually ship to clients.

Try it / get the full version

Free demo (MIT): https://github.com/mahuan2928/pdf-extraction-demo — samples, expected output, and the whole ~100-line extractor.
Full version ($39): PDF & Invoice Data Extraction Pipeline — fields defined in fields.yaml (any fields, not just three), validation including cross-field totals checks, a validation report plus human-review list, Excel outputs, per-field confidence scores, and an OCR path for scanned PDFs.

If you adapt the demo to your own invoice layouts, I'd love to hear which pattern broke first — vendor creativity in invoice design is truly boundless.

— Marty Ma, Senior AI & Data Engineer (Tokyo)

Build a RAG Chatbot That Admits When It Doesn't Know (Pure Python, No Dependencies)

mahuan2928 — Mon, 26 Oct 2026 14:21:18 +0000

The problem: chatbots that guess

Every team wants a chatbot that answers questions from their own documents — support FAQs, internal policies, product manuals. The naive approach is to paste documents into a prompt or fine-tune a model, and the failure mode is always the same: when the documents don't contain the answer, the model invents one. A confident, wrong refund policy is worse than no answer at all.

Retrieval-augmented generation (RAG) fixes the retrieval half of this: you search your documents first, and only answer from what you found. But retrieval alone isn't enough. You also need a gate — a relevance threshold below which the bot refuses to answer — and citations, so a human can verify every claim.

In this tutorial we'll build a minimal version of that pattern in pure Python. No LangChain, no vector database, no dependencies at all: just the standard library. It runs completely offline using TF-IDF, and if you set an OPENAI_API_KEY it transparently switches to real embeddings and a chat model. The full code is about 300 lines across three files.

Step 1: Chunk the documents

Embeddings work on passages, not whole files, so ingestion starts by splitting each document into chunks. A simple paragraph-aware splitter is good enough for a demo: accumulate paragraphs until you approach the size limit, and window-split any single paragraph that's too long.

def chunk_text(text, chunk_size=600, overlap=100):
    paras = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
    chunks, buf = [], ""
    for para in paras:
        if len(para) > chunk_size:
            if buf:
                chunks.append(buf); buf = ""
            step = max(1, chunk_size - overlap)
            for i in range(0, len(para), step):
                chunks.append(para[i:i + chunk_size])
        elif not buf:
            buf = para
        elif len(buf) + len(para) + 2 <= chunk_size:
            buf = buf + "\n\n" + para
        else:
            chunks.append(buf); buf = para
    if buf:
        chunks.append(buf)
    return chunks

Each chunk keeps its source filename and position, which is what later lets us print citations like [Source: product-faq.txt, chunk 2].

Step 2: Build the index (TF-IDF offline, embeddings online)

Offline, we represent every chunk as a TF-IDF vector: term counts weighted by inverse document frequency, L2-normalised so cosine similarity is a plain dot product. It's fifty lines with collections.Counter and math.log — and for small, keyword-friendly document sets it works surprisingly well.

When an API key is available, we embed the chunks with OpenAI's text-embedding-3-small instead. Notice we don't even need the SDK; a JSON POST with urllib does the job:

def openai_embed(texts, api_key, model="text-embedding-3-small"):
    payload = json.dumps({"model": model, "input": texts}).encode()
    req = urllib.request.Request(
        "https://api.openai.com/v1/embeddings", data=payload,
        headers={"Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json"}, method="POST")
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return [item["embedding"] for item in data["data"]]

Either way, the result is one boring index.json: the chunks, their vectors, the mode, and a relevance threshold (0.10 for TF-IDF — its cosine scores run lower than embedding similarities, so each mode gets its own threshold).

Step 3: The honest gate

This is the most important fifteen lines in the project. At query time we embed the question the same way we embedded the chunks, rank by cosine similarity — and then check the best score before generating anything:

scored = sorted(
    ((cosine(query_vec, vec), chunk)
     for chunk, vec in zip(chunks, vectors)),
    key=lambda pair: pair[0], reverse=True)[:3]

if not scored or scored[0][0] < threshold:
    print(FALLBACK)   # "I don't know -- that information isn't
    return            #  in the documents provided."

If nothing retrieved clears the bar, the program prints the fallback and stops. The language model is never called, so it cannot hallucinate an answer the documents don't support. When the gate passes, the chat model receives only the retrieved chunks and is instructed to answer strictly from them, citing every claim. In offline mode there's no model at all: the top cited excerpts are printed as the answer, which is less fluent but arguably even more transparent.

Step 4: Run it

python ingest.py
python ask.py "How long is the warranty for the Nimbus kettle?"

The demo ships with two small sample documents — a fictional product FAQ and a fictional company policy — so it works out of the box. The smoke test, run fully offline:

Retrieved sources:
  [0.383] [Source: product-faq.txt, chunk 1]
  [0.353] [Source: product-faq.txt, chunk 2]

Answer (top excerpts from your documents):
How long is the warranty? Every Nimbus Smart Kettle comes with a
2-year warranty that covers manufacturing defects... [Source: product-faq.txt, chunk 2]

And the question the documents can't answer:

python ask.py "What is the current stock price of Tesla?"

Best match score: 0.000 (threshold: 0.1)
I don't know -- that information isn't in the documents provided.

Exactly the behaviour we want: an answer with receipts when the documents support one, and a refusal when they don't.

Honest limitations

This is a teaching demo, not a product. TF-IDF matches keywords, not meaning, so a paraphrased question ("how long does the guarantee last?") can miss a chunk that says "warranty". A JSON file is a fine index for hundreds of chunks and a poor one for millions. There's no PDF parsing, no web ingestion, no conversation memory, no server — it's a CLI that answers one question at a time. Thresholds also need tuning per document set; 0.10 worked for the samples, not for every corpus.

Where to go next

The demo is MIT-licensed and lives at https://github.com/mahuan2928/rag-chatbot-demo — fork it, drop your own .txt and .md files into docs/, and break it.

If you need the production shape of this pattern — PDF and web-page ingestion, a FastAPI server with an embeddable chat widget, Chroma/pgvector/Pinecone instead of a JSON file, session memory, per-IP rate limiting, and config-driven setup — that's the full RAG Chatbot Starter Kit I sell here: https://mahuan220.gumroad.com/l/ohfaru ($49, one commercial project per license). It's the same architecture, hardened for client work.

Either way: gate your retrieval, cite your sources, and let your chatbot say "I don't know."