DEV Community: Zee

We Built a Custom Playwright Rendering Pipeline for Our MCP Server — Here is What We Learned

Zee — Mon, 20 Apr 2026 19:23:07 +0000

We Built a Custom Playwright Rendering Pipeline for Our MCP Server — Heres What We Learned

At Haunt API, we build web extraction tools for AI agents. Our MCP server lets Claude and other AI assistants extract structured data from any URL. Simple enough on paper — fetch a page, parse the HTML, return JSON.

The problem? Half the internet doesnt want to be fetched.

The Problem With Just Use Playwright

Most web scraping tutorials go something like this:

from playwright.async_api import async_playwright

async with async_playwright() as p:
    browser = await p.chromium.launch()
    page = await browser.new_page()
    await page.goto(url)
    html = await page.content()

And that works! For a demo. For a product that real users depend on, it falls apart fast:

Sites detect headless browsers and serve captchas or empty pages
SPA pages need time to render — how long do you wait? 2 seconds? 5? 10?
You are burning resources loading images, fonts, and CSS when you only need text
Every render costs the same — no caching, no intelligence

We went through all of these. Here is how we solved each one.

Lesson 1: Do Not Use One Tool For Everything

Our pipeline has three tiers, and most requests never hit Playwright:

Direct HTTP — Works for approximately 80% of the web. Fast, cheap, no browser needed.
FlareSolverr — Handles Cloudflare challenges and basic JS rendering.
Playwright — Full browser rendering for JS-heavy SPAs that return empty skeletons.

The key insight: we detect skeleton pages — HTML that has an empty root div but no actual content — and only spin up the browser when we need to.

Lesson 2: Smart Wait Strategies Beat Fixed Timers

The worst thing about browser automation is the waiting. A fixed sleep is either too short or too long. We built three concurrent wait strategies — first one to trigger wins:

Content Stability — Poll visible text every 200ms. If unchanged for 1 second, done.
Network Idle — Wait for no new requests for 500ms.
Meaningful Content — Wait until 500+ chars of visible text exist.

This cut our average render time from 6 seconds to under 3.

Lesson 3: Fingerprint Rotation Matters

Headless Chromium has tells. We rotate fingerprints per-URL — same site sees a consistent browser, different sites see different browsers. 10 viewport variants across Windows, macOS, and Linux UAs.

Lesson 4: Block What You Do Not Need

When extracting text data, images and fonts are dead weight. We block them at the network level plus 20+ tracking domains. This cuts HTML payload by 40-60%.

Lesson 5: Cache Renders, Not Requests

If two users extract data from the same URL within 5 minutes, the page probably has not changed. Cache hits return in 0ms.

The Architecture

Six modules, each with a single job:

server.py — FastAPI orchestration, browser lifecycle
fingerprint.py — UA/viewport/locale rotation
smart_wait.py — Content stability + network idle detection
site_detect.py — Static vs SPA classification
cache.py — LRU render cache with TTL
stealth.py — Resource blocking + headless detection evasion

Each module is approximately 100 lines. Easy to test, easy to modify.

What We Learned

Do not reach for the browser first. Most pages are server-rendered.
Wait smarter, not longer.
Be a moving target with fingerprint rotation.
Cache aggressively.
Build modules, not monoliths.

The Playwright browser engine is the oven. Everything around it — the routing, the waiting, the caching, the stealth — is the recipe. That is where the actual engineering lives.

We are Haunt API — web extraction built for AI agents. If you are building with Claude, Cursor, or any AI assistant, our MCP server gives your agent the ability to extract data from any URL.

I Built an AI That Talks People Out of Cancelling Their Subscriptions

Zee — Mon, 20 Apr 2026 15:35:05 +0000

Here's the thing about churn: by the time someone clicks "Cancel Subscription", they've already decided. Your generic "Would you like 20% off?" popup is too late and too weak.

I spent the last month building SaveMyChurn — an AI-powered churn recovery tool for Stripe SaaS founders. This is how it works, what I learned building it, and why I think most cancellation flows are doing it wrong.

The problem

I was looking at my own Stripe dashboard one day and noticed something: the cancellation flow was the most ignored piece of the entire subscription experience. People pour weeks into onboarding, feature development, marketing — and then the cancel button just... ends things. No conversation. No understanding of why.

For bootstrapped SaaS founders running £5K-50K MRR, every subscription matters. Losing 5% of your customers a month isn't a statistic — it's the difference between growing and dying.

The existing tools didn't fit. Churnkey starts at $250/month — that's a significant chunk of revenue when you're small. The cheaper options are just form builders with a discount code at the end. Nobody was actually talking to the customer.

What I built

SaveMyChurn does three things:

1. Listens to Stripe in real time

When a customer hits cancel, Stripe fires a customer.subscription.deleted webhook. SaveMyChurn catches it instantly, pulls the subscription metadata, payment history, and plan details, and builds a profile of who's leaving and why.

# The webhook handler — this is where it starts
@router.post("/webhooks/stripe")
async def stripe_webhook(request: Request):
    payload = await request.body()
    event = stripe.Webhook.construct_event(
        payload, request.headers["stripe-signature"], webhook_secret
    )

    if event["type"] == "customer.subscription.deleted":
        subscription = event["data"]["object"]
        # Build subscriber profile from Stripe data
        profile = await build_subscriber_profile(subscription)
        # Generate AI retention strategy
        strategy = await generate_retention_strategy(profile)
        # Send personalised recovery email
        await send_retention_email(profile, strategy)

2. Generates a unique retention strategy per subscriber

This is the part I'm most proud of. Instead of a static "here's 20% off" flow, an AI strategist analyses the subscriber's behaviour — how long they've been a customer, what plan they're on, their payment history, any support tickets — and creates a genuinely personalised retention offer.

Someone cancelling after 2 months gets a different approach than someone who's been around for a year. Someone on a basic plan gets a different offer than someone on enterprise. The AI adjusts tone, offer type, discount level, and follow-up timing based on the full context.

3. Follows up automatically

One email rarely saves a cancellation. SaveMyChurn runs a multi-step sequence — initial offer, follow-up with adjusted terms, final value reminder — spaced over a few days. Each step is informed by whether they opened the previous email, clicked anything, or went silent.

The tech stack

Keeping it simple and cheap:

FastAPI backend — async Python, handles webhooks fast
MongoDB for subscriber profiles and strategy storage
Redis for caching and rate limiting
LLM via API for strategy generation — the AI strategist
Resend for transactional emails
Docker on a single VPS — the whole thing runs on one machine

The LLM cost per strategy generation is under a penny. When your competitor charges $250/month, that's a ridiculous margin.

The pricing model (and why it matters)

I went with a commission model. Monthly fee + a percentage of recovered revenue. The idea is simple: if I don't save you money, I don't make money.

This was a deliberate choice. Flat-fee tools have an incentive to get you signed up and keep you paying, regardless of results. Commission pricing means I'm motivated to actually recover subscriptions, not just ship a dashboard.

For founders at the £5K-50K MRR stage, this aligns incentives in a way that $250/month flat fees don't.

What I learned

Webhook reliability is everything. If you miss a customer.subscription.deleted event, you miss the entire recovery window. I ended up implementing retry queues and idempotency keys before anything else.

AI strategy > rules engine. I initially built a simple rule-based system (if cancel reason = "price" → offer discount). It was okay. The AI strategist that replaced it generates strategies I wouldn't have thought of — bundling features differently, offering plan downgrades instead of discounts, timing follow-ups based on engagement patterns.

One email is never enough. The first recovery email has maybe a 15-20% open rate. The follow-up catches another chunk. The third one gets the people who were "going to get around to it." Multi-step sequences doubled recovery rates compared to single emails.

Where it's at

SaveMyChurn is live and in production. It works end-to-end: Stripe webhook → AI strategy → personalised email sequence → dashboard showing what was saved.

If you're a bootstrapped SaaS founder on Stripe watching subscriptions slip away, give it a look. There's a free trial — no credit card required.

Your AI Agent Can't Scrape That Page. Here's How to Fix It.

Zee — Mon, 20 Apr 2026 15:16:08 +0000

Your AI Agent Can't Scrape That Page. Here's How to Fix It.

You built an AI agent that needs real-time web data. Product prices, news articles, competitor info — whatever it is, you need clean HTML or JSON from a URL.

So you fire off a requests.get() and... 403 Forbidden. Cloudflare says no.

Or you get a page, but it's empty — the content loads via JavaScript after the page renders, and your HTTP client never sees it.

Sound familiar? Let's break down what's happening and how to actually solve it.

Why Your Scraping Fails

1. JavaScript Rendering

Modern sites are SPAs. The HTML you get from a raw HTTP request is a shell — the actual content is loaded by JavaScript after the page mounts. requests, axios, fetch — none of them execute JS.

2. Cloudflare and Bot Detection

Cloudflare fingerprints your connection:

TLS fingerprint (does your HTTP client look like a browser?)
HTTP/2 fingerprint
Browser behavior (mouse movements, JS execution patterns)
IP reputation

Regular HTTP clients fail all of these checks.

3. Complex Layouts

Even when you get the HTML, extracting structured data from it is painful. You write brittle CSS selectors that break on every layout change.

The Solutions (From Worst to Best)

Selenium/Playwright Headless Browsers

They work... sometimes. But Cloudflare detects headless Chrome. You'll spend more time maintaining anti-detection patches than building your actual product.

Rotating Proxies + Custom Headers

Expensive, slow, and fragile. You're playing whack-a-mole with detection rules.

Use an API That Handles Everything

This is where tools like Haunt API come in. It's a web extraction API built specifically for AI agents:

import requests

resp = requests.post(
    "https://hauntapi.com/v1/extract",
    headers={"x-api-key": "your-key"},
    json={
        "url": "https://example.com/product/123",
        "prompt": "Get the product name, price, and availability"
    }
)

print(resp.json()["data"])
# {
#   "product_name": "Wireless Headphones Pro",
#   "price": "$79.99",
#   "availability": "In Stock"
# }

That's it. One API call. Cloudflare bypassed, JavaScript rendered, structured data extracted.

How It Works Under the Hood

Smart fetching — tries direct HTTP first, falls back to headless browser with anti-fingerprinting for Cloudflare-protected sites
JavaScript executes — SPA content becomes available
AI extracts the data you described in your natural language prompt
Clean JSON returned to your application

MCP Server for Claude and Cursor

If you're building with AI agents, Haunt also has an MCP server:

{
  "mcpServers": {
    "haunt": {
      "command": "npx",
      "args": ["@hauntapi/mcp-server"],
      "env": {
        "HAUNT_API_KEY": "your-key"
      }
    }
  }
}

Add that to your Claude Desktop or Cursor config and your AI agent can extract data from any website natively. Zero code.

REST API (No SDK Needed)

curl -X POST https://hauntapi.com/v1/extract \
  -H "x-api-key: your-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "prompt": "Get the top 5 stories with titles, points, and URLs"
  }'

Free Tier

100 extractions/month for free. No credit card required. Perfect for prototyping your AI agent before scaling up.

Paid plans start at £19/mo for 1,000 requests with authenticated scraping and priority support.

When to Use What

Approach	Cost	Reliability	Setup Time
Raw requests	Free	Low (30%)	5 min
Selenium + proxies	$$$	Medium (60%)	Hours
Haunt API	Free tier	High (95%+)	5 min

TL;DR

If your AI agent needs web data and you're tired of fighting bot detection, try Haunt API. It handles Cloudflare, JavaScript rendering, and data extraction in a single API call.

Free to start, built for AI agents and RAG pipelines.

Disclosure: I built Haunt API because I was tired of writing the same scraping infrastructure for every project.