DEV Community: phantomDev

I Built a Web Crawler That Scraped Cloudflare Past Their Own Protection

phantomDev — Sun, 15 Mar 2026 00:07:08 +0000

Last week I pointed my crawler at cloudflare.com, set the depth to 1, and walked away.

When I came back this is what I saw:

✓ (layer1) https://cloudflare.com
✓ (layer1) https://cloudflare.com/en-gb
✓ (layer1) https://cloudflare.com/de-de
✓ (layer1) https://cloudflare.com/products
✓ (layer1) https://cloudflare.com/developer-platform
...

Total crawled : 100+
Failed        : 0

Here is a proof

A website protected by Cloudflare, scraped past Cloudflare. 100+ pages. Zero blocks. All Layer 1 meaning not even a headless browser was involved.

That's PhantomCrawl. And this is how it works.

The actual scraped data is in the repo if you want to see it.

Why Scrapers Keep Breaking

Most developers write a Python script with requests, run it against a real site, and get a Cloudflare challenge page or a 403 back. So they switch to Puppeteer or Playwright. That works for a while, then sites start detecting headless Chrome too.

The part nobody talks about is TLS fingerprinting.

When your Go or Python HTTP client connects to a server it sends a TLS handshake. That handshake has a unique fingerprint - specific cipher suites, extensions, and ordering that identify the library you're using. A Go net/http client looks nothing like Chrome at the TLS layer. Cloudflare checks this fingerprint before it even looks at your User-Agent or cookies.

Changing your User-Agent header does nothing if your TLS fingerprint is screaming "I am a Python script."

PhantomCrawl fixes this at the transport level using utls HelloChrome_120 - the exact same TLS fingerprint as a real Chrome 120 browser. To Cloudflare's infrastructure, every request looks like it came from a real person on Windows Chrome. Because at the cryptographic handshake level, it genuinely does.

The Problem With Existing Tools

Before I built PhantomCrawl I looked at everything that existed. Here's the honest picture.

Feature Comparison

Tool	Self-Hosted	Anti-Bot	TLS Fingerprint	AI Cleaning	Binary	Free
PhantomCrawl	✅	✅	✅ HelloChrome_120	✅	✅	✅
Firecrawl	❌ API only	✅	❌	✅	❌	500 pages/mo
Scrapy	✅	❌	❌	❌	❌	✅
Apify	❌ Cloud only	✅	❌	❌	❌	Limited
BeautifulSoup	✅	❌	❌	❌	❌	✅
Puppeteer	✅	⚠️ Detectable	❌	❌	❌	✅
ScrapingBee	❌ API only	✅	❌	❌	❌	1,000 req/mo

What It Actually Costs

This is where things get uncomfortable for the existing tools.

Tool	Free Tier	Paid Entry	100K pages/mo
PhantomCrawl	Unlimited	$0	$0
Firecrawl	500 pages	$16/mo	$83/mo
Apify	~$5 credit	$29/mo	~$123/mo
ScrapingBee	1,000 req	$49/mo	$249+/mo
Bright Data	100 records	$500+/mo	$1,000+/mo
Browserless	6hr/mo	$29/mo	Pay per hour

PhantomCrawl is $0 because it runs on your machine. You are not paying for someone else's servers. The only optional cost is Groq for AI cleaning - which gives you 100,000 tokens free per day, enough for hundreds of pages at zero cost.

Scale to 1 million pages a month. Still $0. You just need a machine and an internet connection.

How PhantomCrawl Works

PhantomCrawl uses a 4-layer escalation engine. Every URL starts at Layer 1 and only moves up if needed. Most sites never leave Layer 1.

Layer 1 - Direct HTTP + TLS Fingerprinting

The fastest method. A direct HTTP request using utls HelloChrome_120 to disguise the TLS handshake as real Chrome. Includes human-like headers (Sec-Fetch-*, Sec-Ch-Ua-*), jittered timing, and user agent rotation.

Covers roughly 90% of the web. SSR sites, static pages, Next.js, WordPress, and yes - Cloudflare-protected sites.

Layer 2 - Network Hijacking

If Layer 1 gets HTML but the content is not useful, Layer 2 inspects the raw HTML for embedded data. It scans for window.__NEXT_DATA__, window.__INITIAL_STATE__, window.__NUXT__, JSON-LD structured data, and API endpoint patterns.

Many modern SPAs ship their data pre-embedded in the HTML before JavaScript even runs. Layer 2 extracts it directly without a browser.

Layer 2.5 - XHR/Fetch Interception

This is what makes PhantomCrawl different from anything else out there. Instead of scraping the rendered HTML from a headless browser, Layer 2.5 intercepts the actual API responses the browser receives during page load - the raw JSON from XHR and fetch calls.

The result is clean structured data with zero boilerplate. No parsing noise. No nav menus or footers. Just the data the page was going to display anyway, captured before it becomes HTML.

Layer 3 - Full Headless Browser

Last resort. Launches a real browser - go-rod if Chrome is installed locally, or Browserless via API - and fully renders the page with JavaScript execution. Handles the most complex SPAs.

You never configure which layer to use. PhantomCrawl decides based on what each site actually returns.

Every Feature

4-layer escalation engine with automatic fallback
TLS fingerprinting via utls HelloChrome_120
AI content cleaning via Groq or OpenAI with chunked processing
Rate limit retry with automatic backoff and resume
Proxy rotation tunneled at TCP level through the TLS transport
Depth crawling with per-parent link limits
Fragment URL deduplication (/#about and / are the same page)
Asset filtering - PDFs, images, and zips skipped during depth crawling
SQLite state tracking with full resume if interrupted
.env key management with $VAR_NAME references
Config generator UI - no terminal needed to configure
Cross-platform binaries for Linux, Mac (Intel + Apple Silicon), Windows, ARM, and Termux
Structured JSON output - raw.json and cleaned.json per page
Absolute URL resolution on all extracted links
Single binary under 20MB, no runtime required

The Sleep and Scrape Workflow 😴

Here is something nobody talks about with web scraping at scale. It takes time. Sites throttle you, rate limits kick in, AI cleaning queues up. Trying to babysit this in real time is exhausting and pointless.

So don't. This is literally the workflow:

# 1. Put your URLs in urls.txt
# 2. Run it
phantomcrawl start
# 3. Go to sleep

Wake up to this:

Total crawled : 847
Failed        : 0
AI cleaned    : 847
Clean pending : 0
Output        : ~/phantomcrawl/scraped

PhantomCrawl batches requests with randomized delays so your timing is never predictable. It retries failures with exponential backoff. If the AI token quota resets overnight, it resumes exactly where it left off. Nothing gets crawled twice.

Put your URLs in. Go to sleep. Wake up to a folder full of clean JSON. ☕

Getting Started

Download the binary for your platform from GitHub Releases:

Platform	Binary
Linux 64-bit	`phantomcrawl-linux-amd64`
Linux ARM / Termux	`phantomcrawl-linux-arm64`
macOS Apple Silicon	`phantomcrawl-darwin-arm64`
macOS Intel	`phantomcrawl-darwin-amd64`
Windows	`phantomcrawl-windows-amd64.exe`

# Linux/Mac
chmod +x phantomcrawl-linux-amd64
sudo mv phantomcrawl-linux-amd64 /usr/local/bin/phantomcrawl

# Three commands to your first crawl
phantomcrawl init
echo "https://example.com" > urls.txt
phantomcrawl start

Want AI cleaning? Get a free Groq key at console.groq.com, add it to .env, and set "ai": { "enabled": true } in crawl.json. That's it.

Full docs at phantomcrawl.vercel.app

Why It's Under 20MB

PhantomCrawl is written in Go and compiled to a single static binary. No runtime, no package manager, no node_modules folder that somehow weighs 300MB. Everything is included - the crawler, the AI pipeline, the API server, the SQLite driver, and the config system.

The binary is about 14MB. A fresh Next.js project's node_modules is 50MB before you've written a line of code. Go was the right choice for this.

Cross-compiling from a phone running Termux on Android to Linux amd64, macOS arm64, and Windows amd64 is one command per platform. Try doing that with Python.

One More Thing

I'm Raphael, 18, from Lagos, Nigeria. I started coding at 12 on a phone with 1GB of RAM. No laptop, no bootcamp, no one teaching me. PhantomCrawl is my 7th shipped product.

I built it because I needed it and nothing else did the job without either costing money or breaking on any site worth scraping.

If it helps you, a star on the repo means a lot.

github.com/var-raphael/PhantomCrawl

I Built a Web Analytics Tool That Needs Zero Signup. One Script Tag and You're Live in 30 Seconds

phantomDev — Tue, 03 Mar 2026 11:04:33 +0000

I was building a side project last year and needed to track visitors.

Google Analytics felt like overkill. 15 minutes of setup, a privacy policy update, cookie consent banners, GDPR configuration, and I still wasn't sure if my data was being used to target ads somewhere. For a side project. That maybe 50 people would visit.

Plausible was clean but $9/month for something I wasn't even monetizing yet felt wrong. Fathom was $14/month. Same problem.

So I did what developers do when nothing fits.

I built my own.

What I Built

PhantomTrack is a privacy-first web analytics tool with no signup, no cookies, and a 30-second setup.

You generate a tracking ID instantly (no account required, no email), drop one script tag into your HTML, and your dashboard is live. That's it.

<script 
  src="https://phantomtrack-cdn.vercel.app/phantom.js" 
  data-track-id="YOUR_TRACK_ID">
</script>

One line. Done.

The Features That Actually Matter

No Signup Required

This is the one I'm most proud of. Every analytics tool makes you create an account before you can track a single pageview. PhantomTrack generates your tracking ID instantly. No account, no email, no friction. You're collecting data before most tools have even sent you a verification email.

No Cookies. Zero.

PhantomTrack doesn't use cookies at all. No cookie consent banners. No GDPR headaches. No annoying popups destroying your UI because a lawyer said so. We only track what matters: pageviews and traffic sources, without storing personal data.

Works on SPAs (React, Next.js, Vue)

This one took the most work to get right. Standard analytics scripts track page loads. SPAs don't reload the page on navigation. They swap content dynamically. Most lightweight analytics tools miss this entirely.

PhantomTrack patches history.pushState under the hood so every route change in your React or Next.js app gets tracked automatically. No extra configuration needed.

Everything on One Dashboard

Visitors, unique visitors, pageviews, session duration, geographic data with country tier analysis, device and browser breakdown, traffic sources, real-time trends. All on one screen. No tab switching. No digging through menus.

AI Weekly Review

Every week PhantomTrack generates an AI-powered analysis of your traffic patterns: what's growing, what dropped, where your best traffic comes from. Useful when you're too busy shipping to manually analyze charts.

Country Tier Analysis

This one is specific to developers building for global audiences. PhantomTrack classifies your visitors by country tier (Tier 1, 2, 3) so you can understand not just where your traffic comes from but what that traffic is worth commercially.

7+ Export Formats

CSV, JSON, XML, Excel, PHP array, HTML, plain text. Your data, your way. No lock-in.

Three Ways to View Your Data

Web dashboard, iframe embed (drop your analytics directly into your own site or app), or JSON API (pull data programmatically and build whatever you want on top of it).

Pricing (And a Lifetime Deal)

Free: 10,000 requests/month. All features included.
Pro: $3/month. 30,000 requests/month.
Premium: $5/month. 60,000 requests/month.
Enterprise: $8/month. 100,000 requests/month.
Lifetime: $20 one-time. 300,000 requests/month, forever.

The lifetime deal is a launch special. It goes up to $50 after the launch period. If you're building side projects and want analytics without a recurring subscription, this is the one.

How It Compares

Feature	PhantomTrack	Google Analytics	Plausible	Fathom
No signup required	✅	❌	❌	❌
Setup time	30 seconds	15-30 minutes	5-10 minutes	5-10 minutes
No cookies	✅	❌	✅	✅
SPA support	✅	✅	✅	✅
AI weekly insights	✅	❌	❌	❌
Country tier analysis	✅	❌	❌	❌
Iframe embed	✅	❌	❌	❌
Export formats	7+	Limited	CSV only	CSV only
Lifetime deal	✅	❌	❌	❌
Starting price	Free	Free*	$9/month	$14/month

*Google Analytics is free but uses your data for ad targeting.

Try It Live

The docs site itself runs PhantomTrack. You can see the actual live dashboard tracking real visitor data right now at phantomtrack-docs.vercel.app.

That's not a demo with fake numbers. That's real traffic, tracked with the same script tag you'd use on your own site.

Get Started

👉 phantomtrack-docs.vercel.app

Generate your tracking ID, drop the script tag, and you'll have your first pageview tracked in under a minute. No account. No credit card. No setup friction.

If you try it, I'd genuinely love to hear what you think: what's missing, what's confusing, what would make it more useful for your workflow. Drop a comment or find me on X.

Built by Raphael, full stack developer. Also building phantomit-cli, a CLI tool that writes your git commit messages using AI.

I built a CLI tool that writes my git commits so I never have to again

phantomDev — Sat, 21 Feb 2026 11:45:01 +0000

Let me paint you a picture.
It's 1am. I've been coding for 4 hours straight. I'm in flow, fixing bugs, adding features, refactoring stuff that was embarrassing to look at. Then I finally decide to push and I stare at the git commit prompt like it personally offended me.

What do I type? fix stuff? update? changes?

Yeah. We've all been there.

The worst part wasn't even the lazy messages. It was that I'd sometimes go days without pushing because I'd get so deep into coding that committing felt like an interruption. Then I'd look at my GitHub contribution graph and it looked like I hadn't touched a computer in a week. Meanwhile I'd written 2000 lines of code.

So I did what any reasonable developer would do at 1am — I decided to build a tool to fix it instead of just... committing.

💡 The idea

The concept is simple: what if a tool just watched my code, noticed when I saved files, figured out what changed, and wrote the commit message for me?

I'd just have to press Y.

That's it. That's the whole pitch.

I called it phantomit because it commits silently in the background like a ghost. Also because I thought it was a cool name at 1am and I stand by that decision.

⚙️ How it works

Here's the basic flow:

You run phantomit watch --on-save
It watches your project directory using chokidar
When you save files, it waits 8 seconds (debounce window)
After the window closes, it runs git add . then git diff --staged
That diff goes to Groq's llama-3.1-8b-instant with a strict prompt
The AI returns a conventional commit message (10-20 words, nothing more)
You see this in your terminal:

[11:42 PM] ✎ src/auth.ts, ✚ src/middleware.ts

✦ Commit message:
"feat(auth): add JWT validation middleware with token expiry handling"

[Y] commit & push   [E] edit   [N] skip

→

You press Y. It commits. It pushes. You go back to coding.

That's literally it.

🐛 The problems I didn't expect

Okay here's the part where it gets interesting. Building the concept took maybe an hour. The edge cases took the rest of the night.

Saving two files felt like saving one and a half

My original debounce was 2 seconds. Save auth.ts, wait 1.8 seconds, save middleware.ts — boom, two separate commit triggers. The first one commits, the second one gets skipped because the first is still processing.

So I increased the debounce to 8 seconds and made it configurable. Now if you save 5 files within 8 seconds they all batch into one commit. One clean diff, one good message.

The race condition nobody warned me about

Here's a fun one. What if you delete a file and save another file at exactly the same time?

Old behavior: first event triggers, commit starts, second event comes in while isCommitting = true, gets silently dropped. You just lost a change.

The fix was a commit queue. If a commit is already running when a new trigger fires, it queues up and runs immediately after. Nothing gets dropped. I felt very smart when I figured this out and then immediately felt dumb for not thinking of it from the start.

Chokidar doesn't read .gitignore

This one annoyed me. Chokidar is great for watching files but it has no idea what a .gitignore is. So by default phantomit would pick up changes in node_modules, .env, dist — all the stuff you'd never want to commit.

I pulled in the ignore npm package which parses .gitignore properly — including negations (!important.js), globs (**/*.log), comments, the whole spec. Now phantomit automatically reads your .gitignore and merges it with whatever extra patterns you define in .phantomit.json.

Your .env is safe. You're welcome.

Only catching "save" events is naive

My first version only listened for change events — file edits. But what about:

Creating a new file? (add)
Deleting a file? (unlink)
Creating a folder? (addDir)
Deleting a folder? (unlinkDir)

All of these are meaningful changes that deserve to be in a commit. So I added all 5 event types and now the terminal shows you exactly what happened:

[9:14 AM] ✚ src/newfile.ts, ✖ src/old.ts, ✎ src/index.ts

Much better context for the AI too — it knows you deleted something, not just edited it.

🤖 The AI part

I'm using Groq because it's genuinely fast and has a free tier that's more than enough for personal use. The model is llama-3.1-8b-instant — small, quick, doesn't overthink it.

The prompt is strict:

You are a Git commit message generator.
Rules:
- Use conventional commits format: type(scope): description
- Types: feat, fix, refactor, chore, docs, style, test, perf
- Keep it between 10-20 words
- Be specific and descriptive, not vague
- No period at the end
- Output ONLY the commit message, nothing else

The "output ONLY the commit message" is doing a lot of work there. Without it you get things like:

"Here is a commit message for your changes: feat(auth)..."

No thanks.

I also truncate diffs over 6000 characters because sending a 50kb diff to an LLM is both slow and expensive and the model doesn't need to read your entire codebase to write a 15 word sentence.

🔑 Key rotation (my favorite feature nobody asked for)

Groq's free tier has rate limits. If you're coding heavily you might hit them.

My solution: support unlimited API keys picked at random on every commit.

GROQ_API_KEY_1=first_key
GROQ_API_KEY_2=second_key
GROQ_API_KEY_3=third_key
# add as many as you want

How it works under the hood:

Object.entries(process.env).forEach(([key, value]) => {
  if (key.startsWith('GROQ_API_KEY_') && value) keys.push(value);
});

Just loops through every env variable matching the pattern. No hardcoded limit. You could have GROQ_API_KEY_99 and it would still work. Each commit picks one at random.

Three free Groq accounts = effectively triple the rate limit. I call that a feature.

🎛️ Three watch modes

Different devs work differently so I built three modes:

--on-save — commits 8 seconds after your last file save. This is the one I use personally.

phantomit watch --on-save
phantomit watch --on-save --daemon  # background, silent

--every <minutes> — commits on a fixed interval if there are changes. Set it and forget it.

phantomit watch --every 30

--lines <count> — commits when your accumulated diff crosses a line threshold.

phantomit watch --lines 20

And if you just want the AI message without any automation:

phantomit push        # generate + commit right now
phantomit push --mock # test without a Groq key

🚀 Install it

npm install -g phantomit-cli

Note: the package is phantomit-cli on npm because some package called phantomjs exists and npm decided our names were too similar, which is honestly a bit offensive. But the command is still just phantomit.

Then:

cd your-project
phantomit init
echo "GROQ_API_KEY=your_key" >> .env
phantomit watch --on-save

Get a free Groq key at console.groq.com. Takes about 30 seconds.

📸 What the commit messages actually look like

Here are some real ones generated during development of phantomit itself (yes, I used phantomit to build phantomit, which felt very appropriate):

feat(ai): implement mock mode for generateCommitMessage function
fix(test): Remove test file as it is no longer needed
feat(test): add test generation feature with mock mode support
fix(test): Remove unused test file and refactor generateCommitMessage function
fix(ai): Update AI commit message generation to use new model

Not perfect but genuinely better than update or misc changes. And I typed zero of those.

🔮 What's next

A few things I want to add:

Auto-retry with next key if one Groq key hits a rate limit instead of just failing
Better daemon logging with timestamps and session stats
Custom prompt templates so you can tune the commit style to your team's conventions

If you want to contribute or have ideas, the repo is open: github.com/var-raphael/phantomit

Docs at phantomit-docs.vercel.app

That's it. Go install it, stop writing commit messages, and put that mental energy toward literally anything else.

Your GitHub graph will thank you. 📈

Raphael, 18, coding on a mobile phone in Nigeria at 1am 🇳🇬