What I learned building 20 small web-checking tools for the Apify Store

#apify #webdev #seo #javascript

I'm a computer science student in France. Over the last two weeks I built and published 20 small "Actors" (serverless tools) on the Apify Store: SSL certificate checks, broken link crawling, SPF/DMARC audits, website screenshots, OCR, sitemap extraction, brand colors and a few more. Each one is pay-per-use, usually between $0.50 and $2 per 1,000 checks.

Writing the happy path was the easy part. Here are the bugs that only showed up when I ran the tools against real websites.

1. A "broken link" is often just a bot protection

My broken link crawler reported 47 broken internal links on a big French news site. None of them were broken. After a few requests, the site started answering 402 Payment Required and 406 Not Acceptable to every robot.

False positives are the worst possible failure for a link checker, so:

401, 402, 403, 406, 429, 999 are now reported as blocked, not broken;
after 10 bot-protection answers in a row, the crawler leaves the site alone and writes one free summary row;
it respects robots.txt and its Crawl-delay, with at most 2 requests per second per site.

2. Don't charge for an empty page

A screenshot of a big news site came back as a white page saying "Unknown Error", with HTTP status 200. It was billed like a real screenshot. Now an error page (HTTP 400+) or an empty page (no title, almost no text, no image) is still saved, but not charged, with the reason in the output.

3. Every URL input is an SSRF risk

If users can give any URL, someone will try http://169.254.169.254/ or a public hostname that resolves to 127.0.0.1. Checking the hostname before the request is not enough (DNS rebinding). The check has to run on the IP address actually used for the connection: a custom DNS lookup in undici, a local proxy for headless Chrome, and a custom network backend in Python.

4. OCR needs a white margin

Tesseract found zero words in a GIF with one clear word on a white background. It needed a white border around the image. Transparent PNGs were worse: converted to grayscale, the text drawn with the alpha channel became a black page. The fix is to flatten transparency onto white, then add a margin.

5. Next.js is hard to detect without running JavaScript

The Wappalyzer fingerprints detect Next.js through __NEXT_DATA__, which the App Router no longer sets. Reading only HTML and headers, nextjs.org itself was not detected. /_next/static/ scripts, self.__next_f.push and x-nextjs-* headers do the job.

6. Test with hard inputs, not just examples

What found most of these bugs: running every tool on purpose-made hard inputs (IDN domains, subdomains, bot-protected sites, transparent images, several sites in one run) and reading every output row, not only the run status.

The tools

All of them are on my Apify profile: https://apify.com/propped_cloak. For example:

Broken Link Checker: crawl a whole site, with the page where each dead link was found, and an HTML report
SSL Certificate Checker: expiry dates of hundreds of hosts
Email Security Checker: SPF, DKIM, DMARC with a grade
Website Color Palette: brand colors of any site, with paste-ready CSS variables and a Tailwind v4 theme
Web Page to Markdown: clean Markdown for LLMs and RAG

They work from the Apify API, on a schedule, and from AI agents through the Apify MCP server. Feedback is welcome, especially bug reports.

This post was drafted with AI assistance and reviewed by me.

DEV Community