How I qualify web-design prospects with an outdated-website checker (no scraping, robots.txt-respecting)

Disclosure: I built the tool used in this post, Website Redesign Signals, and it's a paid actor on Apify. The checks described here are simple enough that you can copy the ideas into your own script if you'd rather not use it.

The problem: 200 tabs and a generic first line

If you sell web design to small businesses, you probably already have a list of prospects: a CRM export, a spreadsheet from a networking event, past enquiries. The slow part isn't finding names. It's working out which of those sites actually need work and what to say in the first line of your email.

Doing that by hand means opening every homepage, resizing the window to check mobile, scrolling to the footer, viewing the source for jQuery versions... and then still writing "I love your website, but..." because you ran out of patience.

So I wrote a small checker that does the boring part: fetch each homepage, look for a handful of observable signals, give it a 0-100 "outdated" score, and write one neutral, factual sentence I can verify in 20 seconds before using it.

Which signals are worth checking

I deliberately stuck to things that are visible in the public HTML or HTTP responses and that a business owner would recognise if you pointed them out:

Signal	Why it matters	How it's detected
No mobile viewport tag	Phones render a shrunken desktop page. Strongest single signal of an old build.	No <meta name="viewport" content="width=device-width...">
Stale © year	"© 2018" in the footer suggests nobody has touched the site in years	Regex on footer text; years written by a script (new Date().getFullYear()) are recognised and not penalised
Old jQuery / Bootstrap / CMS	jQuery 1.x, Bootstrap 3, WordPress < 6.0 point to an old theme	Versions from public script/style URLs, library banners and the generator tag (only if the site exposes it)
HTTPS problems	Plain http only, certificate errors, or http that doesn't forward to https	Try https:// first, fall back to http://, one extra request to check the redirect
Broken internal links	A 404 behind "Services" is a concrete, fixable thing to mention	HEAD (with GET fallback) on up to 25 homepage links
Heavy images	A 3 MB hero JPEG is slow on mobile data	HEAD Content-Length on homepage images, top 3 by size
No enquiry form / not tap-to-call	Conversion basics	Looks for forms on the homepage and the linked contact page; checks for any tel: link (true/false only)

Things I left out on purpose: Lighthouse/Core Web Vitals (slow and expensive per site), anything that needs a login, and anything that probes paths or ports. It's not a security scanner, it only reads what a normal visitor's browser receives.

The polite part (no scraping)

This was a hard requirement for me:

• You bring the list. It doesn't discover businesses or scrape Google Maps, directories or social networks.
• robots.txt is checked before every request, under its own user-agent token (RedesignSignalsBot), and the User-Agent header says what it is.
• At most the homepage plus 2 more pages per site (contact page, optionally one services page). Link and image checks are lightweight HEAD requests with low per-site concurrency.
• No personal data in the output. No emails, phone numbers or names. Contact signals are booleans. Link text that looks like an email or phone number is dropped.

How the 0-100 score works

It's an additive, explainable heuristic. Each finding adds points, and the total is capped at 100:

no mobile viewport            +25
HTTPS certificate error       +22
plain http only               +20
© year 5+ years old           +15   (3-4 years: +10, 2 years: +4)
Flash                         +15
broken homepage links         +4 each, max +14
jQuery 1.x                    +10   (2.x: +7, < 3.5: +3)
Bootstrap 3 or older          +8
old WordPress/Joomla/Drupal   +8
http doesn't redirect to https +8
image over 1 MB               +7   (over 500 KB: +3)
no enquiry form               +6
missing <title>               +5
missing meta description      +4
no tap-to-call link           +3

Every point comes with a reasons[] entry (code, points, plain-English detail), so you never have to trust a mystery number. In my own testing I use it roughly like this:

• 40+: strong redesign prospect
• 15-39: fix-level issues, worth a short "quick fix" email
• under 15: modern enough, skip

The outreach hook is built from the two highest-scoring reasons that have a plain-language phrase, and it avoids scare words like "hacked" or "insecure".

A real example (domain anonymised)

This is an abridged record from one of my test runs against a UK roofing company's homepage. I've replaced the domain:

{
  "domain": "www.roofer-example.co.uk",
  "finalUrl": "https://www.roofer-example.co.uk/",
  "status": "ok",
  "httpsOk": true,
  "httpRedirectsToHttps": true,
  "hasViewportMeta": false,
  "copyrightYear": null,
  "cms": "Weebly",
  "jqueryVersion": "1.8.3",
  "metaDescriptionMissing": false,
  "largestImages": [{ "url": "https://www.roofer-example.co.uk/uploads/logo.png", "kb": 245 }],
  "brokenInternalLinks": [],
  "hasContactOrBookingForm": true,
  "hasTapToCall": false,
  "outdatedScore": 38,
  "reasons": [
    { "code": "no_viewport", "points": 25, "detail": "No <meta name=\"viewport\" content=\"width=device-width\"> tag" },
    { "code": "old_jquery", "points": 10, "detail": "jQuery 1.8.3 (1.x line, no longer supported)" },
    { "code": "no_tap_to_call", "points": 3, "detail": "No tap-to-call (tel:) link on the homepage" }
  ],
  "outreachHook": "The site has no mobile viewport tag and it still loads jQuery 1.8.3.",
  "primaryIssueType": "NOVIEWPORT"
}

Note what it didn't penalise: HTTPS works and redirects properly, the contact page has a form, there's no footer year to judge. 25 + 10 + 3 = 38, a fix-level-to-redesign borderline case, and the hook is something I can confirm in a few seconds by opening the site on my phone.

Running it from the Apify API

You'll need an Apify account and an API token. Keep the token in an environment variable, never in your code.

Python

pip install apify-client
export APIFY_TOKEN="your-token-here"   # placeholder, set your own

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("shanmukhkiransagar/website-redesign-signals").call(
    run_input={
        "domains": ["example.co.uk", "https://www.another-example.com/"],
        "maxPagesPerSite": 2,      # homepage + contact page
        "checkBrokenLinks": True,
        "checkImages": True,
    }
)

rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
rows.sort(key=lambda r: r.get("outdatedScore") or 0, reverse=True)

for r in rows:
    print(f'{r.get("outdatedScore", 0):>3}  {r["domain"]:<35} {r.get("outreachHook") or "-"}')

JavaScript (Node)

npm install apify-client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('shanmukhkiransagar/website-redesign-signals').call({
  domains: ['example.co.uk', 'https://www.another-example.com/'],
  maxPagesPerSite: 2,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items
  .filter((r) => r.status === 'ok' && r.outdatedScore >= 40)
  .forEach((r) => console.log(r.outdatedScore, r.domain, r.outreachHook));

From there you can export the rows to your CRM and segment by primaryIssueType (NOVIEWPORT, HTTP, BROKEN, DATED, HEAVY, NOFORM...), so each batch of emails talks about one specific thing.

Limitations (honestly)

• It doesn't run JavaScript. It reads HTML over plain HTTP, so content rendered only by scripts isn't seen. A form injected by a script can be missed, which means "no enquiry form" can be a false positive on JS-heavy sites.
• No page-weight totals, load times or Core Web Vitals. Image sizes come from HEAD requests on homepage images only.
• CMS versions only when the site exposes them. Many don't.
• Bot-protected sites (e.g. Cloudflare challenge pages) can come back as http_error 403/503.
• The weights are mine and haven't been calibrated against campaign reply rates. In a pre-launch test on 10 real UK small-business homepages, the main issue matched my manual audit on 7 of 10. Treat it as a way to prioritise, not a verdict.
• Check every hook by eye before sending. Sites change. And if you use it for cold email, follow the rules where your recipient is (PECR in the UK, CAN-SPAM in the US, CASL in Canada).

Try it

The actor is here: Website Redesign Signals on Apify. It's pay-per-event at $0.015 per site analysed ($15 per 1,000), and sites it couldn't reach, robots.txt skips, duplicates and invalid inputs aren't charged. You can set a maximum charge per run.

If you'd rather build your own version, the table above is basically the spec: start with the viewport check and the © year regex, they catch a surprising number of neglected sites on their own.

I'd love feedback on the scoring weights, especially from anyone who has reply-rate data on which observations actually land with small-business owners.