Detect the tech stack of hundreds of websites in one API call (Python + Apify)

#webdev #python #webscraping #tutorial

Disclosure: I publish the Apify actor used below (pennydata/website-tech-stack-detector), and it was built with AI assistance (Claude). This is a how-to for one specific tool; it is pay-per-event, so check the price on the actor page before a big run. It is not affiliated with Wappalyzer or BuiltWith.

Say you have a spreadsheet of 500 company websites and want to know which ones run Shopify, WordPress or Next.js, which sit behind Cloudflare, and which load Google Tag Manager. Opening each in a browser with a devtools extension does not scale. This actor does it with one plain HTTP request per site, no browser.

1. Input

urls is the only required field. Bare domains work too.

{
  "urls": [
    { "url": "https://getbootstrap.com" },
    { "url": "example.com" }
  ],
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "timeoutSecs": 20
}

maxConcurrency is 1-20 and timeoutSecs is 5-60. With respectRobotsTxt on (the default), URLs disallowed for all bots are skipped and not charged.

2. Run it from Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

domains = ["getbootstrap.com", "nextjs.org", "wordpress.org"]  # your list, as long as you like
run = client.actor("pennydata/website-tech-stack-detector").call(
    run_input={
        "urls": [{"url": d} for d in domains],
        "respectRobotsTxt": True,
        "maxConcurrency": 10,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["input"], item.get("httpStatus"), item.get("technologyNames"))

3. What comes back

One item per URL. This is a real item from a test run on getbootstrap.com:

{
  "input": "https://getbootstrap.com",
  "finalUrl": "https://getbootstrap.com",
  "httpStatus": 200,
  "technologyNames": ["Cloudflare", "Fastly", "GitHub (hosting)", "Astro", "Bootstrap"],
  "technologies": [
    { "name": "Cloudflare", "category": "CDN", "evidence": "header cf-ray" },
    { "name": "Astro", "category": "JS framework", "evidence": "meta generator \"Astro v7.3.2\"" },
    { "name": "Bootstrap", "category": "UI framework", "evidence": "html bootstrap css" }
  ],
  "capturedAt": "2026-10-16T02:00:53.501Z"
}

(I trimmed technologies to three of its five entries.) The useful part is evidence: every detection says which header, meta tag, script URL, HTML marker or cookie name triggered it, so you can spot-check a result instead of trusting it blindly. Cookie names are read, cookie values never are.

4. Filter to the sites you care about

items = list(client.dataset(run["defaultDatasetId"]).iterate_items())

shopify = [i["finalUrl"] for i in items if "Shopify" in (i.get("technologyNames") or [])]

by_category = {}
for i in items:
    for t in i.get("technologies") or []:
        by_category.setdefault(t["category"], set()).add(t["name"])

print(len(shopify), "Shopify sites")
print({k: sorted(v) for k, v in by_category.items()})

Typical uses: qualifying leads by platform (agencies selling Shopify or WordPress work), spotting migration targets, or a quick competitor survey.

Limits worth knowing

One HTTP request to the URL you give, no JavaScript execution. Things that only appear after client-side rendering, or on other pages, can be missed. An empty list does not prove nothing is used.
Signature-based, about 60 common technologies, no version numbers. A CDN header may only reflect a proxy in front of the site. Treat results as strong hints, not an audit.
Only the first 1 MB of each page is inspected. Sites behind bot walls or logins may return a block page; there is no proxy rotation or captcha solving.
Only public http(s) URLs. You are responsible for having the right to analyze what you submit.
Failed URLs are not billed, and you can set a maximum cost per run in Apify's run options.

Actor page: https://apify.com/pennydata/website-tech-stack-detector

Written with AI (Claude).

DEV Community