Detect the tech stack of 500 websites in one run (no browser, about $2.50)

#webdev #api #automation #tutorial

Say you have a list of 500 company websites and you want to know which run WordPress, which are on Shopify, who uses Cloudflare or Google Tag Manager. That's a typical prospecting, competitor-research or migration-audit question. Services like BuiltWith are great but priced for heavier use. For a quick, coarse pass, a single HTTP request per site gets you surprisingly far.

This post walks through doing that with an Apify actor I built, Website Tech Stack Detector. Disclosure: it's my actor and I earn a little when it's used.

How it works

For each URL it makes one plain HTTP request (no browser, no JavaScript execution), follows up to 5 redirects, and looks at:

response headers (server, cf-ray, x-vercel-id, ...)
<meta name="generator">
script URLs and a few well-known HTML markers (__NEXT_DATA__, /wp-content/)
cookie names (values are never read or stored)

About 60 technologies are recognised: CMS (WordPress, Drupal, Wix...), ecommerce (Shopify, WooCommerce, Magento...), analytics/tag managers, CDN/hosting, web servers, JS frameworks. Every detection includes the evidence that triggered it, so you can judge it yourself. It respects robots.txt by default.

Price

Pay-per-event: $0.005 per URL analyzed (at time of writing), so 500 sites is about $2.50. You're only charged for URLs fetched successfully (HTTP below 400). Timeouts, DNS failures, 4xx/5xx, non-public URLs and robots.txt skips are free. Compute is included.

Input

{
  "urls": [
    { "url": "https://wordpress.org" },
    { "url": "https://nextjs.org" },
    { "url": "https://www.drupal.org" },
    { "url": "example.com" }
  ],
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "timeoutSecs": 20
}

Bare domains are accepted. For 500 sites, paste or generate the urls array from your spreadsheet; maxConcurrency goes up to 20.

Real output

I ran the actor's source locally on 10 URLs: 8 analyzed, 2 not analyzed and not charged (a 404 page and http://127.0.0.1, which is rejected as non-public). Two of the results:

[
  {
    "input": "https://nextjs.org",
    "finalUrl": "https://nextjs.org",
    "httpStatus": 200,
    "technologies": [
      {
        "name": "Vercel",
        "category": "Hosting",
        "evidence": "header x-vercel-id"
      },
      {
        "name": "Next.js",
        "category": "JS framework",
        "evidence": "script src /\\/_next\\/static\\//"
      }
    ],
    "capturedAt": "2026-10-16T05:53:25.120Z"
  },
  {
    "input": "https://www.drupal.org",
    "finalUrl": "https://new.drupal.org/",
    "httpStatus": 200,
    "technologies": [
      {
        "name": "Drupal",
        "category": "CMS",
        "evidence": "meta generator \"Drupal 11 (https://www.drupal.org)\""
      },
      {
        "name": "Google Tag Manager",
        "category": "Tag manager",
        "evidence": "html GTM noscript iframe"
      },
      {
        "name": "Fastly",
        "category": "CDN",
        "evidence": "header x-served-by"
      },
      {
        "name": "Apache",
        "category": "Web server",
        "evidence": "header server"
      },
      {
        "name": "jQuery",
        "category": "JS library",
        "evidence": "script src /jquery([.-]\\d[\\d.]*)?(\\.min)?\\.js/"
      }
    ],
    "capturedAt": "2026-10-16T05:53:25.120Z"
  }
]```
{% endraw %}


Failed rows come back with an {% raw %}`error`{% endraw %} field instead of technologies, so you can retry or drop them.

## Be realistic about accuracy

This is a coarse tool, and my test run shows why:

- {% raw %}`https://www.shopify.com`{% endraw %} returned only **Cloudflare**. A single request saw nothing else. Absence of a detection is never proof a technology isn't used.
- {% raw %}`github.com`{% endraw %} came back as GitHub hosting plus Contentful, not a full stack.
- No JavaScript is executed, so tech that appears only after client-side rendering, or only on other pages, is missed. No version numbers are reported (except what a generator tag states).
- Sites behind bot walls or logins may return a block page. No proxies or captcha solving are used.

It's best for questions like "which of these 500 are on WordPress / Shopify / Cloudflare?", where the signal is in headers or the HTML, and weaker for building a complete technographic profile. I didn't run a 500-URL batch for this post, so I won't quote timing for one. Try 20 of your own sites first and compare with what you know.

## Call it from code

Small lists, one synchronous request:
{% raw %}


```bash
curl -X POST \
  "https://api.apify.com/v2/acts/pennydata~website-tech-stack-detector/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":[{"url":"https://wordpress.org"},{"url":"https://nextjs.org"}]}'

For 500 URLs use an async run with the Python client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
urls = [{"url": u.strip()} for u in open("sites.txt") if u.strip()]
run = client.actor("pennydata/website-tech-stack-detector").call(
    run_input={"urls": urls, "maxConcurrency": 10}
)
rows = client.dataset(run["defaultDatasetId"]).list_items().items

wordpress = [r["input"] for r in rows if "WordPress" in r.get("technologyNames", [])]
print(len(wordpress), "WordPress sites out of", len(rows))

technologyNames is a flat list for easy filtering; technologies has name, category and evidence per detection.

Wrap-up

For a few dollars you can tag a whole prospect list by CMS, ecommerce platform, CDN and analytics, with the evidence attached. Treat it as a first-pass filter and verify anything important by hand.

Written with AI (Claude).

DEV Community