How to find out what technology a website uses, in bulk

#sales #leadgen #python #api

Sales teams and agencies qualify prospects by technology: every Shopify store using a certain email tool, WordPress sites without a chat widget, companies on a particular CDN. Browser extensions answer the question for one site at a time. When you have a list of 5,000 domains, you need the answer as a table.

Tech Stack Detector by Hay Equipos takes a list of domains and returns, for each one, the CMS, ecommerce platform, analytics and tag managers, payment providers, live chat widgets, JavaScript and web frameworks, CDN, hosting and web server, with versions where the site exposes them.

How it works

For each domain, the actor fetches the public home page and its response headers, plus robots.txt, and matches them against about 3,600 technology fingerprints from an open source, MIT licensed fingerprint database. It checks headers, cookie names, meta tags, script sources, inline scripts, CSS, HTML and page structure, then runs a second pass for technologies that only make sense when another one is present.

No browser is started, which keeps it fast and cheap. The actor uses an honest user agent, respects robots.txt by default, sends no cookies, needs no login, and returns technology names only, with no personal data.

What you get back

One row per domain. Trimmed example with illustrative values:

{
  "domain": "example-shop.com",
  "finalUrl": "https://www.example-shop.com/",
  "statusCode": 200,
  "title": "Example Shop: Outdoor Gear",
  "technologyCount": 11,
  "cms": [],
  "ecommerce": ["Shopify"],
  "analytics": ["Google Analytics"],
  "tagManagers": ["Google Tag Manager"],
  "payments": ["PayPal", "Apple Pay"],
  "liveChat": [],
  "cdn": ["Cloudflare"],
  "security": ["HSTS"],
  "technologies": [
    { "name": "Shopify", "version": null, "confidence": 100, "categories": ["Ecommerce"], "website": "https://www.shopify.com" },
    { "name": "jQuery", "version": "3.6.0", "confidence": 100, "categories": ["JavaScript libraries"], "website": "https://jquery.com" }
  ],
  "possiblyBlocked": false,
  "error": null,
  "checkedAt": "2026-09-27T05:45:12.000Z"
}

Every row has the same summary columns, which makes filtering in a spreadsheet easy: cms, ecommerce, analytics, tagManagers, payments, liveChat, marketingAutomation, crm, advertising, jsFrameworks, webFrameworks, uiFrameworks, jsLibraries, cdn, hosting, webServers, programmingLanguages, security, cookieCompliance, abTesting and reviews. The technologies array holds the full detection list with version, confidence and categories.

Step by step in the Apify Console

Open the actor on the Apify Store (link at the end) and click Try for free.
In Domains, paste domains or URLs, one per line. Only the home page of each domain is checked, and duplicates are removed.
Keep Respect robots.txt on. A domain whose robots.txt closes its home page to crawlers is skipped and not charged.
Set Minimum confidence (default 50, from 0 to 100) to control how strict detections are.
Leave Include the full technology list on, or switch it off for a compact table with only the summary columns.
Set Domains checked in parallel (default 5, up to 20) and Request timeout (default 20 seconds). Parallelism is also capped by the run's memory, so give large runs more memory if you want more checks at once.
Click Start, then export to CSV or Excel and filter on the summary columns.

Calling it from code

With curl:

curl -X POST \
  "https://api.apify.com/v2/acts/pistachio_implementation~tech-stack-detector/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "domains": ["example-shop.com", "example.org"],
    "minConfidence": 50,
    "includeAllTechnologies": false
  }'

With Python and the apify-client package, finding Shopify stores in a lead list:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

domains = ["example-shop.com", "example.org", "store.example.net"]

run = client.actor("pistachio_implementation/tech-stack-detector").call(
    run_input={
        "domains": domains,
        "maxConcurrency": 10,
        "includeAllTechnologies": True,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["error"]:
        print("skipped", row["domain"], row["error"])
    elif "Shopify" in row["ecommerce"]:
        print(row["domain"], "payments:", row["payments"], "chat:", row["liveChat"])

Pricing

Pay per event, with no subscription and no platform usage charge on top:

Event	Price
Actor start (Apify's standard start event)	$0.00005 per run
Domain analyzed	$0.002 per domain ($2 per 1,000 domains)

A domain is charged only when the site actually served its home page with a status below 400. Domains that do not resolve, time out, answer with an error or bot wall status, or are skipped because of robots.txt are free. You can set a maximum charge per run and the actor stops when it is reached.

Limits and what it does not do

Static analysis only. Tools injected later by a tag manager or loaded on scroll (analytics loaded through a tag manager, many chat widgets) do not appear in the page source and may be missed. The tag manager itself is detected. This is the most common reason a technology you know about is missing.
Home page only. A shop on a subdomain such as shop.example.com should be entered separately.
Bot walls. Some large sites block every automated visitor. Those rows come back with the HTTP status, possiblyBlocked: true and whatever could be read from the headers.
Fixed fingerprint snapshot. The fingerprints come from a release dated January 2023. Technologies launched since then may be missing or named differently.
No DNS or TLS certificate checks.

The actor reads only public pages. Use the results in line with each site's terms.

Try it on the Apify Store: https://apify.com/pistachio_implementation/tech-stack-detector

DEV Community