How I check the tech stack of hundreds of websites with Python

Devarq — Mon, 05 Oct 2026 09:47:19 +0000

I'd just finished building a tech stack lookup on Apify, and before trusting it on domains I don't know, I wanted to see what it said about ones I do. So I picked 20, a mix of big SaaS companies and Shopify stores, and ran them through it.

The lookup is mine, so take my enthusiasm with a pinch of salt. It reads three things for each domain:

The homepage HTML, headers and cookies, matched against the open-source Wappalyzer fingerprints (about 7,600 technologies)
The DNS records: MX for the email provider, SPF for sending tools, and the TXT "domain verification" records companies add when they sign up for things like Atlassian or DocuSign
WHOIS and the SSL certificate (switched off below, I only cared about the stack)

The DNS part ended up mattering more than I thought. About 18% of homepages block automated requests, but a store can't hide that its www CNAME points at Shopify.

The script

pip install apify-client

import os
from collections import Counter
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

domains = [
    "stripe.com", "shopify.com", "notion.so", "linear.app", "vercel.com",
    "airtable.com", "figma.com", "canva.com", "zapier.com", "calendly.com",
    "allbirds.com", "gymshark.com", "glossier.com", "basecamp.com", "ghost.org",
    "wordpress.org", "hubspot.com", "intercom.com", "mailchimp.com", "loom.com",
]

run = client.actor("devarq/domain-tech-stack-whois-dns").call(
    run_input={"domains": domains, "includeWhois": False, "includeTls": False}
)
rows = client.dataset(run.default_dataset_id).list_items().items

for row in rows:
    print(f'{row["domain"]:14} {row["cms"] or row["ecommerce_platform"] or "-":16} '
          f'{row["email_provider"] or "-":17} {row["tech_count"]:>3} technologies')

tools = Counter(name for row in rows for name in row["tech_names"])
print("\nMost common across all 20:")
for name, count in tools.most_common(10):
    print(f"  {count:2}  {name}")

You need an Apify token for this. The free plan's $5 of monthly credit is plenty for a test like this one. The run took about 11 seconds.

What came back

shopify.com    Shopify          Google Workspace   33 technologies
stripe.com     -                Google Workspace   25 technologies
airtable.com   -                Google Workspace   31 technologies
notion.so      Contentful       -                  12 technologies
figma.com      -                Google Workspace   36 technologies
vercel.com     -                Google Workspace   27 technologies
calendly.com   -                Google Workspace   37 technologies
canva.com      -                Google Workspace   34 technologies
gymshark.com   Shopify          Proofpoint         22 technologies
zapier.com     Contentful       Google Workspace   35 technologies
linear.app     -                Google Workspace   29 technologies
allbirds.com   Shopify          Microsoft 365       9 technologies
basecamp.com   -                -                  11 technologies
glossier.com   Shopify          Google Workspace   19 technologies
ghost.org      -                Google Workspace   16 technologies
wordpress.org  WordPress        -                  14 technologies
mailchimp.com  -                -                  23 technologies
loom.com       -                Proofpoint         26 technologies
intercom.com   -                Google Workspace   25 technologies
hubspot.com    HubSpot CMS Hub  Google Workspace   33 technologies

Most common across all 20:
  20  Google Search Console
  16  Apple
  16  HSTS
  16  Open Graph
  15  Google Workspace
  15  Microsoft 365
  14  Atlassian
  14  DocuSign
  13  Amazon Web Services
  13  Cloudflare

The Microsoft line surprised me more than it should have. 15 of the 20 have a Microsoft 365 verification record, but only Allbirds actually runs its email on Microsoft. Everyone else is on Google Workspace and verified their domain with Microsoft for Teams or Azure or whatever. If you're sorting leads by "uses Microsoft", those are very different things.

Atlassian and DocuSign in 14 of 20 was the other one. None of that is on a homepage, it's all DNS.

Allbirds only shows 9 technologies because its homepage blocked the request. It still comes out as a Shopify store, from DNS alone, which is the bit you'd actually want.

And yes, the CMS column is mostly empty. Big SaaS sites run their own Next.js or React frontends, there's nothing to name.

What broke while I was writing this

This is the part I didn't plan on writing.

My first run of a related email-security check said mailchimp.com and oracle.com had no SPF record. That's obviously wrong for a company that sends email for a living. It turned out Cloudflare's DNS-over-HTTPS endpoint truncates large TXT answers (mailchimp.com has 34 TXT records) and sets a TC flag that my code ignored. The SPF record was one of the ones that got cut. Now a truncated answer means "ask Google's resolver instead", and both domains show up fine.

The hosting column was worse. It said HubSpot and Ghost were hosted on "Atlassian Statuspage", because the fingerprint data files Statuspage under PaaS and my code believed it. Looking closer, most of that column was guessing hosting from DNS servers and email senders, so I rewrote it to only trust the site's own response headers or the network its IP belongs to. It's blank more often now, but it's right when it isn't.

Every detection comes with an evidence field showing the exact header, script or DNS record that matched, which is how I track this stuff down:

{
  "name": "Atlassian",
  "categories": ["Development", "SaaS"],
  "confidence": 100,
  "evidence": "dns:txt=atlassian-domain-verification=upLp21qQ...",
  "source": "dns"
}

Cost and the catch

$3 per 1,000 domains. If you only want Shopify stores out of a big list, "onlyWithTechnologies": ["Shopify"] drops the rest at a third of the price.

The catch: it reads the HTML the server sends and doesn't run a browser, so anything injected purely by JavaScript can be missed. And a DNS verification record means a company set something up at some point, not that they still pay for it.

It's on Apify as Tech Stack & WHOIS Lookup, and there's a Shopify store example you can run without writing any code.

If you give it a go, I'd really appreciate hearing how it went. Open an issue on the Actor page if something looks wrong (detection mistakes are what I fix first), and if it saves you some time, a quick review there helps more than you'd think.

DEV Community: Devarq

How I check the tech stack of hundreds of websites with Python

The script

What came back

What broke while I was writing this

Cost and the catch