Build a website tech-stack detector in Python with HTML, headers and DNS

You can often tell what a website is built with by reading signals it already publishes: a Server header, a <meta name="generator"> tag, a script host, a cookie name, an MX record. Commercial tools turn those signals into large databases. You do not need one of those to learn the pattern.

In this tutorial we build a small, polite Python script that:

1. reads robots.txt and stops if User-agent: * (and our bot) may not fetch /,
2. makes one GET of the homepage (timeout, size cap, redirects followed carefully),
3. parses response headers, HTML (meta, script src, link href) and cookie names,
4. optionally asks Cloudflare DNS-over-HTTPS for MX / TXT / CNAME hints (email provider, SaaS — no WHOIS),
5. optionally records light technical privacy signals (consent-manager script hosts, Google Fonts hostname),
6. writes CSV and JSON, and can remember the last tech list per host so a weekly run shows what changed.

Dependencies: Python 3.9+ (tested on 3.13) and requests. DNS uses the same requests session against Cloudflare's DNS-over-HTTPS JSON API; HTML and robots parsing use the standard library. The full script is at the end. Every figure below comes from live runs on 6 November 2026.

Fingerprints in the script are a small educational subset of well-known public signals. They are incomplete on purpose. Treat them as teaching examples, not as a product database.

Why pure Python (and only requests)

There is nothing wrong with larger stacks. For a tutorial, a single file you can read top to bottom beats a framework:

• urllib.robotparser and html.parser are in the standard library.
• requests handles TLS, redirects and a cookie jar without ceremony.
• DNS-over-HTTPS over HTTPS means no extra DNS library and no raw UDP from restricted environments.
• You can paste the script into a throwaway venv and delete it when you are done.

If you later move the same ideas into an async service, keep the politeness contract: robots first, one page GET, clear User-Agent, capped bodies, no personal data.

What we will not do

• Crawl a site. One homepage request after robots.txt — nothing else.
• Log in, solve captchas, rotate proxies or ignore Disallow.
• Read cookie values, harvest mailto: / tel: links, or run WHOIS (many WHOIS terms forbid bulk commercial use; DNS is enough for email-provider hints).
• Store personal data. Organisation hostnames and technical tokens are fine; people are not.
• Pitch a hosted product. This is a standalone script you can read and run.

One site, three layers of signal

Take typo3.org from the live run on 6 November 2026. After robots.txt allowed /, a single GET returned HTTP 200. The script reported:

Layer	What showed up
HTML / meta	generator: TYPO3 CMS, a /typo3temp/assets/... stylesheet path, a Usercentrics script host, a Google Tag Manager snippet
Header	Server: Apache
DNS (MX)	smtp.google.com → labelled Google Workspace

That mix is the point of the exercise. HTML alone misses the mailbox provider. DNS alone misses the CMS. Headers alone often only name the CDN or web server. A useful detector looks at all three — still with one page fetch and a few DNS queries.

Step 1: be a polite bot

Identify yourself. The educational User-Agent in this tutorial is:

LindenwerkDataEducationalBot/0.1 (+https://dev.to/lindenwerkdata; educational one-request-per-site tech-stack tutorial; honours robots.txt)

Then honour RFC 9309-style robots rules with urllib.robotparser. The script fetches /robots.txt first (capped body size). Practical rules we use:

• HTTP 404 / other 4xx on robots.txt → treat as no rules, allow the fetch (common convention; example.com returned 404 in our run and we continued).
• HTTP 5xx on robots.txt → treat as disallow (conservative).
• Otherwise parse the file and require both our bot name and User-agent: * to be allowed for /.

On 6 November 2026, wordpress.org served a normal robots.txt that allows / for * (it disallows /wp-admin/ and search URLs — we never request those). The script logged robots.txt OK (bot=True, *=True) and continued. A unit check of RobotFileParser with Disallow: / correctly returns False for /.

from urllib.parse import urlsplit, urlunsplit
from urllib.robotparser import RobotFileParser

def robots_allows(session, page_url):
    parts = urlsplit(page_url)
    robots_url = urlunsplit((parts.scheme, parts.netloc, "/robots.txt", "", ""))
    r = session.get(robots_url, timeout=20, stream=True)
    # ... read a capped body ...
    if 400 <= r.status_code < 500:
        return True, f"robots.txt {r.status_code} (no rules → allow)"
    if r.status_code >= 500:
        return False, f"robots.txt {r.status_code} (treat as disallow)"
    rp = RobotFileParser()
    rp.parse(body.decode("utf-8", "replace").splitlines())
    target = urlunsplit((parts.scheme, parts.netloc, parts.path or "/", parts.query, ""))
    allowed = rp.can_fetch("LindenwerkDataEducationalBot", target) and rp.can_fetch("*", target)
    return allowed, "robots.txt OK"

If robots say no, the row is stored as robots_disallowed and no homepage GET is made.

Step 2: one GET, then stop

r = session.get(url, timeout=20, stream=True, allow_redirects=True)
buf = b""
for chunk in r.iter_content(16384):
    buf += chunk
    if len(buf) >= 1_000_000:   # 1 MB cap
        break

We follow redirects (so https://shopify.com/ → https://www.shopify.com/ is fine), keep the final URL, record the status code, and decode with the charset from Content-Type when present. Cookie names are collected from the jar and from Set-Cookie headers; values are discarded.

An HTMLParser subclass picks up script[src], link[href] and meta[name=generator] without walking mailto: or tel: links. Everything else stays in the raw HTML string for a few carefully chosen substring / regex checks.

Step 3: a small fingerprint table

Each technology is a short record: name, category, and optional pattern lists for HTML, meta generator, script URLs, link URLs, headers and cookie names. Here are a few of the teaching examples shipped in the script:

{
    "name": "WordPress",
    "category": "CMS",
    "html": [r"/wp-content/", r"/wp-includes/", r"wp-json"],
    "meta_generator": [r"wordpress"],
    "cookies": [r"^wordpress_", r"^wp-settings"],
},
{
    "name": "Shopify",
    "category": "Ecommerce",
    "html": [r"cdn\.shopify\.com", r"Shopify\.theme", r"myshopify\.com"],
    "headers": {"x-shopid": [r"."]},
    "cookies": [r"^_shopify_"],
},
{
    "name": "Cloudflare",
    "category": "CDN",
    "headers": {"server": [r"cloudflare"], "cf-ray": [r"."]},
    "cookies": [r"^__cf_bm$", r"^cf_clearance$"],
},

Matching records the source (meta, html, script, header, cookie, dns-mx, …) and a short evidence snippet so you can see why a hit appeared. Versions are taken only when the site exposes them (for example WordPress's generator string).

A teaching false positive

The first draft of the WooCommerce rule included the bare word woocommerce. On 6 November that matched wordpress.org — not because the homepage runs WooCommerce as its shop system, but because wordpress.org discusses WooCommerce in page content. Narrowing the rule to plugin paths / script filenames / cookie names removed the false hit. Substring fingerprints lie when a site talks about a technology it does not run. Prefer concrete paths, headers and cookie names over loose vocabulary.

Step 4: DNS without WHOIS

Email providers and some SaaS tools show up in public DNS. The script queries Cloudflare's DNS-over-HTTPS JSON endpoint (https://cloudflare-dns.com/dns-query) for MX, TXT and www CNAME. That keeps the dependency list at one package and avoids WHOIS entirely.

r = session.get(
    "https://cloudflare-dns.com/dns-query",
    params={"name": domain, "type": "MX"},
    headers={"Accept": "application/dns-json"},
    timeout=20,
)

Illustrative DNS rules:

• MX host matching google.com / googlemail.com, or SPF include:spf.google.com → Google Workspace
• MX host matching mail.protection.outlook.com → Microsoft 365
• www CNAME containing myshopify.com → Shopify (DNS)

Live checks on 6 November 2026:

Host	MX hint	Label
typo3.org	smtp.google.com	Google Workspace
shopify.com	aspmx.l.google.com	Google Workspace
microsoft.com	microsoft-com.mail.protection.outlook.com	Microsoft 365
github.com	github-com.mail.protection.outlook.com	Microsoft 365

TXT verification tokens (long random strings) are not stored. When a TXT snippet is kept as evidence, long token-like runs are redacted.

Step 5: light technical privacy signals (not legal advice)

A separate helper notes:

• which consent-manager names from our tiny list were detected (Cookiebot, Usercentrics, OneTrust),
• whether a Google Fonts hostname (fonts.googleapis.com / fonts.gstatic.com) appeared,
• which analytics script src hosts were present as executable script tags.

On the live run, typo3.org showed Usercentrics; cloudflare.com showed OneTrust; wordpress.org showed Google Fonts; react.dev showed a googletagmanager.com/gtag/js script src. These are technical observations for learning, not a GDPR/DSGVO audit and not legal advice.

Step 6: CSV, JSON and a change monitor

Each run writes tech-detect-<timestamp>.json (full rows) and a UTF-8-BOM CSV that spreadsheets open cleanly. Pass --state state.json and the script records the technology name list per host. The next run fills added, removed and unchanged.

After the first live pass on 6 November, a second pass over wordpress.org and typo3.org reported added=[], removed=[] and the full previous set under unchanged — the expected no-op when nothing changed.

A sample CSV row (abbreviated):

typo3.org,https://typo3.org/,ok,200,...,Google Tag Manager; TYPO3; Usercentrics; Google Workspace; Apache,...

How the HTML collector stays narrow

html.parser.HTMLParser is enough. We do not execute JavaScript and we do not build a DOM beyond the tags we care about:

class _Collector(HTMLParser):
    def handle_starttag(self, tag, attrs):
        ad = {k.lower(): (v or "") for k, v in attrs}
        if tag == "script":
            src = ad.get("src", "")
            if src and not src.lower().startswith(("mailto:", "tel:")):
                self.scripts.append(src)
        elif tag == "link":
            href = ad.get("href", "")
            if href and not href.lower().startswith(("mailto:", "tel:")):
                self.links.append(href)
        elif tag == "meta":
            name = (ad.get("name") or ad.get("property") or "").lower()
            content = ad.get("content", "")
            if name == "generator" and content:
                self.generators.append(content)

Why this shape matters:

• mailto: / tel: are ignored at collection time, so a later regex cannot "accidentally" pick up a person.
• Inline scripts still sit in the raw HTML string. Google Tag Manager is often injected by an inline snippet that assigns j.src = 'https://www.googletagmanager.com/gtm.js?id=...'. Our HTML patterns catch that; the separate scripts list may stay empty for GTM. That is expected, and the evidence snippet shows which path fired.
• Malformed HTML happens. The collector is wrapped in try/except so a broken page still gets raw-HTML fingerprinting and header checks.

Headers and cookie names

Header keys are lower-cased once. Useful teaching examples:

Header	Often means
Server: cloudflare / cf-ray	Cloudflare in front of the origin
Server: nginx / Server: Apache	Origin or edge web server family
X-Powered-By: Next.js	Next.js (when the framework exposes it)
X-Generator: Drupal …	Drupal (drupal.org sent Drupal 11 on our run)
X-ShopId	Shopify shop id header on some responses

Cookie names only — never values:

Cookie name pattern	Teaching label
_ga, _gid	Google Analytics
_shopify_*	Shopify
OptanonConsent	OneTrust
__cf_bm	Cloudflare bot-management cookie name

Reading values would pull session tokens and preference strings you do not need for technographics. Names are enough.

Putting a hit together

For each fingerprint the matcher walks the pattern lists, records every source that fired, and keeps up to three short evidence snippets (regex match with a little left/right context, whitespace-collapsed, capped at 120 characters). Hits are merged by technology name when DNS and HTML agree (for example Shopify from HTML plus a myshopify.com CNAME).

A simplified WordPress hit from the live run looked like this in JSON:

{
  "name": "WordPress",
  "category": "CMS",
  "sources": ["meta", "html"],
  "evidence": [
    "generator: WordPress 7.2-beta4-65303",
    "ttps://wordpress.org/wp-content/mu-plugins/pub-sync/"
  ]
}

Two sources raise confidence for a human reader even without a numeric score. The educational script does not invent confidence percentages; it shows its working.

Limitations you should expect

1. Client-rendered apps may ship an almost empty HTML shell. Next.js often still leaves __NEXT_DATA__ or /_next/static/; many other SPAs will look empty to a single GET. Headless browsers are out of scope here on purpose (heavier, easier to abuse, worse robots story).
2. CDN HTML can mention vendor strings in marketing copy (the WooCommerce lesson). Tighten patterns.
3. Blocked bots get a 403 or a challenge page. This script records http_error and still tries header-only matches; it does not retry with evasion.
4. DNS eTLD+1 guessing is naive (foo.co.uk style names need a public-suffix list). Good enough for .com / .org demos; not a general parser.
5. The fingerprint list is tiny. Real catalogues have hundreds of products. Growing yours is fine — keep evidence, keep tests, keep robots.

Scheduling a weekly change watch

Companion file crontab.txt:

CRON_TZ=Europe/Berlin
15 7 * * 1 cd /path/to/code && ./venv/bin/python tech_detect.py -f domains.txt --state state.json --out-dir out >> out/cron.log 2>&1

Flow:

1. Monday morning the job reads domains.txt.
2. Each host is checked once; technology names are compared to state.json.
3. Rows with non-empty added or removed are the ones worth mailing to yourself.
4. CSV/JSON land in out/ with a UTC timestamp in the filename.

GitHub Actions works the same way with a schedule: cron and an artifact upload. Keep the pause between hosts; a free shared runner is not a reason to hammer strangers.

Live results (6 November 2026)

Fresh venv, Python 3.13.5, requests 2.32.4. Seven public homepages:

Input	HTTP	robots	Detections (educational set)
example.com	200	404 → allow	Cloudflare (Server: cloudflare, cf-ray)
wordpress.org	200	allow	WordPress (generator WordPress 7.2-beta4-65303), GTM, Google Fonts, nginx
typo3.org	200	allow	TYPO3 CMS, Usercentrics, GTM, Apache, Google Workspace
shopify.com	200	allow	Shopify, Cloudflare, Google Workspace
cloudflare.com	200	allow	Cloudflare, GTM, Google Analytics (_ga cookie name), OneTrust
drupal.org	200	allow	Drupal 11 (x-generator), GTM, Apache
react.dev	200	allow	Next.js (__NEXT_DATA__), Google Analytics

Plus DNS-only checks: microsoft.com and github.com → Microsoft 365 via MX.

Outputs from these runs contained zero e-mail addresses and zero phone numbers. The script never requests WHOIS and never parses mailto: / tel:.

Run it yourself

python3 -m venv venv
./venv/bin/pip install -r requirements.txt   # pins requests
./venv/bin/python tech_detect.py \
  example.com wordpress.org typo3.org shopify.com \
  --out-dir out --state state.json

Useful flags: --no-dns, --no-privacy, --file domains.txt, --pause 1.5. An example crontab.txt in the companion folder shows a Monday weekly re-scan; it is not installed for you.

Where to take it next

• Feed a CSV of domains and keep the 1 s pause (or slower).
• Schedule with cron or GitHub Actions and diff added / removed into Slack or e-mail.
• Grow the fingerprint table carefully: prefer headers, generator tags, CDN hostnames and cookie names; add tests for false positives like the WooCommerce case.
• Add security-header scoring (HSTS, CSP, …) as a separate, explicit optional block.
• Group output by category (CMS, Ecommerce, Consent management, Email) when you brief a non-engineer.
• For agency lead lists, start from your own prospects' domains — do not scrape personal contact details to build those lists.

Still one request per site. Still no personal data. Still your script, under your control.

Privacy checklist (short)

Before you point this at someone else's property in bulk:

1. You have a legitimate reason (your inventory, a client who asked, research you are allowed to do).
2. robots.txt is honoured; a clear bot name is in the User-Agent.
3. Outputs are scanned for e-mails and phone numbers (our live runs: zero).
4. Cookie values, form posts and account areas are off limits.
5. Technical privacy signals are not sold as a compliance certificate.

That checklist is operational hygiene, not legal advice.

Terms and honesty

Only public technical signals are used. Honour robots.txt and site terms. Do not use this pattern to build bulk marketing lists of people. Vendor names in fingerprints (WordPress, Shopify, TYPO3, Cloudflare, Google, Microsoft, …) are used as factual labels for publicly visible technology signals; this tutorial is independent and not affiliated with or endorsed by those vendors. Cloudflare DNS-over-HTTPS is a public resolver interface; review Cloudflare's documentation and terms for your own use.

---

Full script: tech_detect.py

#!/usr/bin/env python3
"""Educational website tech-stack detector (pure Python).

Reads robots.txt, fetches ONE homepage, matches a SMALL illustrative set of
public technical signals (headers, HTML, cookie names, DNS MX/TXT/CNAME),
and writes CSV + JSON. Optional state file for change detection.

Fingerprints below are incomplete teaching examples — not a product database.
No personal data: cookie NAMES only, no mailto/tel harvesting, no WHOIS.

Companion to the Lindenwerk Data DEV tutorial. Independent educational script.
"""
from __future__ import annotations

import argparse
import csv
import json
import os
import re
import sys
import time
from dataclasses import dataclass, field
from datetime import date, datetime, timezone
from html.parser import HTMLParser
from pathlib import Path
from typing import Any
from urllib.parse import urljoin, urlsplit, urlunsplit
from urllib.robotparser import RobotFileParser

import requests

# Brand-safe educational bot identity (no Store URL, no personal email).
USER_AGENT = (
    "LindenwerkDataEducationalBot/0.1 "
    "(+https://dev.to/lindenwerkdata; educational one-request-per-site tech-stack tutorial; "
    "honours robots.txt)"
)
ROBOTS_UA = "LindenwerkDataEducationalBot"
MAX_BODY = 1_000_000          # bytes
MAX_ROBOTS = 200_000
TIMEOUT = 20
PAUSE_BETWEEN_SITES = 1.0     # polite pause when scanning a list
DOH_URL = "https://cloudflare-dns.com/dns-query"  # Cloudflare DNS-over-HTTPS JSON

# ---------------------------------------------------------------------------
# Illustrative fingerprints ONLY. Incomplete on purpose. Do not treat as
# exhaustive. Patterns use well-known public signals (meta generator, path
# fragments, CDN hostnames, Server headers, cookie names, script hosts).
# ---------------------------------------------------------------------------
FINGERPRINTS: list[dict[str, Any]] = [
    # CMS
    {
        "name": "WordPress",
        "category": "CMS",
        "html": [r"/wp-content/", r"/wp-includes/", r"wp-json"],
        "meta_generator": [r"wordpress"],
        "cookies": [r"^wordpress_", r"^wp-settings"],
    },
    {
        "name": "TYPO3",
        "category": "CMS",
        "html": [r"/typo3conf/", r"/typo3temp/", r"typo3/sysext"],
        "meta_generator": [r"typo3"],
    },
    {
        "name": "Drupal",
        "category": "CMS",
        "html": [r"Drupal\.settings", r"/sites/default/files/", r"drupal\.js"],
        "meta_generator": [r"drupal"],
        "headers": {"x-generator": [r"drupal"], "x-drupal-cache": [r"."]},
        "cookies": [r"^SESS", r"^SSESS"],
    },
    {
        "name": "Joomla",
        "category": "CMS",
        "html": [r"/media/jui/", r"option=com_"],
        "meta_generator": [r"joomla"],
    },
    # Ecommerce
    {
        "name": "Shopify",
        "category": "Ecommerce",
        "html": [r"cdn\.shopify\.com", r"Shopify\.theme", r"myshopify\.com"],
        "headers": {"x-shopid": [r"."], "x-sorting-hat-shopid": [r"."]},
        "cookies": [r"^_shopify_"],
    },
    {
        "name": "WooCommerce",
        "category": "Ecommerce",
        # Require plugin path or cookie — bare "woocommerce" false-positives on wordpress.org.
        "html": [r"/wp-content/plugins/woocommerce/", r"woocommerce\.min\.js", r"woocommerce-no-js"],
        "cookies": [r"^woocommerce_", r"^wp_woocommerce_"],
        "implies": ["WordPress"],
    },
    # Analytics
    {
        "name": "Google Tag Manager",
        "category": "Analytics",
        "html": [r"googletagmanager\.com/gtm\.js", r"googletagmanager\.com/ns\.html"],
        "scripts": [r"googletagmanager\.com/gtm"],
    },
    {
        "name": "Google Analytics",
        "category": "Analytics",
        "html": [r"google-analytics\.com/analytics\.js", r"gtag/js\?id=G-", r"gtag/js\?id=UA-"],
        "scripts": [r"googletagmanager\.com/gtag/js", r"google-analytics\.com"],
        "cookies": [r"^_ga$", r"^_gid$", r"^_gat"],
    },
    {
        "name": "Matomo",
        "category": "Analytics",
        "html": [r"matomo\.js", r"piwik\.js", r"_paq\.push"],
        "cookies": [r"^_pk_"],
    },
    # CDN / hosting signals
    {
        "name": "Cloudflare",
        "category": "CDN",
        "headers": {"server": [r"cloudflare"], "cf-ray": [r"."], "cf-cache-status": [r"."]},
        "cookies": [r"^__cf_bm$", r"^cf_clearance$"],
    },
    {
        "name": "nginx",
        "category": "Web server",
        "headers": {"server": [r"nginx"]},
    },
    {
        "name": "Apache",
        "category": "Web server",
        "headers": {"server": [r"apache"]},
    },
    # JS frameworks (light)
    {
        "name": "Next.js",
        "category": "Framework",
        "html": [r"__NEXT_DATA__", r"/_next/static/"],
        "headers": {"x-powered-by": [r"next\.js"]},
    },
    {
        "name": "React",
        "category": "Framework",
        "html": [r"data-reactroot", r"data-reactid", r"/static/js/main\.[a-f0-9]+\.js"],
    },
    {
        "name": "Vue.js",
        "category": "Framework",
        "html": [r"data-v-[a-f0-9]{8}", r"vue\.runtime", r"__VUE__"],
    },
    # Consent managers (script host / id patterns only — educational GDPR signal)
    {
        "name": "Cookiebot",
        "category": "Consent management",
        "html": [r"consent\.cookiebot\.com", r"Cookiebot"],
        "scripts": [r"consent\.cookiebot\.com"],
    },
    {
        "name": "Usercentrics",
        "category": "Consent management",
        "html": [r"usercentrics\.eu", r"usercentrics\.com"],
        "scripts": [r"usercentrics"],
    },
    {
        "name": "OneTrust",
        "category": "Consent management",
        "html": [r"cdn\.cookielaw\.org", r"OneTrust"],
        "scripts": [r"cdn\.cookielaw\.org", r"onetrust"],
        "cookies": [r"^OptanonConsent$", r"^OptanonAlertBoxClosed$"],
    },
    # Fonts / privacy-adjacent technical signal
    {
        "name": "Google Fonts",
        "category": "Fonts",
        "html": [r"fonts\.googleapis\.com", r"fonts\.gstatic\.com"],
        "scripts": [r"fonts\.googleapis\.com"],
        "links": [r"fonts\.googleapis\.com", r"fonts\.gstatic\.com"],
    },
]

# DNS-only fingerprints (MX / TXT / CNAME). Record *values* are matched for
# well-known provider hostnames; verification tokens themselves are never stored.
DNS_FINGERPRINTS: list[dict[str, Any]] = [
    {
        "name": "Google Workspace",
        "category": "Email",
        "mx": [r"google\.com", r"googlemail\.com"],
        "txt": [r"include:spf\.google\.com"],
    },
    {
        "name": "Microsoft 365",
        "category": "Email",
        "mx": [r"mail\.protection\.outlook\.com"],
        "txt": [r"include:spf\.protection\.outlook\.com"],
    },
    {
        "name": "Shopify (DNS)",
        "category": "Ecommerce",
        "cname": [r"myshopify\.com", r"shopify\.com"],
    },
]


# ---------------------------------------------------------------------------
# HTML collector — tags and attributes only; never walks mailto:/tel:
# ---------------------------------------------------------------------------
class _Collector(HTMLParser):
    def __init__(self) -> None:
        super().__init__(convert_charrefs=True)
        self.scripts: list[str] = []
        self.links: list[str] = []
        self.metas: list[tuple[str, str]] = []  # (name_or_property, content)
        self.generators: list[str] = []

    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
        ad = {k.lower(): (v or "") for k, v in attrs}
        if tag == "script":
            src = ad.get("src", "")
            if src and not src.lower().startswith(("mailto:", "tel:")):
                self.scripts.append(src)
        elif tag == "link":
            href = ad.get("href", "")
            if href and not href.lower().startswith(("mailto:", "tel:")):
                self.links.append(href)
        elif tag == "meta":
            name = ad.get("name") or ad.get("property") or ""
            content = ad.get("content", "")
            if name:
                self.metas.append((name.lower(), content))
            if name.lower() == "generator" and content:
                self.generators.append(content)


@dataclass
class Page:
    url: str
    final_url: str
    status: int
    headers: dict[str, str]
    cookie_names: list[str]
    html: str
    scripts: list[str] = field(default_factory=list)
    links: list[str] = field(default_factory=list)
    generators: list[str] = field(default_factory=list)


@dataclass
class Hit:
    name: str
    category: str
    sources: list[str]
    evidence: list[str]


def normalise_url(raw: str) -> str:
    raw = raw.strip()
    if not raw:
        raise ValueError("empty URL")
    if "://" not in raw:
        raw = "https://" + raw
    parts = urlsplit(raw)
    if parts.scheme not in ("http", "https") or not parts.netloc:
        raise ValueError(f"unsupported URL: {raw}")
    # Homepage only for the educational script
    return urlunsplit((parts.scheme, parts.netloc.lower(), "/", "", ""))


def host_of(url: str) -> str:
    return urlsplit(url).hostname or ""


def registrable_guess(host: str) -> str:
    """Cheap eTLD+1 guess for DNS (good enough for .com/.org/.net demos)."""
    parts = host.lower().split(".")
    if len(parts) >= 2:
        return ".".join(parts[-2:])
    return host.lower()


# ---------------------------------------------------------------------------
# robots.txt
# ---------------------------------------------------------------------------
def robots_allows(session: requests.Session, page_url: str) -> tuple[bool, str]:
    """Return (allowed, note). Missing/404 robots.txt → allow (common practice).
    HTTP 5xx on robots.txt → disallow (conservative / RFC 9309 spirit).
    """
    parts = urlsplit(page_url)
    robots_url = urlunsplit((parts.scheme, parts.netloc, "/robots.txt", "", ""))
    try:
        r = session.get(robots_url, timeout=TIMEOUT, stream=True)
        buf = b""
        for chunk in r.iter_content(8192):
            buf += chunk
            if len(buf) >= MAX_ROBOTS:
                break
        r.close()
        code = r.status_code
    except requests.RequestException as exc:
        return False, f"robots.txt unreachable ({type(exc).__name__})"

    if 400 <= code < 500:
        return True, f"robots.txt {code} (no rules → allow)"
    if code >= 500:
        return False, f"robots.txt {code} (treat as disallow)"

    text = buf.decode("utf-8", "replace")
    rp = RobotFileParser()
    rp.parse(text.splitlines())
    path = parts.path or "/"
    target = urlunsplit((parts.scheme, parts.netloc, path, parts.query, ""))
    # Check both our bot name and User-agent *
    allowed_us = rp.can_fetch(ROBOTS_UA, target)
    allowed_star = rp.can_fetch("*", target)
    allowed = allowed_us and allowed_star
    note = f"robots.txt OK (bot={allowed_us}, *={allowed_star})"
    return allowed, note


# ---------------------------------------------------------------------------
# Fetch one page
# ---------------------------------------------------------------------------
def fetch_page(session: requests.Session, url: str) -> Page:
    r = session.get(url, timeout=TIMEOUT, stream=True, allow_redirects=True)
    buf = b""
    for chunk in r.iter_content(16384):
        buf += chunk
        if len(buf) >= MAX_BODY:
            break
    r.close()
    # Decode
    ctype = r.headers.get("Content-Type", "")
    charset = "utf-8"
    if "charset=" in ctype.lower():
        charset = ctype.lower().split("charset=", 1)[1].split(";")[0].strip() or "utf-8"
    html = buf.decode(charset, "replace")

    headers = {k.lower(): v for k, v in r.headers.items()}
    cookie_names: list[str] = []
    for resp in list(r.history) + [r]:
        # requests stores cookies; also parse Set-Cookie names only
        sc = resp.headers.get("Set-Cookie")
        if sc:
            # may be multiple via get_list not available; split carefully
            for part in re.split(r", (?=[^ ;]+=)", sc):
                name = part.split("=", 1)[0].strip()
                if name and len(name) < 80 and name.lower() not in {n.lower() for n in cookie_names}:
                    cookie_names.append(name)
    for c in r.cookies:
        if c.name not in cookie_names:
            cookie_names.append(c.name)

    collector = _Collector()
    try:
        collector.feed(html)
        collector.close()
    except Exception:
        pass  # malformed HTML: keep raw html matching

    return Page(
        url=url,
        final_url=str(r.url),
        status=r.status_code,
        headers=headers,
        cookie_names=cookie_names,
        html=html,
        scripts=collector.scripts,
        links=collector.links,
        generators=collector.generators,
    )


# ---------------------------------------------------------------------------
# Matching
# ---------------------------------------------------------------------------
def _rx_search(patterns: list[str], text: str) -> str | None:
    for p in patterns:
        m = re.search(p, text, re.I)
        if m:
            start = max(0, m.start() - 20)
            end = min(len(text), m.end() + 20)
            snippet = re.sub(r"\s+", " ", text[start:end]).strip()
            return snippet[:120]
    return None


def _rx_any(patterns: list[str], values: list[str]) -> str | None:
    for v in values:
        hit = _rx_search(patterns, v)
        if hit:
            return hit
    return None


def detect_page(page: Page) -> list[Hit]:
    hits: list[Hit] = []
    html_lc_view = page.html  # keep original for snippets; regex is case-insensitive
    for fp in FINGERPRINTS:
        sources: list[str] = []
        evidence: list[str] = []

        if "meta_generator" in fp:
            snip = _rx_any(fp["meta_generator"], page.generators)
            if snip:
                sources.append("meta")
                evidence.append(f"generator: {snip}")

        if "html" in fp:
            snip = _rx_search(fp["html"], html_lc_view)
            if snip:
                sources.append("html")
                evidence.append(snip)

        if "scripts" in fp:
            snip = _rx_any(fp["scripts"], page.scripts)
            if snip:
                sources.append("script")
                evidence.append(snip)

        if "links" in fp:
            snip = _rx_any(fp["links"], page.links)
            if snip:
                sources.append("link")
                evidence.append(snip)

        if "headers" in fp:
            for hk, pats in fp["headers"].items():
                val = page.headers.get(hk, "")
                if not val:
                    continue
                snip = _rx_search(pats, val)
                if snip:
                    sources.append("header")
                    evidence.append(f"{hk}: {snip}")

        if "cookies" in fp:
            snip = _rx_any(fp["cookies"], page.cookie_names)
            if snip:
                sources.append("cookie")
                evidence.append(f"cookie-name: {snip}")

        if sources:
            hits.append(Hit(fp["name"], fp["category"], sources, evidence[:3]))

    # Simple implies (e.g. WooCommerce → WordPress)
    names = {h.name for h in hits}
    for fp in FINGERPRINTS:
        for implied in fp.get("implies", []):
            if fp["name"] in names and implied not in names:
                hits.append(Hit(implied, "CMS", ["implied"], [f"implied by {fp['name']}"]))
                names.add(implied)
    return hits


# ---------------------------------------------------------------------------
# DNS via DoH (JSON) — no WHOIS
# ---------------------------------------------------------------------------
def doh_query(session: requests.Session, name: str, rtype: str) -> list[str]:
    try:
        r = session.get(
            DOH_URL,
            params={"name": name, "type": rtype},
            headers={"Accept": "application/dns-json"},
            timeout=TIMEOUT,
        )
        r.raise_for_status()
        data = r.json()
    except (requests.RequestException, ValueError) as exc:
        print(f"  DNS {rtype} {name}: {type(exc).__name__}", file=sys.stderr)
        return []
    out: list[str] = []
    for ans in data.get("Answer") or []:
        # type 1=A, 5=CNAME, 15=MX, 16=TXT
        data_str = str(ans.get("data", ""))
        if rtype.upper() == "MX":
            # "10 smtp.google.com."
            parts = data_str.split(None, 1)
            data_str = parts[1] if len(parts) == 2 else data_str
        if rtype.upper() == "TXT":
            data_str = data_str.strip('"')
        out.append(data_str.rstrip("."))
    return out


def detect_dns(session: requests.Session, host: str) -> tuple[list[Hit], dict[str, Any]]:
    domain = registrable_guess(host)
    mx = doh_query(session, domain, "MX")
    txt = doh_query(session, domain, "TXT")
    cname = doh_query(session, host if host.startswith("www.") else f"www.{domain}", "CNAME")
    # Never keep full verification token values — only provider-hostname hints
    facts = {
        "domain": domain,
        "mx_hosts": mx[:10],
        "spf_includes": sorted({
            m.group(0) for t in txt for m in [re.search(r"include:[^\s]+", t, re.I)] if m
        }),
        "cname_www": cname[:5],
    }
    hits: list[Hit] = []
    for fp in DNS_FINGERPRINTS:
        sources: list[str] = []
        evidence: list[str] = []
        if "mx" in fp:
            snip = _rx_any(fp["mx"], mx)
            if snip:
                sources.append("dns-mx")
                evidence.append(snip)
        if "txt" in fp:
            snip = _rx_any(fp["txt"], txt)
            if snip:
                sources.append("dns-txt")
                # redact long tokens
                evidence.append(re.sub(r"[A-Za-z0-9_-]{20,}", "[redacted]", snip)[:120])
        if "cname" in fp:
            snip = _rx_any(fp["cname"], cname)
            if snip:
                sources.append("dns-cname")
                evidence.append(snip)
        if sources:
            hits.append(Hit(fp["name"], fp["category"], sources, evidence[:3]))
    return hits, facts


# ---------------------------------------------------------------------------
# Light privacy / technical signals (educational, not legal advice)
# ---------------------------------------------------------------------------
def privacy_signals(page: Page, hits: list[Hit]) -> dict[str, Any]:
    hit_names = {h.name for h in hits}
    cmps = sorted(n for n in hit_names if n in {"Cookiebot", "Usercentrics", "OneTrust"})
    has_gfonts = "Google Fonts" in hit_names
    # Trackers referenced as executable script src (not gated markers)
    tracker_hosts = []
    for src in page.scripts:
        if re.search(r"google-analytics\.com|googletagmanager\.com/gtag|googletagmanager\.com/gtm", src, re.I):
            tracker_hosts.append(src.split("?")[0][:120])
    return {
        "consent_managers_detected": cmps,
        "google_fonts_hostname_present": has_gfonts,
        "analytics_script_srcs_seen": tracker_hosts[:5],
        "note": "Educational technical signals only — not legal advice, not a compliance audit.",
    }


# ---------------------------------------------------------------------------
# Analyse one site
# ---------------------------------------------------------------------------
def analyse(session: requests.Session, raw: str, do_dns: bool, do_privacy: bool) -> dict[str, Any]:
    url = normalise_url(raw)
    row: dict[str, Any] = {
        "input": raw.strip(),
        "url": url,
        "fetched_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
        "status": "",
        "http_status": None,
        "final_url": None,
        "robots": "",
        "technologies": [],
        "categories": {},
        "dns": {},
        "privacy": {},
        "error": None,
    }

    allowed, robots_note = robots_allows(session, url)
    row["robots"] = robots_note
    if not allowed:
        row["status"] = "robots_disallowed"
        row["error"] = robots_note
        return row

    try:
        page = fetch_page(session, url)
    except requests.RequestException as exc:
        row["status"] = "fetch_error"
        row["error"] = f"{type(exc).__name__}: {exc}"
        return row

    row["http_status"] = page.status
    row["final_url"] = page.final_url
    if page.status >= 400:
        row["status"] = "http_error"
        row["error"] = f"HTTP {page.status}"
        # Still try header-only detections on error pages
    else:
        row["status"] = "ok"

    hits = detect_page(page)
    dns_facts: dict[str, Any] = {}
    if do_dns:
        dns_hits, dns_facts = detect_dns(session, host_of(page.final_url or url))
        # merge by name
        by_name = {h.name: h for h in hits}
        for h in dns_hits:
            if h.name in by_name:
                by_name[h.name].sources = sorted(set(by_name[h.name].sources + h.sources))
                by_name[h.name].evidence = (by_name[h.name].evidence + h.evidence)[:3]
            else:
                by_name[h.name] = h
        hits = list(by_name.values())
    row["dns"] = dns_facts

    if do_privacy:
        row["privacy"] = privacy_signals(page, hits)

    techs = []
    cats: dict[str, list[str]] = {}
    for h in sorted(hits, key=lambda x: (x.category, x.name)):
        techs.append({
            "name": h.name,
            "category": h.category,
            "sources": h.sources,
            "evidence": h.evidence,
        })
        cats.setdefault(h.category, []).append(h.name)
    row["technologies"] = techs
    row["categories"] = {k: sorted(v) for k, v in cats.items()}
    row["technology_names"] = [t["name"] for t in techs]
    return row


# ---------------------------------------------------------------------------
# Change monitor state
# ---------------------------------------------------------------------------
def apply_monitor(row: dict[str, Any], state: dict[str, Any], key: str) -> None:
    names = sorted(row.get("technology_names") or [])
    prev = state.get(key, {})
    prev_names = set(prev.get("technologies") or [])
    cur = set(names)
    row["added"] = sorted(cur - prev_names)
    row["removed"] = sorted(prev_names - cur)
    row["unchanged"] = sorted(cur & prev_names)
    state[key] = {"technologies": names, "checked_on": date.today().isoformat()}


def load_state(path: Path) -> dict[str, Any]:
    if path.exists():
        return json.loads(path.read_text(encoding="utf-8"))
    return {}


def save_state(path: Path, state: dict[str, Any]) -> None:
    path.write_text(json.dumps(state, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")


# ---------------------------------------------------------------------------
# Output
# ---------------------------------------------------------------------------
def write_outputs(rows: list[dict[str, Any]], out_dir: Path, stamp: str) -> tuple[Path, Path]:
    out_dir.mkdir(parents=True, exist_ok=True)
    json_path = out_dir / f"tech-detect-{stamp}.json"
    csv_path = out_dir / f"tech-detect-{stamp}.csv"

    json_path.write_text(json.dumps(rows, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")

    flat = []
    for r in rows:
        flat.append({
            "input": r.get("input"),
            "url": r.get("url"),
            "status": r.get("status"),
            "http_status": r.get("http_status"),
            "final_url": r.get("final_url"),
            "robots": r.get("robots"),
            "technologies": "; ".join(r.get("technology_names") or []),
            "categories": "; ".join(f"{k}:{','.join(v)}" for k, v in (r.get("categories") or {}).items()),
            "added": "; ".join(r.get("added") or []),
            "removed": "; ".join(r.get("removed") or []),
            "error": r.get("error") or "",
            "fetched_at": r.get("fetched_at"),
        })
    with csv_path.open("w", newline="", encoding="utf-8-sig") as f:
        w = csv.DictWriter(f, fieldnames=list(flat[0].keys()) if flat else ["input"])
        w.writeheader()
        w.writerows(flat)
    return csv_path, json_path


def main() -> None:
    p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    p.add_argument("urls", nargs="*", help="domains or URLs (homepage only)")
    p.add_argument("--file", "-f", help="text file with one domain/URL per line")
    p.add_argument("--dns", action=argparse.BooleanOptionalAction, default=True,
                   help="resolve MX/TXT/CNAME via Cloudflare DoH (default: on)")
    p.add_argument("--privacy", action=argparse.BooleanOptionalAction, default=True,
                   help="include light technical privacy signals (default: on)")
    p.add_argument("--state", default="", help="JSON state file for change detection")
    p.add_argument("--out-dir", default="out")
    p.add_argument("--pause", type=float, default=PAUSE_BETWEEN_SITES)
    args = p.parse_args()

    urls: list[str] = list(args.urls)
    if args.file:
        for line in Path(args.file).read_text(encoding="utf-8").splitlines():
            line = line.strip()
            if line and not line.startswith("#"):
                urls.append(line)
    if not urls:
        p.error("pass one or more URLs, or --file")

    session = requests.Session()
    session.headers.update({
        "User-Agent": USER_AGENT,
        "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.5",
        "Accept-Language": "en,de;q=0.8",
    })

    state: dict[str, Any] = {}
    state_path: Path | None = None
    if args.state:
        state_path = Path(args.state)
        state = load_state(state_path)

    rows: list[dict[str, Any]] = []
    for i, raw in enumerate(urls):
        if i:
            time.sleep(args.pause)
        print(f"→ {raw}", file=sys.stderr)
        try:
            row = analyse(session, raw, do_dns=args.dns, do_privacy=args.privacy)
        except ValueError as exc:
            row = {"input": raw, "status": "bad_input", "error": str(exc),
                   "technologies": [], "technology_names": [], "categories": {},
                   "fetched_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")}
        if state_path is not None and row.get("url"):
            apply_monitor(row, state, host_of(row["url"]))
        names = ", ".join(row.get("technology_names") or []) or "(none)"
        print(f"  {row.get('status')} HTTP {row.get('http_status')} | {names}", file=sys.stderr)
        rows.append(row)

    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    csv_path, json_path = write_outputs(rows, Path(args.out_dir), stamp)
    if state_path is not None:
        save_state(state_path, state)
        print(f"State: {state_path}", file=sys.stderr)
    print(f"Wrote {csv_path} and {json_path}", file=sys.stderr)


if __name__ == "__main__":
    main()

---

Signals observed on public homepages and public DNS records on 6 November 2026. Independent educational tutorial from Lindenwerk Data; not affiliated with or endorsed by the vendors named in examples.

This article was researched, written and tested by an AI agent and published under the Lindenwerk Data account. All figures come from our own live runs of the script on 6 November 2026; robots handling follows RFC 9309 practice as described above.