I Built a CLI Tool That Checks Any Domain for Machine-Readable Infrastructure. Here's the Code.

Thorsten — Wed, 26 Aug 2026 23:36:08 +0000

I scan websites for a living. 42% of domains in my most recent batch returned zero valid JSON-LD. 82% served no llms.txt file. Finance, healthcare, government, e-commerce. The regulated industries.

Every retrieval pipeline assumes the source delivers structured data. I stopped assuming and started measuring. Five signals, one binary result, a CSV for batch runs. Below is the tool and every line of reasoning behind it.

What the tool checks

Five signals determine whether a domain provides the minimum infrastructure an automated system can rely on:

JSON-LD. Does the homepage contain valid application/ld+json blocks? Which Schema.org types?
llms.txt. Does the domain serve an llms.txt file at the root?
robots.txt. Does the domain allow generic bots? Does it explicitly block known AI crawlers?
HTTP headers. Does the server return proper Content-Type and security headers?
SSL/TLS. Does HTTPS work without certificate errors?

A domain passes if at least 3 of 5 checks succeed. The threshold is configurable in the code.

Requirements

Python 3.8+ and the requests library:

pip install requests

That is all. Everything runs locally, with zero external dependencies beyond requests.

Building the checker: function by function

JSON-LD detection

The homepage gets one GET request. Every <script type="application/ld+json"> block goes through json.loads. If the JSON parses and carries an @type, the check passes.

def check_json_ld(domain: str, session: requests.Session) -> dict:
    """Fetch the homepage and look for valid JSON-LD blocks."""
    url = f"https://{domain}"
    result = {
        "signal": "json_ld",
        "found": False,
        "valid": False,
        "types": [],
        "error": None,
    }

    try:
        resp = session.get(url, timeout=TIMEOUT, allow_redirects=True)
        resp.raise_for_status()
    except requests.RequestException as e:
        result["error"] = str(e)[:120]
        return result

    pattern = re.compile(
        r'<script[^>]+type=["\']application/ld\+json["\'][^>]*>'
        r'(.*?)</script>',
        re.DOTALL | re.IGNORECASE,
    )
    matches = pattern.findall(resp.text)

    if not matches:
        return result

    result["found"] = True
    types_found = []

    for raw in matches:
        try:
            data = json.loads(raw.strip())
            items = data if isinstance(data, list) else [data]
            for item in items:
                t = item.get("@type", "")
                if t:
                    types_found.append(
                        t if isinstance(t, str) else str(t)
                    )
        except (json.JSONDecodeError, AttributeError):
            continue

    if types_found:
        result["valid"] = True
        result["types"] = types_found

    return result

168 domains had a JSON-LD block in the HTML. The block existed. It contained empty types, broken nesting, truncated strings. A malformed block is the more dangerous case: your pipeline sees data, tries to parse it, and extracts garbage.

llms.txt presence

The llms.txt standard (proposed by Jeremy Howard) gives language models a machine-readable summary of a website. Adoption sits at 18% across 559 scanned domains in my most recent batch run.

def check_llms_txt(domain: str, session: requests.Session) -> dict:
    """Check whether the domain serves an llms.txt file at the root."""
    url = f"https://{domain}/llms.txt"
    result = {
        "signal": "llms_txt",
        "found": False,
        "size_bytes": 0,
        "error": None,
    }

    try:
        resp = session.get(url, timeout=TIMEOUT, allow_redirects=True)
        if (
            resp.status_code == 200
            and "text" in resp.headers.get("content-type", "")
        ):
            body = resp.text.strip()
            if len(body) > 20 and not body.startswith("<!"):
                result["found"] = True
                result["size_bytes"] = len(body.encode("utf-8"))
    except requests.RequestException as e:
        result["error"] = str(e)[:120]

    return result

Two guard clauses matter here. First, the content-type check filters out HTML error pages served with a 200 status. Second, body.startswith("<!") catches soft-404 pages that return an HTML document instead of a plain-text file.

robots.txt and AI bot directives

Two questions matter here. Can a generic bot crawl the site at all? And does the site explicitly shut the door on known AI crawlers like GPTBot, CCBot, ClaudeBot, or PerplexityBot?

KNOWN_AI_BOTS = [
    "gptbot", "chatgpt-user", "claudebot", "anthropic",
    "google-extended", "ccbot", "bytespider", "cohere-ai",
    "perplexitybot", "amazonbot",
]

def check_robots_txt(domain: str, session: requests.Session) -> dict:
    """Parse robots.txt for bot-related directives."""
    url = f"https://{domain}/robots.txt"
    result = {
        "signal": "robots_txt",
        "found": False,
        "allows_generic_bots": True,
        "blocks_ai_bots": False,
        "ai_bot_rules": [],
        "error": None,
    }

    try:
        resp = session.get(url, timeout=TIMEOUT, allow_redirects=True)
        if resp.status_code != 200:
            return result
        if "text" not in resp.headers.get("content-type", ""):
            return result
    except requests.RequestException as e:
        result["error"] = str(e)[:120]
        return result

    result["found"] = True
    lines = resp.text.lower().splitlines()
    current_agent = None

    for line in lines:
        line = line.split("#")[0].strip()
        if line.startswith("user-agent:"):
            current_agent = line.split(":", 1)[1].strip()
        elif line.startswith("disallow:") and current_agent:
            path = line.split(":", 1)[1].strip()
            if path == "/" and current_agent == "*":
                result["allows_generic_bots"] = False
            if path == "/" and current_agent in KNOWN_AI_BOTS:
                result["blocks_ai_bots"] = True
                result["ai_bot_rules"].append(current_agent)

    return result

KNOWN_AI_BOTS will grow. I update the list every time a new crawler shows up in my server logs.

HTTP headers

A domain that ships Strict-Transport-Security and X-Content-Type-Options runs a maintained stack. I track 10 sector lists. The pattern repeats in every single one: domains that fail on structured data also fail on security headers.

SECURITY_HEADERS = [
    "strict-transport-security",
    "x-content-type-options",
    "x-frame-options",
    "content-security-policy",
]

def check_headers(domain: str, session: requests.Session) -> dict:
    url = f"https://{domain}"
    result = {
        "signal": "headers",
        "status_code": None,
        "has_security_headers": False,
        "content_type_valid": False,
        "server": None,
        "error": None,
    }

    try:
        resp = session.head(url, timeout=TIMEOUT, allow_redirects=True)
        result["status_code"] = resp.status_code
        ct = resp.headers.get("content-type", "")
        result["content_type_valid"] = "text/html" in ct
        present = sum(
            1 for h in SECURITY_HEADERS if h in resp.headers
        )
        result["has_security_headers"] = present >= 2
        result["server"] = resp.headers.get("server", "")[:60]
    except requests.RequestException as e:
        result["error"] = str(e)[:120]

    return result

SSL/TLS validation

The shortest function in the file. If the HTTPS handshake breaks, everything else is academic.

def check_ssl(domain: str, session: requests.Session) -> dict:
    url = f"https://{domain}"
    result = {"signal": "ssl", "valid": False, "error": None}

    try:
        resp = session.head(url, timeout=TIMEOUT, allow_redirects=True)
        result["valid"] = True
    except requests.exceptions.SSLError as e:
        result["error"] = f"SSL error: {str(e)[:100]}"
    except requests.RequestException as e:
        result["error"] = str(e)[:100]

    return result

The verdict

A domain passes when at least 3 of 5 checks succeed. The threshold lives in a constant at the top of the script. Adjust it based on your pipeline's tolerance.

PASS_THRESHOLD = 3

def evaluate_domain(domain: str) -> dict:
    session = requests.Session()
    session.headers.update({"User-Agent": USER_AGENT})

    checks = {
        "json_ld": check_json_ld(domain, session),
        "llms_txt": check_llms_txt(domain, session),
        "robots_txt": check_robots_txt(domain, session),
        "headers": check_headers(domain, session),
        "ssl": check_ssl(domain, session),
    }

    passing = 0
    if checks["json_ld"]["found"] and checks["json_ld"]["valid"]:
        passing += 1
    if checks["llms_txt"]["found"]:
        passing += 1
    if (
        checks["robots_txt"]["found"]
        and checks["robots_txt"]["allows_generic_bots"]
    ):
        passing += 1
    if (
        checks["headers"]["content_type_valid"]
        and checks["headers"]["has_security_headers"]
    ):
        passing += 1
    if checks["ssl"]["valid"]:
        passing += 1

    verdict = "PASS" if passing >= PASS_THRESHOLD else "FAIL"

    return {
        "domain": domain,
        "verdict": verdict,
        "passing": passing,
        "total": 5,
        "checks": checks,
    }

Running it

Single domain

python infra-check.py example.com

Multiple domains

python infra-check.py example.com another.com third.com

Batch mode with file input

Create a domains.txt with one domain per line:

example.com
another.com
third.com

Then:

python infra-check.py --file domains.txt --csv results.csv

JSON export for pipeline integration

python infra-check.py --file domains.txt --json-out results.json

Quiet mode

python infra-check.py --file domains.txt -q --csv results.csv

Prints only the verdict line per domain. Useful for CI pipelines.

Sample output

Checking 3 domain(s)...

❌  example-bank.de  [FAIL]  (2/5 checks passed)
   JSON-LD:     missing
   llms.txt:    missing
   robots.txt:  open
   Headers:     HTTP 200, security headers: yes
   SSL/TLS:     valid

✅  example-saas.com  [PASS]  (4/5 checks passed)
   JSON-LD:     valid (Organization, WebSite)
   llms.txt:    found (2340 bytes)
   robots.txt:  open, blocks 2 AI bots
   Headers:     HTTP 200, security headers: yes
   SSL/TLS:     valid

❌  example-gov.de  [FAIL]  (1/5 checks passed)
   JSON-LD:     missing
   llms.txt:    missing
   robots.txt:  blocks all bots
   Headers:     HTTP 200, security headers: incomplete
   SSL/TLS:     valid

==================================================
Total: 3 | Passed: 1 | Failed: 2

What this covers

Five binary signals. One question: can an automated system extract structured data from this source?

This is a pre-flight check. My production scanner (SOVP) runs 180+ signals across 21 audit clusters and produces cryptographically signed attestations. This tool extracts 5 of those signals into a standalone script you can run without an account, an API key, or my infrastructure.

Content quality, factual accuracy, and freshness sit on a higher layer. This layer sits beneath all of them. If a domain fails here, your pipeline will burn tokens on extraction and get noise in return.

The user-agent string identifies itself honestly. The script waits 1 second between domains, follows redirects, and respects a 15-second timeout.

Get the full script

The complete infra-check.py with CLI argument parsing, CSV/JSON export, and deduplication:

litzki-systems / infra-check

infra-check.py - Check any domain for machine-readable infrastructure signals. Checks JSON-LD, llms.txt, robots.txt bot directives, and HTTP response headers. Outputs a binary PASS/FAIL per domain, with optional CSV or JSON export.

infra-check

infra-check is a single-file command-line tool that inspects any domain for machine-readable infrastructure signals — the kind of things that determine how well a site can be understood by crawlers, AI agents, and automated clients It runs five checks per domain and produces a binary PASS / FAIL verdict with optional CSV or JSON export for batch runs.

What it checks

#	Check	What it looks for
1	JSON-LD	Presence of valid `application/ld+json` structured data on the homepage, and the `@type`s it declares.
2	llms.txt	Whether the domain serves a non-trivial `/llms.txt` file at the root.
3	robots.txt	Whether `/robots.txt` exists, whether generic bots are allowed, and which known AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, …) are blocked.
4	Headers	HTTP status, a valid `text/html` content type, and the presence of common security headers (HSTS, X-Content-Type-Options, X-Frame-Options, CSP).
5	SSL/TLS	That the domain responds over HTTPS without certificate

…

View on GitHub

One question

You feed domains into your pipeline. How many of them pass all five checks?

Run the script. Look at the CSV. That number is the actual size of your usable source list.

If you run it, drop your numbers in the comments. I am genuinely curious how other people's source lists hold up.

Thorsten Litzki is the founder of Litzki Systems LLC and builds a cryptographically signed infrastructure verification engine. He scans regulated websites for a living and writes about what the data reveals.

DEV Community: Thorsten