Does your JSON-LD form one connected graph? Check @id references in Python

#python #seo #webdev #tutorial

Structured data on a page is rarely one block. A theme prints an Organization, a plugin adds a WebSite, an SEO tool adds a WebPage and a breadcrumb. Each block can be valid on its own and the page can still describe a broken graph: a WebPage that points at a breadcrumb nobody defined, an Organization declared twice, or one block that fails to parse and silently drops out.

This post is a short, dependency-free Python script that extracts every JSON-LD block from a page and checks whether the entities connect through @id references. It is part of my series on small Python checks for technical SEO, after the canonical signals audit, the noindex header comparison and the sitemap lastmod audit.

What the script checks

In JSON-LD, an object with an @id is a named node. Another object can refer to it with just {"@id": "..."} instead of repeating all its properties. That is how a WebSite points to its publisher, or a WebPage to its breadcrumb. The script flags:

Blocks that do not parse. A trailing comma or an unescaped quote makes the whole block unreadable.
References without a definition. A {"@id": "..."} that no block on the page defines.
Duplicate definitions. The same @id fully defined more than once.
Nodes without a @type.

The script

Save it as jsonld_graph_check.py. It uses only the standard library.

#!/usr/bin/env python3
"""Check whether the JSON-LD on a page forms a connected graph.

Usage:
    python jsonld_graph_check.py https://example.com/page
    python jsonld_graph_check.py --file page.html
"""
import json
import sys
import urllib.request
from html.parser import HTMLParser


class JsonLdExtractor(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_ld = False
        self.buf = []
        self.blocks = []

    def handle_starttag(self, tag, attrs):
        if tag == "script" and dict(attrs).get("type", "").lower() == "application/ld+json":
            self.in_ld = True
            self.buf = []

    def handle_endtag(self, tag):
        if tag == "script" and self.in_ld:
            self.in_ld = False
            self.blocks.append("".join(self.buf))

    def handle_data(self, data):
        if self.in_ld:
            self.buf.append(data)


def load_html(args):
    if args[0] == "--file":
        with open(args[1], encoding="utf-8") as f:
            return f.read()
    req = urllib.request.Request(args[0], headers={"User-Agent": "jsonld-check/1.0"})
    with urllib.request.urlopen(req, timeout=20) as resp:
        return resp.read().decode("utf-8", errors="replace")


def flatten(node, out):
    """Collect every dict that carries an @type or @id, at any depth."""
    if isinstance(node, list):
        for item in node:
            flatten(item, out)
    elif isinstance(node, dict):
        if "@type" in node or "@id" in node:
            out.append(node)
        for value in node.values():
            flatten(value, out)


def is_reference_only(node):
    return set(node.keys()) == {"@id"}


def main():
    if len(sys.argv) < 2:
        sys.exit(__doc__)
    html = load_html(sys.argv[1:])
    parser = JsonLdExtractor()
    parser.feed(html)

    problems, nodes = [], []
    for i, raw in enumerate(parser.blocks, 1):
        try:
            flatten(json.loads(raw), nodes)
        except json.JSONDecodeError as exc:
            problems.append(f"block {i}: invalid JSON ({exc.msg} at line {exc.lineno})")

    defined, referenced = {}, set()
    for node in nodes:
        node_id = node.get("@id")
        if not node_id:
            continue
        if is_reference_only(node):
            referenced.add(node_id)
        else:
            if node_id in defined:
                problems.append(f"duplicate definition of @id {node_id}")
            defined[node_id] = node.get("@type", "?")

    for ref in sorted(referenced - set(defined)):
        problems.append(f"@id referenced but never defined on this page: {ref}")
    for node in nodes:
        if "@type" not in node and not is_reference_only(node):
            problems.append(f"node without @type: {json.dumps(node)[:80]}")

    print(f"JSON-LD blocks found: {len(parser.blocks)}")
    print(f"Entities with @id: {len(defined)}")
    for node_id, node_type in defined.items():
        print(f"  {node_type}: {node_id}")
    print(f"Problems: {len(problems)}")
    for p in problems:
        print(f"  - {p}")
    sys.exit(1 if problems else 0)


if __name__ == "__main__":
    main()

Run it on a test page

I tested it on a small HTML file with two blocks. The first is a @graph with an Organization, a WebSite and a WebPage, where the WebPage points to a breadcrumb that is never defined. The second is a Product block with a deliberate trailing comma.

JSON-LD blocks found: 2
Entities with @id: 3
  Organization: https://example.com/#org
  WebSite: https://example.com/#website
  WebPage: https://example.com/p/#webpage
Problems: 2
  - block 2: invalid JSON (Illegal trailing comma before end of object at line 1)
  - @id referenced but never defined on this page: https://example.com/p/#breadcrumb

That output comes from my own test file, not from a live site. The exit code is 1 when there are problems, so you can drop the script into a CI step or a scheduled crawl check.

Reading the results

Invalid JSON is always worth fixing. The block cannot be read at all, so whatever it described is lost.
A reference that is not defined on this page is a warning, not always an error. A common pattern is to define the Organization once on the homepage and reference its @id from every other page. In that case the reference is intentional. Check that the homepage really defines it with the same @id string.
A duplicate definition often means two plugins are both printing the same entity. Pick one source and switch the other off.

Limitations

It reads the raw HTML response. If your JSON-LD is injected by JavaScript, save the rendered HTML from your browser or crawler and use --file.
It checks the graph structure only. It does not validate property names against schema.org or confirm rich result eligibility. Use Google's Rich Results Test and the Schema Markup Validator for that.
It does not follow references across pages. For that you would run it on several URLs and merge the @id sets.

Sources

I'm Hanin, an SEO specialist working on technical SEO, GEO and entity building. I write small, checkable scripts like this one for the series "Python checks for technical SEO". If you run it on a messy site, tell me what it caught.

DEV Community