I Replaced Browser-Based Social Card Checks with One Metadata API

I used to open a browser in CI just to confirm that release pages had titles, descriptions, and social images. The checks were slow, the browser image needed regular maintenance, and a passing page render still did not tell me whether the Open Graph tags were complete.

I replaced that step with the Link Preview & Metadata Extractor and a small Python policy checker. It fetches the HTML over HTTP, returns the metadata and response details as structured JSON, and leaves the project-specific pass or fail rules in my code.

The browser was testing the wrong layer

A browser is useful when the question is whether a page renders correctly. My actual question was narrower: what will a link unfurler or crawler find in the document metadata?

For that check I care about:

• A successful HTML response
• A usable title and description
• An absolute social image URL
• A canonical URL on the expected host
• An Open Graph type
• Twitter Card metadata where the product supports X sharing
• Robots directives that do not accidentally block indexing

None of those requires screenshots or layout inspection. The actor uses HTTP and parses the document metadata directly, including Open Graph fields, Twitter Cards, favicons, language, canonical URLs, all meta tags, status code, content type, and response timing.

That also creates a better failure boundary. DNS, SSL, timeout, HTTP, and non-HTML errors come back as structured errors instead of a generic browser navigation failure.

Fetch the exact deployed URL

The audit runs after deployment against the public URL, not against rendered template source. That catches reverse proxy redirects, missing production environment variables, and CDN responses that local tests never see.

import os
from typing import Any

import requests

APIFY_TOKEN = os.getenv("APIFY_TOKEN", "YOUR_APIFY_TOKEN")
ACTOR_ID = "weeknds~link-preview-metadata-extractor"
RUN_URL = (
    f"https://api.apify.com/v2/acts/{ACTOR_ID}/"
    "run-sync-get-dataset-items"
)


def extract_metadata(url: str) -> dict[str, Any]:
    response = requests.post(
        RUN_URL,
        params={"token": APIFY_TOKEN},
        json={
            "url": url,
            "includeFavicon": True,
            "userAgent": "Release-Metadata-Audit/1.0",
            "timeout": 20,
        },
        timeout=75,
    )
    response.raise_for_status()
    items = response.json()
    if len(items) != 1:
        raise RuntimeError(f"Expected one result for {url}, got {len(items)}")
    return items[0]

A custom user agent makes the audit identifiable in server logs. It should describe the client honestly, not impersonate a search crawler.

Flatten the response before applying policy

Metadata extractors can group Open Graph and HTTP data into nested objects. A recursive flattener lets the policy work with either nested values or literal keys such as og:title.

from collections.abc import Mapping
from typing import Any


def flatten(value: Mapping[str, Any], prefix: str = "") -> dict[str, Any]:
    output = {}
    for key, child in value.items():
        path = f"{prefix}.{key}" if prefix else key
        output[path] = child
        if isinstance(child, Mapping):
            output.update(flatten(child, path))
    return output


def find(item: dict[str, Any], *keys: str) -> Any:
    fields = flatten(item)
    for key in keys:
        for path, value in fields.items():
            if path == key or path.endswith(f".{key}"):
                if value not in (None, "", []):
                    return value
    return None

I keep this adapter separate from the rules. If the actor adds a new output field, the audit can use it without changing the network layer.

Turn metadata expectations into explicit rules

A release check should produce actionable messages. "Metadata invalid" sends someone back to the logs, while "og:image must be an absolute HTTPS URL" points at the fix.

from dataclasses import dataclass
from urllib.parse import urlparse


@dataclass(frozen=True)
class Finding:
    severity: str
    field: str
    message: str


def audit_page(item: dict, expected_host: str) -> list[Finding]:
    findings = []

    if item.get("error"):
        return [Finding(
            "error",
            item.get("error_type", "fetch"),
            item["error"],
        )]

    status = find(item, "status_code")
    content_type = str(find(item, "content_type") or "")
    title = find(item, "og:title", "title")
    description = find(item, "og:description", "description")
    image = find(item, "og:image")
    canonical = find(item, "canonical_url", "canonical", "og:url")
    og_type = find(item, "og:type")
    robots = str(find(item, "robots") or "").lower()

    if status and not 200 <= int(status) < 300:
        findings.append(Finding("error", "status_code", f"HTTP {status}"))
    if content_type and "text/html" not in content_type:
        findings.append(Finding("error", "content_type", content_type))
    if not title or not 10 <= len(str(title)) <= 70:
        findings.append(Finding("error", "title", "Use a 10-70 character title"))
    if not description or not 50 <= len(str(description)) <= 200:
        findings.append(Finding(
            "warning",
            "description",
            "Use a 50-200 character description",
        ))
    if not image or urlparse(str(image)).scheme != "https":
        findings.append(Finding(
            "error",
            "og:image",
            "Use an absolute HTTPS image URL",
        ))
    if not canonical:
        findings.append(Finding("warning", "canonical", "Canonical URL is missing"))
    elif urlparse(str(canonical)).hostname != expected_host:
        findings.append(Finding(
            "error",
            "canonical",
            f"Expected canonical host {expected_host}",
        ))
    if not og_type:
        findings.append(Finding("warning", "og:type", "Open Graph type is missing"))
    if "noindex" in robots:
        findings.append(Finding("error", "robots", "Page is marked noindex"))

    return findings

The length ranges are policy, not universal laws. I use them to catch obviously empty or accidentally huge values, not to pretend every social platform truncates at the same character.

Audit a release manifest instead of crawling everything

A full-site crawl on every release wastes time and makes ownership unclear. My deployment exports a short manifest of URLs affected by the release, then the audit checks only those pages.

import json
from pathlib import Path


def load_release_urls(path: Path) -> list[dict[str, str]]:
    payload = json.loads(path.read_text(encoding="utf-8"))
    urls = payload.get("urls", [])
    if not urls:
        raise ValueError("Release manifest contains no URLs")
    return urls


def audit_release(manifest_path: Path) -> int:
    error_count = 0

    for entry in load_release_urls(manifest_path):
        url = entry["url"]
        expected_host = entry["expected_host"]

        try:
            item = extract_metadata(url)
            findings = audit_page(item, expected_host)
        except (requests.RequestException, RuntimeError, ValueError) as error:
            findings = [Finding("error", "request", str(error))]

        if not findings:
            print(f"PASS {url}")
            continue

        for finding in findings:
            print(
                f"{finding.severity.upper()} {url} "
                f"[{finding.field}] {finding.message}"
            )
            if finding.severity == "error":
                error_count += 1

    return error_count


if __name__ == "__main__":
    failures = audit_release(Path("release-urls.json"))
    raise SystemExit(1 if failures else 0)

The manifest stays simple and reviewable:

{
  "urls": [
    {
      "url": "https://docs.example.com/python/cache-invalidation",
      "expected_host": "docs.example.com"
    },
    {
      "url": "https://example.com/releases/summer-update",
      "expected_host": "example.com"
    }
  ]
}

Warnings appear in the job output but do not block the release. Errors return a non-zero exit code. That distinction stopped minor description preferences from becoming production incidents while still blocking broken canonicals, failed fetches, insecure image URLs, and accidental noindex directives.

Save evidence for failed checks

The worst version of this job only printed a summary. By the time someone investigated, a redeploy had changed the page and the original response was gone.

Now the audit writes the actor item whenever a page fails. The artifact contains the metadata and HTTP details observed at release time, without requiring another request.

import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path

ARTIFACT_DIR = Path("metadata-audit-artifacts")


def save_failure_artifact(url: str, item: dict, findings: list[Finding]) -> Path:
    ARTIFACT_DIR.mkdir(parents=True, exist_ok=True)
    digest = hashlib.sha256(url.encode()).hexdigest()[:12]
    timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
    path = ARTIFACT_DIR / f"{timestamp}-{digest}.json"
    path.write_text(
        json.dumps(
            {
                "url": url,
                "findings": [finding.__dict__ for finding in findings],
                "metadata": item,
            },
            indent=2,
            sort_keys=True,
        ),
        encoding="utf-8",
    )
    return path

Treat these files as operational artifacts, not permanent content. A 30-day retention window is usually enough to diagnose a release without building an unbounded archive.

What the switch changed

The audit no longer downloads a browser image or waits for client rendering. It tests the metadata layer directly, and each rule names the field that failed. Browser-based visual checks still exist for pages where layout matters, but they are no longer the default tool for validating social cards.

The Link Preview & Metadata Extractor is currently listed as pay per usage on Apify. The live price depends on platform resource consumption and the account plan, so the job should check the current store pricing before turning a small release audit into a large crawl.

Run the audit only after the deployment URL is reachable, keep the token in the CI secret store, and cap retries for network errors. A timeout can deserve one retry; a noindex directive, non-HTML response, or wrong canonical host needs a code or configuration change, not another request.