Batulzii Ulziikhuu

Posted on Oct 18

Free SEO Audit Crawler API: Automate Technical SEO Checks with JavaScript or Python

#seo #api #webdev #javascript

Desktop crawlers like Screaming Frog are excellent, but they're manual: open the app, crawl, export, repeat. If you manage several sites, run an agency, or want SEO checks in CI after every deploy, you want an SEO audit crawler you can call as an API and get JSON back.

This guide shows how to do that with the SEO Audit Toolkit on Apify, from JavaScript, Python and curl, and how to turn the results into automated alerts.

What the audit checks

For every crawled page:

Title and meta description: missing, too short/long, duplicates across the site
H1: missing or multiple
Canonical, noindex / robots meta (including none), hreflang
Structured data (JSON-LD/schema.org) and Open Graph tags
Image alt text, word count / thin content
Broken internal and external links (HTTP status checks)
HTTP errors and redirect problems

Site-wide:

robots.txt parsing (including "blocks all crawlers" detection)
sitemap.xml validation (index sitemaps, URL counts, invalid XML)
Tech stack detection
Optional Google PageSpeed / Lighthouse scores with your own API key

You get a 0–100 site score plus a prioritized issue list with example URLs, so you know what to fix first.

Is it free?

Honestly: almost. You pay per page audited ($0.004 per page on Apify's Free plan, down to $0.0015 on paid plans) plus a tiny start fee. A 50-page audit is about $0.20. Apify's free plan includes monthly credits, so small sites and testing effectively cost nothing out of pocket. The summary record and unreachable-URL errors are free, and you can cap spend per run.

Step 1: API token

Create an account at apify.com, copy your token from Settings → API & Integrations:

export APIFY_TOKEN=your_token

Step 2: Run an audit with JavaScript

npm install apify-client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('batuweb/seo-audit-toolkit').call({
  startUrls: ['https://example.com'],
  crawlMode: 'sitemap',   // or crawl links from the start URL
  maxPages: 50,
  checkBrokenLinks: true,
  maxLinkChecks: 200,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const summary = items.find((i) => i.type === 'summary'); // summary is the last record
console.log('Score:', summary.score);
for (const issue of summary.prioritizedIssues.slice(0, 10)) {
  console.log(`${issue.severity.padEnd(8)} ${issue.code} (${issue.pages} pages) e.g. ${issue.exampleUrls[0]}`);
}

Step 3: Python

pip install apify-client

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("batuweb/seo-audit-toolkit").call(run_input={
    "startUrls": ["https://example.com"],
    "maxPages": 100,
})

items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
summary = next(i for i in items if i.get("type") == "summary")
pages = [i for i in items if i.get("type") == "page"]

print("Site score:", summary["score"], "| pages:", summary["pagesAudited"])
print(summary["issuesBySeverity"])

Step 4: curl / no-code

curl -X POST \
  "https://api.apify.com/v2/acts/batuweb~seo-audit-toolkit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://example.com"],"maxPages":20}'

The same endpoint works from n8n, Make, Zapier or Google Sheets (Apps Script UrlFetchApp).

Output example (summary record)

{
  "type": "summary",
  "score": 97,
  "pagesAudited": 10,
  "pagesFailed": 0,
  "issuesBySeverity": { "critical": 0, "high": 1, "medium": 2, "low": 11 },
  "prioritizedIssues": [
    { "code": "BROKEN_INTERNAL_LINKS", "severity": "high", "pages": 1, "exampleUrls": ["https://example.com/about/"] },
    { "code": "TITLE_TOO_SHORT", "severity": "medium", "pages": 2, "exampleUrls": ["https://example.com/contact/"] }
  ],
  "robotsTxt": { "https://example.com": { "status": 200, "blocksAllCrawlers": false } },
  "sitemaps": [{ "url": "https://example.com/sitemap-0.xml", "status": 200, "valid": true, "urlCount": 51 }]
}

Each page also gets its own record with all checks and issues, which is handy for spreadsheets.

Automation ideas

SEO regression check in CI: run after deploy and fail the pipeline if the score drops or a critical issue appears:

if (summary.issuesBySeverity.critical > 0 || summary.score < 85) process.exit(1);

Weekly client reports: an Apify schedule per client site, with a webhook to Slack or email.
Broken-link monitoring across a portfolio of sites.
Pre-migration baselines: audit before and after a redesign and diff the page records.
AI agents: call it via the Apify MCP server so an assistant can audit a URL on request.

Honest limitations

It is an HTML crawler, not a headless browser. Content rendered only by client-side JavaScript may be missed.
PageSpeed/Lighthouse metrics need your own free Google API key and cost extra per page.
The score is a heuristic for prioritization, not a Google ranking factor.
Only audit sites you own or have permission to audit. Private/internal network addresses are blocked by design.

FAQ

How is this different from Screaming Frog? Screaming Frog is a powerful desktop app. This is a lightweight cloud API for automation, scheduling and integrations. Many people use both.

How many pages can I audit? Set maxPages. Cost scales linearly, and you can set a maximum cost per run.

DEV Community