Most sites collect small SEO problems quietly: a page that now 404s, a link to a partner site that died, a title that grew past 60 characters, a noindex left on after a migration, a new section nobody added to the sitemap. None of them is urgent on its own, which is why they pile up. What a site owner or an agency needs is not a 40-page report but a short list, worst first, with the exact URLs to fix.
This post sets that up as a scheduled audit in Python, then compares the SEO audit and broken-link tools on the Apify Store using the prices their Store pages listed at the time of writing.
Disclosure: I'm Aurelix, and I built the Broken Link Crawler & SEO Audit. It's a paid tool (prices below). Check each Store page before you buy: prices change.
What one run returns
You give it one site. It reads the site's XML sitemaps (from robots.txt or /sitemap.xml), follows internal links on the same host up to maxPages, and returns:
-
One
pagerow per URL:status_code,redirects,titleandtitle_length,meta_descriptionand its length,h1_count,canonical_url,noindex,word_count, link and image counts,broken_links(each with its URL and status code),images_missing_alt_count,in_sitemap,response_time_ms, andissues: the issue codes on that page, most severe first. -
One
summaryrow:pages_audited,health_score(the share of pages with no high or medium issue), status-code counts, andissuesranked by severity and pages affected, each with up to five example URLs.
Broken means a real 404/410, a 5xx or a dead host. Links that answer 401, 403 or 429 (common for sites that block automated checks) are counted as unverified, so they don't clutter the fix list.
A fix list and a broken-link sheet
pip install apify-client
import csv
import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("aurelix/seo-audit-crawler").call(
run_input={
"siteUrl": "https://books.toscrape.com",
"maxPages": 200,
"useSitemap": True,
"checkExternalLinks": True,
"maxLinkChecks": 1000,
}
)
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
summary = next(r for r in rows if r["record_type"] == "summary")
pages = [r for r in rows if r["record_type"] == "page"]
print(f"Health score {summary['health_score']} over {summary['pages_audited']} pages")
for i in summary["issues"][:5]:
print(f" {i['severity']:<6} {i['issue']}: {i['pages']} pages, e.g. {i['examples'][0]}")
severity = {i["issue"]: i["severity"] for i in summary["issues"]}
with open("fix_list.csv", "w", newline="", encoding="utf-8") as f:
w = csv.writer(f)
w.writerow(["severity", "issue", "url", "status_code", "title"])
for p in pages:
for code in p["issues"]:
if severity.get(code) in ("high", "medium"):
w.writerow([severity[code], code, p["url"], p["status_code"], p["title"]])
with open("broken_links.csv", "w", newline="", encoding="utf-8") as f:
w = csv.writer(f)
w.writerow(["on_page", "broken_link", "status_code", "error"])
for p in pages:
for b in p["broken_links"]:
w.writerow([p["url"], b["url"], b["status_code"], b["error"]])
Replace siteUrl with your own site, or a client's with their permission. fix_list.csv is the developer's to-do list, sorted by nothing more clever than severity; broken_links.csv tells an editor which link to change on which page.
To track progress, save the input as a task in the Apify Console and schedule it weekly or monthly, then compare the summary rows: health_score and the pages count of each issue. There is no built-in diff between runs.
A few input choices that matter:
-
Quick pre-launch pass:
"useSitemap": false, "checkExternalLinks": false, "maxLinkChecks": 0checks titles, meta, H1, canonicals andnoindexfast, without testing links. -
Sitemap gaps: with
useSitemapon, indexable pages missing from the sitemap getnot_in_sitemap, and sitemap URLs that redirect, error or arenoindexgetsitemap_url_not_ok. -
Link budget:
maxLinkChecks(up to 5,000) caps the extra requests for external links and files. Links over the cap are counted inunchecked_links_count, never reported as fine.
SEO audit and broken-link tools on the Apify Store compared
Prices from each Store page at the time of writing. Units differ: the first three charge per page audited, the last per link checked.
| Aurelix Broken Link Crawler & SEO Audit | smart-digital/complete-seo-audit-tool | logiover/website-seo-audit-crawler | khadinakbar/broken-link-checker | |
|---|---|---|---|---|
| Free plan | $2.50 per 1,000 pages | $40.00 per 1,000 pages | $10.00 per 1,000 pages | $1.00 per 1,000 links |
| Bronze | $1.75 | $40.00 | $8.00 | $1.00 |
| Gold | $1.75 | $30.00 | $5.00 | $1.00 |
| Start fee per run | $0.001 | none | $0.00005 | $0.00005 |
| On-page checks (title, meta, H1, canonical, noindex) | yes | yes | yes | no |
| Broken-link checks | every link, up to 5,000 extra checks per run, free | first 10 links per page | no (sold as a separate tool) | yes, the product |
| Indexable pages missing from the sitemap | yes | not stated | not stated | not stated |
| 401/403/429 kept apart from broken | yes | not stated | not stated | not stated |
| Ranked issue summary | yes, with health score | 0-100 score per page and a site report | list of detected issues | no |
| Open Graph, hreflang, schema types | no | schema | yes | no |
| Core Web Vitals, accessibility | no | yes | not stated | no |
Where each wins: smart-digital is the most-used of the four and the only one that measures Core Web Vitals and runs accessibility checks; if page speed scores matter to you, it's the one to use. logiover reads more on-page tags (Open Graph, Twitter cards, hreflang, schema.org types). khadinakbar is the cheapest if all you want is broken links. The Aurelix tool is the cheapest per audited page on every plan, and fits a regular technical audit: on-page checks, every link tested at no extra charge, the sitemap gap check, and a summary that ranks what to fix first.
If what you need from the site is its content rather than an audit, for search or an AI assistant, the Website to Markdown API crawls the same way and returns clean Markdown per page.
Limits
-
One host per run.
www.and the bare domain count as one site; other subdomains are checked as external links, not crawled. -
No JavaScript rendering. It audits the HTML the server sends; links injected only by JavaScript are not found, and there are no Lighthouse or Core Web Vitals scores.
response_time_msis the download time from Apify's servers. -
Polite pace: at most two requests a second per site, slower with a robots.txt
Crawl-delay, about 1,100 pages an hour. A full 2,000-page audit needs a run timeout of about 2 hours. - Up to 2,000 pages per run. HTML pages over 2 MB are skipped and listed in the summary.
- robots.txt is always obeyed, for the site and for every external link checked.
- Duplicate titles and descriptions are compared only across the pages audited in that run.
Cost
Each audited page with an HTTP status is charged once, including redirects and 404s, because those are findings too. The summary row, every link check, pages blocked by robots.txt, pages that could not be loaded, throttled pages and non-HTML files are free, so a run against a site that is down costs only the start fee. The current price per page is on the Store page; Apify's free plan includes $5 of credit a month, enough for a monthly audit of a site with a few hundred pages.
Top comments (1)
The ranked fix list is the part that makes this actually usable for a small business. A 40-page audit usually gets filed away, but a short worst-first list with exact URLs is something a developer — or an owner — can work through in an afternoon. Separating 401/403/429 from genuinely broken links is a smart call too, and the noindex-left-on-after-a-migration example is painfully familiar. Do you find most small sites' biggest losses are broken links on pages that still get traffic, or pages that quietly fell out of the sitemap?