DEV Community: Calimero

From sitemap.xml to a full audit report

Calimero — Mon, 02 Nov 2026 19:51:19 +0000

Most audits start with a crawl. That is usually the wrong first move: a crawl discovers what the crawler can reach, not what the site actually publishes. A sitemap is a better seed — the site's own list of canonical URLs, usually one HTTP request away.

This tutorial runs a sitemap-to-report workflow with the Sitemap Audit Actor: one audit item per URL, then a CSV export.

What the Actor does

The Sitemap Audit Actor:

Fetches the sitemap URL.
Parses <loc> entries.
If the document is a sitemap index, follows its child sitemaps (up to 10 per index).
Fetches each page as static HTML and runs the same audit engine as the Bulk Website Audit Actor.
Pushes one dataset item per URL with a score, issue list, and page statistics.

It does not render JavaScript, crawl links, or check robots.txt. It audits exactly the URLs listed in the sitemap, up to maxUrls. That makes it predictable: the input list is the scope.

Step 1 — Prepare the input

The input is two fields:

{
  "sitemapUrl": "https://example.com/sitemap.xml",
  "maxUrls": 100
}

sitemapUrl (required): the public URL of the sitemap or sitemap index.
maxUrls (optional, default 100): the maximum number of page URLs to audit.

Start with a small maxUrls. A sitemap that returns an HTML error page, a login wall, or a compressed file fails early, and you want to know that in 30 seconds, not 30 minutes.

Step 2 — Run it through the API

You need an Apify API token in an environment variable — not in a notebook or a commit.

export APIFY_TOKEN="your_token_here"

curl -sS -X POST \
  "https://api.apify.com/v2/acts/D49xJAuq6tledUTzL/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "sitemapUrl": "https://example.com/sitemap.xml",
    "maxUrls": 100
  }' \
  -o audit.json

The run-sync-get-dataset-items endpoint starts the run, waits for it to finish, and returns the dataset items as JSON. It is the shortest path for small and medium sitemaps. The synchronous endpoint has a platform timeout (around five minutes), so keep maxUrls small; for larger sitemaps, use the asynchronous pattern:

# Start the run
curl -sS -X POST \
  "https://api.apify.com/v2/acts/D49xJAuq6tledUTzL/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sitemapUrl":"https://example.com/sitemap.xml","maxUrls":1000}'

# Poll the run, then fetch its default dataset
curl -sS \
  "https://api.apify.com/v2/datasets/$DATASET_ID/items?format=json&token=$APIFY_TOKEN" \
  -o audit.json

The run response contains id and defaultDatasetId. Poll /v2/actor-runs/{id} until SUCCEEDED, then fetch the dataset.

Step 3 — Read the output

Each dataset item describes one URL:

Field	Type	Meaning
`url`	string	Final URL after redirects
`status`	number or null	HTTP status; `null` on a network error
`score`	number	Heuristic 0–100 triage score
`title`	string	Page `<title>`
`lang`	string or null	`<html lang>` value
`issues`	array	Issue objects with `code`, `severity`, `message`, `count`, `examples`
`stats`	object	Counts for words, images, links, form fields, headings, JSON-LD blocks

The stats object:

"stats": {
  "words": 812,
  "images": 14,
  "images_without_alt": 1,
  "links": 96,
  "empty_links": 3,
  "form_fields": 6,
  "unlabeled_fields": 0,
  "headings": 24,
  "json_ld_blocks": 1
}

Do not trust score without checking status: a 403 challenge page can still parse as HTML and produce a misleading score. Filter to status == 200 and reject challenge titles such as "Just a moment..." or "Access Denied".

Step 4 — Export CSV

From the Apify console, open the run, go to the dataset, and use Export → CSV. From the API:

curl -sS \
  "https://api.apify.com/v2/datasets/$DATASET_ID/items?format=csv&token=$APIFY_TOKEN" \
  -o audit.csv

Top-level fields export cleanly. Nested issues and stats may arrive as JSON strings or flattened keys; for spreadsheet work, build the columns you need from JSON with jq:

curl -sS \
  "https://api.apify.com/v2/datasets/$DATASET_ID/items?format=json&token=$APIFY_TOKEN" \
| jq -r '
  ["url","status","score","words","images","images_without_alt","form_fields","unlabeled_fields"],
  (.[] | [.url, .status, .score, .stats.words, .stats.images,
          .stats.images_without_alt, .stats.form_fields, .stats.unlabeled_fields])
  | @csv' > audit.csv

That gives you a stable schema you can open in any spreadsheet or load into a database.

A repeatable workflow

Run with maxUrls: 20 to validate the sitemap and see whether the site blocks the Actor.
Raise maxUrls to the real size, or split the sitemap by section.
Filter status == 200 and remove challenge pages.
Group by issue code to get prevalence, and sum stats for element-level rates.
Schedule the Actor weekly and store each dataset to track regressions.

The last step matters: scheduling turns a one-off report into a trend.

Caveats

Static HTML only. No JavaScript execution. Client-rendered pages may return an empty shell or missing content; treat that as a rendering limitation, not an accessibility finding.

Sitemap indexes are capped. The Actor follows up to 10 child sitemaps per index and takes the first maxUrls URLs it finds. Large sitemaps will be truncated; split them or run multiple inputs.

No lastmod filtering. Every <loc> is treated equally. To audit only recently changed pages, filter the sitemap first.

No robots.txt check. The Actor fetches each URL directly. Keep maxUrls sane, avoid hammering production, and schedule runs outside peak hours. Targets may still return 429 or 403; those items are not valid results.

Compressed sitemaps are not explicitly handled. Plain XML is the safe input. If your sitemap is served as .xml.gz, decompress it first or expose an uncompressed variant.

The score is a triage signal. It weights template-level issues such as missing titles, canonical links, labels, and alt text. It is not a WCAG conformance score and cannot detect contrast, keyboard traps, focus order, or runtime ARIA problems.

Wrap-up

The sitemap is the scope; the Actor is the executor; the dataset is the report. Once the pipeline works, the hard part is deciding which recurring issue codes your team will actually fix.

Run the workflow with the Sitemap Audit Actor. For arbitrary URL lists, use the Bulk Website Audit Actor.

I audited 100 public-sector websites in the EU and the US. The regional ranking is noise.

Calimero — Mon, 02 Nov 2026 19:51:18 +0000

Fifty European and fifty US public-sector homepages, same static audit, same day. Seventy-nine were auditable. The average scores differ by 2.2 points. The failures are almost identical.

My first EU-vs-US audit used 40 URLs and produced a 3.7-point gap. On a sample that small, that is sampling error with a chart. So I scaled to 100 URLs and looked for failures that survive a larger n. The ranking stayed noise; the same template-level defects stayed at the top.

Methodology

I built two convenience samples of 50 homepages each:

EU (50): EU institutions (Parliament, Commission, ECB, EIB, agencies), national government portals, and major city portals.
US (50): federal departments and agencies, state portals, and major city portals.

Each URL was audited once with the Bulk Website Audit Actor on Apify, in four batches of 25. The Actor fetches static HTML with a 20-second timeout, does not execute JavaScript, and returns a 0–100 heuristic score plus issue codes and page statistics. The score is a triage signal, not a WCAG conformance score. Data was collected on 2 November 2026. I did not retry failures: bot protection is part of what an automated audit actually sees.

Sample size and exclusions

Of the 100 URLs, 21 were excluded:

Reason	EU	US	Total
HTTP 403 (WAF / bot challenge)	6	12	18
HTTP 429 (rate limit / checkpoint)	1	0	1
HTTP 202 with no content	1	1	2
Excluded total	8	13	21

The blocked pages carried titles such as "Just a moment...", "Access Denied", "Browser check", "Azure WAF", "Not allowed", and "Vercel Security Checkpoint". None were scored as content. No HTTP 200 page was removed for an empty title or a language selector.

That left 79 auditable homepages: 42 EU and 37 US (79% of the original sample).

Big federal domains — state.gov, defense.gov, hhs.gov, transportation.gov — were among the excluded. A pipeline that ignores status codes would score challenge pages and report nonsense. The US sample therefore skews toward agencies and states that allow automated fetching.

Results

Metric	EU (n=42)	US (n=37)
Average score	85.5	87.7
Median score	88.0	90.0
Score range	61–97	73–97
Pages scoring ≥90	18 (42.9%)	19 (51.4%)

A 2.2-point gap on 42 and 37 pages is not a ranking: the ranges overlap, and the EU sample contains both the lowest score (61) and nine pages at 95 or above.

Issue prevalence (share of auditable pages with at least one occurrence)

Issue	EU (n=42)	US (n=37)
Missing canonical	42 (100%)	37 (100%)
Missing JSON-LD structured data	26 (61.9%)	27 (73.0%)
Missing meta description	14 (33.3%)	8 (21.6%)
Links without accessible text	20 (47.6%)	11 (29.7%)
Multiple `<h1>`	8 (19.0%)	10 (27.0%)
Form fields without a label	9 (21.4%)	3 (8.1%)
Images without `alt`	4 (9.5%)	6 (16.2%)
Missing `<h1>`	4 (9.5%)	4 (10.8%)
Heading level skip	5 (11.9%)	10 (27.0%)
Title outside 10–65 characters	3 (7.1%)	5 (13.5%)
Short meta description	2 (4.8%)	2 (5.4%)
Thin content (<200 words)	3 (7.1%)	2 (5.4%)
Missing `lang`	1 (2.4%)	0 (0%)

Aggregate element counts

Metric	EU	US
Images	852	660
Images without `alt`	30 (3.5%)	24 (3.6%)
Links	7,038	4,912
Links without accessible text	136 (1.9%)	29 (0.6%)
Form fields	391	292
Form fields without a label	17 (4.4%)	4 (1.4%)

What the numbers say

Canonical is not a regional problem. It is a template problem. All 79 auditable pages — 100% in both regions — had no canonical link. On sites with aliases, translated paths, and legacy URLs, that is one template omission repeated across thousands of pages.

Structured data is missing on most pages, and the US is slightly worse. 61.9% of EU pages and 73.0% of US pages had no JSON-LD block. For public services, structured data tells search engines about opening hours, locations, services, and FAQs. It is the cheapest visible improvement here.

Meta descriptions are missing on a third of EU pages and a fifth of US pages. The gap (33.3% vs 21.6%) is larger than the score gap, but the absolute numbers matter: 22 homepages left search engines to generate the snippet.

Empty links are widespread per page, rare per link. 47.6% of EU pages and 29.7% of US pages contained at least one link with no accessible text. Those links are only 1.9% of all EU links and 0.6% of all US links. In practice this is a few icon links, logo wrappers, or cards repeated in a template. The bug is localised.

Form labels are the clearest EU weakness. 21.4% of EU pages had at least one unlabeled field, versus 8.1% in the US. Aggregated, 17 of 391 EU fields (4.4%) lacked a label, against 4 of 292 US fields (1.4%).

Image alt is close to parity in aggregate. 3.5% of EU images and 3.6% of US images had no alt attribute. Page-level rates differ more (9.5% vs 16.2%) because a few pages concentrate many decorative images.

Heading structure is worse in the US sample. 27.0% of US pages had multiple <h1> tags and 27.0% skipped a heading level, against 19.0% and 11.9% in the EU. Missing <h1> is almost identical (9.5% vs 10.8%).

Caveats

This is a static audit of one homepage per domain, on one day, on a non-random sample. It cannot detect contrast, keyboard traps, focus order, runtime ARIA state, screen-reader behaviour, or JavaScript-injected issues. Several large sites returned a shell or a challenge page and were excluded.

Scores are heuristic; that gap is sampling variation. The robust signal is binary: 79 of 79 pages missing canonical is not a sampling artifact.

Conclusion

The sample produced no regional winner, only the same five failures on both sides:

Canonical — missing on every page in both regions.
Structured data — absent on 62–73% of pages.
Meta descriptions — absent on 22–33% of pages.
Empty links — on 30–48% of pages, concentrated in templates.
Form labels — missing on 8–21% of pages.

None require a redesign, just template ownership and release checks. The regional ranking is noise; the shared backlog is the finding.

Audit run with the Bulk Website Audit Actor on Apify. Sample: 50 EU + 50 US public-sector homepages; 79 auditable after excluding 21 non-200 or rate-limited URLs. Data collected 2 November 2026.

Europe vs the US: I audited 40 public-sector websites. Both fail the same basics.

Calimero — Mon, 02 Nov 2026 19:23:46 +0000

I ran the same static accessibility and SEO audit on 20 European and 20 US public-sector websites. 25 were auditable. The ranking is noise; the shared failures are the story.

The setup

I wanted a clean comparison, so I used the same checks on both sides:

20 EU URLs: EU institutions, national governments, a few central portals.
20 US URLs: federal departments and agencies.
Same static audit: title, meta description, lang, headings, image alt, form labels, viewport, canonical, noindex, thin content, JSON-LD.

The audit is a triage, not a WCAG compliance test. It runs on static HTML and does not execute JavaScript.

What happened before the scores

15 of the 40 URLs could not be audited:

5 EU sites returned HTTP 403 or a "Just a moment..." challenge.
8 US sites returned 403, 202 or an "Access Denied" page from a WAF.

That is a finding in itself: a large share of public-sector sites now sit behind bot protection. Any audit pipeline that scores a 403 page as if it were content will produce garbage. The first rule of automated auditing is: check the status code and the title before you trust the score.

After removing challenge pages and language selectors:

EU: 13 auditable pages, average score 80.8 / 100.
US: 12 auditable pages, average score 84.5 / 100.

A 3.7-point difference on a sample of 25 is not a ranking. It is noise.

The shared failures

Issue	EU (n=13)	US (n=12)
Missing canonical	100%	100%
Missing JSON-LD structured data	69%	83%
Missing meta description	46%	42%
Multiple `<h1>`	15%	25%
Missing `<h1>`	31%	25%
Links without accessible text	23%	33%
Form fields without a label	23%	17%
Images without `alt`	15%	0%
Missing `lang`	8%	8%

Aggregate counts:

EU: 338 images, 9 without alt (2.7%); 100 form fields, 5 unlabeled (5%).
US: 182 images, 0 without alt; 87 form fields, 3 unlabeled (3.4%).

What stands out

Canonical is missing everywhere. Not "most sites". Every single auditable page in both regions. For public-sector sites with multiple domains, aliases and old URLs, this is a template-level bug that multiplies across thousands of pages.

Structured data is rare. 69% of EU pages and 83% of US pages have no JSON-LD at all. For public services, structured data is how search engines understand opening hours, locations, events and FAQs. It is also the cheapest fix on this list.

Meta descriptions are missing on roughly half the pages. Search engines will write their own snippet, often from the first paragraph.

Accessibility basics are better than expected, but inconsistent. The US sample had zero images without alt; the EU sample had 15% of sites with at least one. Form labels are unlabeled on 17-23% of sites in both regions. These are template fixes, not redesigns.

The caveat

This is a static audit on a small, non-random sample. It cannot detect:

contrast ratios,
keyboard traps,
ARIA state that lies at runtime,
screen-reader behaviour,
issues introduced by JavaScript.

A 100% missing-canonical rate is a real signal because it is a binary check on a template. A 3.7-point average difference between regions is not.

The takeaway

"Europe vs the US" makes a good headline and a bad conclusion. The interesting finding is that both regions fail the same basics: canonical, structured data, meta descriptions, empty links and form labels. These are not exotic accessibility problems. They are template hygiene.

If you run public-sector sites, fix these five things before you buy another overlay:

Add a canonical URL to every template.
Add JSON-LD for the organisation and the main services.
Add a meta description to the templates that lack one.
Give every form field a real <label>.
Audit the media library for alt, not just the homepage.

None of these require a redesign. They require someone to own the template.

The audit was run with the Bulk Website Audit Actor on Apify. Data collected on 2 November. Sample: 20 EU + 20 US public-sector URLs; 25 auditable after excluding 403/202 challenge pages and language selectors.

How to Audit 1,000 Pages for Accessibility and SEO in One Run

Calimero — Mon, 02 Nov 2026 18:46:29 +0000

A practical workflow for agencies that need a full-site baseline before a manual review.

The problem

A client sends you a website with 1,000 pages and asks: "Is it accessible? Is the SEO clean?"

You open the first page. You check the title, the meta description, the images, the form labels, the heading structure. You do it again on page two. By page ten you already know you will not finish today.

Manual page-by-page audits do not scale. What scales is a triage: run the obvious checks on every URL, sort by severity, and spend your human time on the pages that actually need it.

The tool

I use two Apify Actors for this:

Website Accessibility & SEO Audit — crawls one site and audits its pages.
Bulk Website Audit — takes a list of URLs and audits each one in a single run.

Both return one JSON item per page, with a score and a list of concrete issues.

What it checks

For each page:

<title> presence and length
meta description presence and length
lang on <html>
<h1> presence, multiple H1s, heading-level skips
images without alt
links without accessible text
form fields without an associated label
viewport meta
noindex
canonical link
thin content (< 200 words)
JSON-LD structured data

It is a static check. It does not replace a manual WCAG 2.2 audit, but it removes the boring part.

Step 1 — Build the URL list

You can start from:

a sitemap.xml (use the Sitemap Audit Actor to extract and audit URLs directly),
a CSV of landing pages,
a list of client sites.

For this example, a JSON array is enough:

{
  "urls": [
    "https://example.com",
    "https://example.com/about",
    "https://example.com/contact"
  ]
}

Step 2 — Run the Actor

From the Apify Console, paste the JSON and click Start.

From the API:

curl -X POST "https://api.apify.com/v2/acts/calimero-projects~bulk-website-audit/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com","https://example.com/about"]}'

Or with the Python client:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_TOKEN>")
run = client.actor("calimero-projects/bulk-website-audit").call(
    run_input={"urls": ["https://example.com", "https://example.com/about"]}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["score"], len(item["issues"]))

Step 3 — Read the output

One item per page:

{
  "url": "https://example.com/",
  "status": 200,
  "score": 65,
  "title": "Example Domain",
  "issues": [
    {
      "code": "missing_meta_description",
      "severity": "medium",
      "message": "Missing meta description",
      "count": 1,
      "examples": []
    }
  ],
  "stats": {
    "words": 27,
    "images": 0,
    "images_without_alt": 0,
    "links": 0,
    "empty_links": 0,
    "form_fields": 0,
    "unlabeled_fields": 0,
    "headings": 0
  }
}

Export the dataset as CSV or Excel and you have a client-ready baseline in minutes.

Step 4 — Triage, not fix

Sort by score. Ignore the pages above 90 unless the client asked for perfection. Start from the pages with:

missing alt on images,
form fields without labels,
missing or multiple <h1>,
noindex on pages that should rank.

These are template-level issues: fix the template, fix hundreds of pages at once.

What it costs

The Bulk Website Audit uses pay-per-event pricing:

$0.07 per run (start event, 1 GB memory)
$0.01 per audited page

A 1,000-page audit costs about $10.07. An agency can bill that as a few minutes of work.

What it does not do

It does not render JavaScript. For SPA-heavy sites, use a browser-based check.
It does not test contrast, keyboard navigation or screen-reader behaviour.
It does not decide whether a site is legally compliant.

It tells you where to look first. That is usually the difference between a two-day audit and a two-hour triage.

Try it

If you run it on a client site, I would like to know which issues show up most often. That is the data I use to decide what the next version should check.

I audited 12 Italian public-sector websites for accessibility and SEO. Here's what I found.

Calimero — Mon, 02 Nov 2026 07:50:23 +0000

Method: static HTML audit with an Apify Actor. 15 URLs sampled, 12 returned HTML and were analysed. This is a quick triage, not a full WCAG 2.2 audit.

Why this matters

Italian public-sector websites are used by millions of people, including older citizens and people with low vision. The European Accessibility Act has made accessibility a legal requirement for many digital services. Yet most public bodies still publish pages with missing alternative text, unlabeled form fields or broken heading structures.

I ran a static audit on a small sample of Italian public administration websites to see what a quick, automated triage can catch.

Sample

15 URLs: government, tax agency, social security, statistics, major cities and regions.

12 returned HTML and were analysed.
3 could not be analysed: one returned HTTP 403 to the bot, two timed out.
Average score: 81.6 / 100.
Best: 97. Worst: 64.

What the audit checks

The Actor runs static checks on each page:

<title> and meta description
lang on <html>
<h1> presence, multiple H1s, heading-level skips
images without alt
links without accessible text
form fields without an associated label
viewport meta
noindex
canonical link
thin content
JSON-LD structured data

It does not replace a manual WCAG audit. It is a fast way to find the obvious issues before spending hours with a screen reader.

Results

Issue	Sites affected	Share
Missing canonical	12 / 12	100%
Missing structured data (JSON-LD)	10 / 12	83%
Missing meta description	6 / 12	50%
Multiple `<h1>`	5 / 12	42%
Heading-level skips	5 / 12	42%
Links without accessible text	5 / 12	42%
Images without `alt`	3 / 12	25%
Form fields without a label	3 / 12	25%

Aggregate counts:

467 images found; 13 without alt (~2.8%).
121 form fields found; 16 without an associated label (~13%).

What stands out

Canonical is missing everywhere. 12 out of 12. For public-sector sites with multiple domains, aliases and old URLs, this is a real SEO problem: it splits ranking signals and makes duplication harder to control.

Structured data is rare. 10 out of 12 have no JSON-LD at all. For public services, structured data is how search engines understand opening hours, locations, events and FAQs.

Half the sample has no meta description. Search engines will generate their own snippet, often from the first paragraph of the page.

Accessibility issues are concentrated. Only 3 sites have images without alt and 3 have unlabeled form fields, but those that do have multiple instances. The aggregate numbers are small, which suggests that the basics are often handled — the gap is in consistency, not awareness.

The caveat

This is a static audit. It cannot detect:

contrast ratios
keyboard traps
ARIA misuse that only appears at runtime
screen-reader behaviour
issues introduced by JavaScript

A 25% failure rate on image alt does not mean 25% of the site is inaccessible. It means the quick triage found a fixable issue on 3 of 12 homepages.

How I ran it

I used an Apify Actor that crawls a page, runs the checks and returns one JSON item per page with a score and a list of issues. For a bulk run, the same engine accepts a list of URLs.

The data in this article comes from one run on 15 URLs. The full run took a few minutes.

What I'd fix first

If I were responsible for one of these sites:

Add a canonical URL to every page.
Add a meta description to the templates that lack one.
Run an alt audit on the media library, not just the homepage.
Check every form field has a real <label>.
Add JSON-LD for the organisation and the main services.

None of these require a redesign. They are template-level fixes.

The audit was run with the Website Accessibility & SEO Audit Actor on Apify. The sample is small and public; results are indicative, not a compliance assessment.