Find broken links and dead sitemap URLs without installing a crawler

Salih Küçükkayış — Tue, 13 Oct 2026 14:06:15 +0000

Desktop crawlers are great until you need the check to run every week, on a server, for several sites, and land in a spreadsheet. Then you want something you can call from a schedule or an API.

I built two small Apify Actors for this. Disclosure: I am the developer, and both are paid per result (cents per run for a small site).

1. Broken links on the pages people actually see

Broken Link Checker crawls a site, collects every link and image on the pages it visits, and checks each URL once:

{
  "startUrls": [{ "url": "https://crawler-test.com/" }],
  "maxPagesToCrawl": 5,
  "maxLinksToCheck": 150
}

On that test site, 5 pages gave 143 unique URLs and 8 broken ones in under a minute. Each row says which pages link to the broken URL and with what link text, so you know what to edit:

{
  "url": "https://crawler-test.com/robots_protocol/non_200_with_noindex",
  "status": 404,
  "severity": "error",
  "problem": "Not found (404)",
  "foundOn": [{ "page": "https://crawler-test.com/", "text": "Non-200 with Noindex", "tag": "a" }]
}

Details that matter in practice:

HEAD first, and every error is confirmed with a GET (many servers answer HEAD wrongly), without downloading the file.
401, 403 and 429 are warnings, not "broken": the page exists but refused a bot.
robots.txt is respected; a host that does not answer at all is reported as unreachable, not hidden.
With reportRedirects, internal links that go through a 301/302 are listed too. Updating them saves a hop.

2. Pages your sitemap promises but cannot deliver

A sitemap should list only live pages. Sitemap Extractor reads every sitemap (robots.txt, indexes, .gz) and, with checkStatus, checks each URL:

{
  "startUrls": ["https://crawler-test.com/test_sitemap.xml"],
  "checkStatus": true,
  "maxUrlsPerSite": 60
}

18 URLs checked in 9 seconds. Redirects come back with their final URL, robots-blocked pages are marked, dead domains have a checkError.

Schedule it weekly with Changed since = 7 days to check only new or changed pages.

Run it from code

curl -X POST "https://api.apify.com/v2/acts/saliga~broken-link-checker/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com/"}],"maxPagesToCrawl":50}'

The response is the list of problem links as JSON. Pipe it into a Slack message, a GitHub issue or a sheet.

More copy-paste examples (Python, Node.js, n8n): github.com/msalihk/apify-actor-examples

My PDF converter turned every borderless table into loose numbers. The fix runs on a CPU.

Salih Küçükkayış — Sun, 11 Oct 2026 18:35:01 +0000

Document to Markdown is an Apify Actor I published this week. You give it links to PDFs, Word, PowerPoint or Excel files and get Markdown back, ready for an LLM: headings, lists, page markers and, optionally, chunks for a RAG pipeline with the page number and the nearest heading on each one.

The first version handled tables the way most text extraction does, which is to say it did not. I only found out how badly when I put it next to the competition.

Four files, four tools

I picked four public test PDFs that people who build PDF tools already use to embarrass them:

a report page with a borderless table: horizontal rules only, a header that spans columns;
a dense budget table, nine columns, no lines at all;
a scanned two-column page with bullet points;
a two-column article with a real table on its last page.

Then I ran the same four files through my Actor and three others in the Apify Store, each once, from Apify's free credit.

My converter lost on the two things that mattered. On both tables the cells came out one per line, so the budget became a long column of loose numbers. A model reading that cannot tell which number belongs to which row. The scanned page came out with "¢" where the bullets were, and every paragraph broken at the end of each scanned line.

The best result came from an ML-based converter with OCR switched on. Its tables were clean Markdown. It also took 13 minutes and 5 seconds for the four files and cost $0.148. One page of OCR alone took about eight minutes.

That is a fine trade for a handful of important documents. It is not one for a pipeline that ingests a few thousand reports a month.

What closed the gap

Borderless tables. Text in a PDF has coordinates. When numbers line up in the same horizontal positions down a page, they are a table, lines or no lines. Camelot's "stream" mode (MIT licensed) finds those columns from the spacing alone, in 0.1 to 0.3 seconds a page on one CPU.

The catch is that a two-column article is also text in two aligned columns, so stream mode happily turns prose into a two-column "table". I only run it on pages that look numeric, and I throw away anything with fewer than three columns, too few numbers, or cells full of long sentences. The two-column article now stays prose, and the real table on its last page is still found.

Rows a PDF packs into one cell. Some generators put a whole column into one text box with line breaks. Those are split back into rows when there are at least three lines or they are mostly numbers, so a two-line header stays one header.

Scanned pages. Tesseract already knows which words form a line, a paragraph and a block. Rebuilding the Markdown from that instead of from raw lines fixed the broken paragraphs, and mapping the characters Tesseract reads bullets as ("¢", "©", "•" and friends) to - fixed the lists. Lines much taller than the page's median become headings.

After the changes, on the same four files: three tables on the two-page borderless report, the budget as one nine-column table with the same rows as the ML converter, the scanned page with its list and headings. All of it took 23 seconds on one CPU, 21 of them OCR. On Apify, the four files took 16 seconds and $0.001 of compute.

It is not as good as the ML converter everywhere. A header cell that spans columns comes out as empty cells next to it, and a table inside a scanned page comes out as text. But it is close on the tables that matter, and it is priced at $2 per 1,000 pages ($5 per 1,000 pages that need OCR), with no per-run file limit on any Apify plan, the free one included.

Using it from n8n

The simplest way in is the Apify API's run-sync-get-dataset-items endpoint: one HTTP request that runs the Actor, waits, and returns the results.

Copy the workflow below and paste it onto an empty n8n canvas (Ctrl/Cmd+V).
Create a Header Auth credential: name Authorization, value Bearer YOUR_APIFY_TOKEN.
Put your file URLs in the Run Actor node and run it.

A Code node turns every chunk into one n8n item, with what you need to cite the source:

{ "text": "...", "source": "https://.../report.pdf", "pageStart": 4, "pageEnd": 5, "heading": "2.1 Results" }

Connect that to an embeddings node and a vector store, and keep source and pageStart as metadata, so an answer can say which page it came from.

The n8n workflow (JSON)

{
  "name": "PDF and Office files → Markdown RAG chunks (Apify Document to Markdown)",
  "nodes": [
    { "parameters": {}, "name": "Start", "type": "n8n-nodes-base.manualTrigger", "typeVersion": 1, "position": [0, 0] },
    {
      "parameters": {
        "method": "POST",
        "url": "https://api.apify.com/v2/acts/saliga~document-to-markdown/run-sync-get-dataset-items",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "{\n  \"urls\": [\n    \"https://raw.githubusercontent.com/microsoft/markitdown/main/packages/markitdown/tests/test_files/test.pdf\",\n    \"https://raw.githubusercontent.com/microsoft/markitdown/main/packages/markitdown/tests/test_files/test.docx\"\n  ],\n  \"chunkSize\": 1500,\n  \"chunkOverlap\": 200,\n  \"extractTables\": true\n}",
        "options": { "timeout": 310000 }
      },
      "name": "Run Actor", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.2, "position": [220, 0]
    },
    {
      "parameters": {
        "jsCode": "// One n8n item per RAG chunk, with the metadata you need for citations.\nconst out = [];\nfor (const doc of $input.all()) {\n  for (const c of doc.json.chunks ?? []) {\n    out.push({ json: { text: c.text, source: doc.json.url, title: doc.json.title ?? doc.json.fileName,\n      pageStart: c.pageStart, pageEnd: c.pageEnd, heading: c.heading, chunk: c.index } });\n  }\n}\nreturn out;"
      },
      "name": "One item per chunk", "type": "n8n-nodes-base.code", "typeVersion": 2, "position": [440, 0]
    }
  ],
  "connections": {
    "Start": { "main": [[{ "node": "Run Actor", "type": "main", "index": 0 }]] },
    "Run Actor": { "main": [[{ "node": "One item per chunk", "type": "main", "index": 0 }]] }
  },
  "settings": { "executionOrder": "v1" }
}

Without n8n it is one request:

curl -X POST "https://api.apify.com/v2/acts/saliga~document-to-markdown/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/report.pdf"], "chunkSize": 1500}'

An AI agent can use it the same way through the Apify MCP server: it passes file URLs and gets Markdown and chunks back.

The lesson I keep relearning: my own test files said the converter was fine. Four files someone else picked to be hard said otherwise in an afternoon.

DEV Community: Salih Küçükkayış