DEV Community: Mohamed Zaghloul

Turn any website into clean Markdown for RAG (incremental sync, llms.txt)

Mohamed Zaghloul — Sun, 18 Oct 2026 16:18:25 +0000

If you've ever fed a website into a RAG pipeline, you know the boring part isn't the embeddings. It's getting clean text. Nav menus, cookie banners, "related posts" widgets and footers all end up in your chunks, and then every time the site changes you either re-scrape everything or guess what changed.

I built a small Apify Actor for exactly this, called Sitemap to Markdown. This post walks through a real run on the htmx essays: what goes in, what comes out, how re-runs only pick up changes, and what it costs.

The idea: use the sitemap, not a crawler

Most sites already publish a list of every page and when it last changed: sitemap.xml. So instead of crawling links, the Actor:

Finds the sitemap on its own (robots.txt, /sitemap.xml, sitemap indexes, .xml.gz)
Keeps only the URLs under the path you gave it
Downloads each page with plain HTTP (no headless browser)
Strips the boilerplate and converts the main content to Markdown
Optionally splits it into RAG-sized chunks and writes llms.txt / llms-full.txt

If a site has no sitemap, it falls back to a shallow same-site link crawl.

Input

This is the exact input I ran:

{
  "startUrls": ["https://htmx.org/essays"],
  "maxPages": 0,
  "incremental": true,
  "incrementalStateKey": "devto-demo-htmx",
  "chunkSize": 400,
  "createLlmsTxt": true
}

maxPages: 0 means no limit: every essay URL in the sitemap.
incremental: true makes the Actor remember what it has seen, under the name in incrementalStateKey.
chunkSize: 400 splits each page into chunks of about 400 tokens, by heading and paragraph.

Output

You get one dataset record per page. Each page's markdown starts with YAML front matter, so it can go straight into a file-based pipeline:

---
title: "</> htmx ~ A Real World wasm to htmx Port"
url: "https://htmx.org/essays/a-real-world-wasm-to-htmx-port/"
canonical_url: "https://htmx.org/essays/a-real-world-wasm-to-htmx-port/"
lastmod: "2025-01-10"
language: "en"
word_count: 617
thin_content: false
---

# A Real World wasm to htmx Port

Joe Fioti

January 10, 2025

When I was in college, I wrote some customer service software that tied together ...

The rest of the record (trimmed):

{
  "type": "page",
  "url": "https://htmx.org/essays/a-real-world-wasm-to-htmx-port/",
  "canonicalUrl": "https://htmx.org/essays/a-real-world-wasm-to-htmx-port/",
  "lastmod": "2025-01-10",
  "changeStatus": "new",
  "contentHash": "958fdb17b6246b5bea12b422b88cc12d12204a467949cfee7522513928b8917b",
  "wordCount": 617,
  "tokenEstimate": 897,
  "chunkCount": 3,
  "chunks": [
    { "index": 0, "heading": "A Real World wasm to htmx Port", "text": "# A Real World wasm to htmx Port\n\nJoe Fioti ...", "tokenEstimate": 324 }
  ]
}

A few fields that matter for RAG:

chunks: each chunk keeps its heading, so you can store it as metadata and cite sources.
contentHash: a SHA-256 of the Markdown, handy for "only re-embed what changed".
canonicalUrl: lets you skip duplicate paths. The Actor already skips exact duplicates by default.
thin_content: pages with almost no main content (under 50 words by default) are flagged as thin_page instead of being written like a normal page. That includes JS-only pages, since there's no browser rendering. They're also left out of llms.txt, so they don't end up embedded as noise.

The run also saves llms.txt (a title + link + description index) and llms-full.txt (every page's Markdown in one file) to the key-value store. The second one is handy for uploading a whole site to ChatGPT, Claude Projects or NotebookLM. Here's the start of the generated llms.txt:

# htmx.org

> Pages from https://htmx.org/essays, converted to Markdown by the Sitemap to Markdown Actor on 2026-10-18.

## Pages

- [</> htmx ~ Architectural Sympathy](https://htmx.org/essays/architectural-sympathy/): In this essay, Carson Gross explores architectural sympathy as a software design principle ...

Incremental sync: the part I actually care about

Re-scraping a whole docs site every night to catch two edited pages is wasteful. With incremental: true, the Actor keeps a small state record in a named key-value store in your own Apify account. On each run it:

skips pages whose sitemap lastmod hasn't changed (they aren't even downloaded)
downloads the rest, compares content hashes, and only outputs pages that are new or changed
writes a CHANGES record listing new, changed, unchanged and removed URLs

Here's what happened on the last two runs of this input. Earlier runs, with a page cap, had already synced 41 of the 56 essay pages:

Run	Pages output	Runtime	CHANGES
Catch-up run	the 15 remaining pages, all `new`	7.1 s	`new: 15, unchanged: 41`
Re-run right after	0 pages	1.6 s	`new: 0, changed: 0, unchanged: 56, removed: 0`

The CHANGES record is what you wire into your vector store: upsert new and changed, delete removed. Then a nightly schedule only does real work on days the site actually changed.

Two honest caveats:

removed is only reliable when the run reads the whole sitemap: maxPages: 0 and no lastmodAfter date filter.
maxPages limits new and changed pages per run, so with a cap, a big site fills in over several runs.

What it costs

It's pay-per-result on Apify:

$0.50 per 1,000 pages saved (that's $0.0005 a page), plus a $0.00005 start fee per run
Chunks live inside the page record, so chunking costs nothing extra; llms.txt is free too
A re-run where nothing changed outputs nothing, so you pay only the start fee

The full htmx essays section (56 pages) costs under 3 cents the first time and about nothing on a quiet night. You can set a max cost per run in Apify, and the Actor stops cleanly at that limit.

When not to use it

It's plain HTTP, so it won't render JavaScript-only SPAs, log in, or get past aggressive bot protection. Pages from those sites come back flagged as thin, or as a BLOCKED error instead of silently empty. For those, a browser-based crawler like Apify's Website Content Crawler is the better (more expensive) tool. This one is aimed at docs, blogs, help centers and marketing sites, which is most of what people actually want in a RAG index.

Try it

Store page: Sitemap to Markdown
It runs from the Apify console, the API, or as a tool for AI agents through the Apify MCP server.

It's new, so I'd genuinely like feedback. If the Markdown isn't clean enough for your pipeline, or a field is missing, tell me in the comments or open an issue on the Actor page.

Validate an IBAN and VAT number before paying an invoice

Mohamed Zaghloul — Sun, 18 Oct 2026 16:17:04 +0000

A mistyped IBAN or a made-up VAT number on a supplier invoice is cheap to catch before payment and expensive to fix after. Here's a small pre-payment check you can add to any accounts-payable flow with two HTTP calls.

I'll use the Invoice & Payment ID Validator API on RapidAPI (I built it; there's a free plan with 100 requests a month): https://rapidapi.com/MohamedBahaa/api/invoice-payment-id-validator

1. Check the IBAN

curl --request GET \
  --url 'https://invoice-payment-id-validator.p.rapidapi.com/v1/iban?value=DE89370400440532013000' \
  --header 'x-rapidapi-host: invoice-payment-id-validator.p.rapidapi.com' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY'

Real response:

{
  "input": "DE89370400440532013000",
  "iban": "DE89370400440532013000",
  "country": "DE",
  "check_digits": "89",
  "bban": "370400440532013000",
  "valid": true,
  "country_name": "Germany",
  "bank_code": "37040044",
  "branch_code": null,
  "account_number": "0532013000",
  "formatted": "DE89 3704 0044 0532 0130 00",
  "sepa": true,
  "errors": []
}

What it checks: the country's IBAN length, the ISO 13616 mod-97 checksum, and SEPA membership. It also splits out the bank code, branch code, and account number for 20 common countries. Change one digit and you get "valid": false with "errors": ["bad_checksum"].

What it doesn't check: whether the account is open or who owns it. A valid checksum means the IBAN is well-formed, not that it belongs to your supplier. For that you still need the supplier's confirmation, or your bank's payee-verification service where one exists.

2. Check the EU VAT number, live against VIES

curl --request GET \
  --url 'https://invoice-payment-id-validator.p.rapidapi.com/v1/vat/eu?value=IE6388047V&live=true' \
  --header 'x-rapidapi-host: invoice-payment-id-validator.p.rapidapi.com' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY'

Real response (request_date trimmed):

{
  "input": "IE6388047V",
  "valid": true,
  "country": "IE",
  "number": "6388047V",
  "vat": "IE6388047V",
  "format_valid": true,
  "checksum": "not_checked",
  "errors": [],
  "vies": {
    "checked": true,
    "registered": true,
    "name": "GOOGLE IRELAND LIMITED",
    "address": "3RD FLOOR, GORDON HOUSE, BARROW STREET, DUBLIN 4",
    "error": null
  }
}

With live=true, the API asks the EU's VIES service whether the number is registered. When the member state shares it, you also get the registered name and address, which you can compare with the invoice. Without live=true, it only checks the format (fast, offline). VIES is sometimes down for individual countries. When it is, vies.error tells you, so treat that as "retry later", not "invalid".

UK VAT (/v1/vat/gb), US EIN (/v1/ein) and BIC/SWIFT (/v1/bic) work the same way with ?value=.

3. Check a whole invoice in one call

curl --request POST \
  --url 'https://invoice-payment-id-validator.p.rapidapi.com/v1/batch' \
  --header 'content-type: application/json' \
  --header 'x-rapidapi-host: invoice-payment-id-validator.p.rapidapi.com' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY' \
  --data '{"items":[{"type":"iban","value":"DE89370400440532013000"},{"type":"eu_vat","value":"IE6388047V","live":true}]}'

Up to 50 IDs per call (up to 10 of them live VIES checks).

4. A simple pay / hold rule

Hold if iban.valid is false
Hold if vies.registered is false
Flag for review if the VIES name doesn't roughly match the supplier name on the invoice
Retry later if vies.error is set
Otherwise, pass it to approval

5. Do it in n8n

If your invoices arrive by email, my free n8n template turns vendor invoice PDFs in Gmail into clean rows in Google Sheets: https://mohabaha0.gumroad.com/l/ap-invoice-triage-free

To add the checks above, put an HTTP Request node after the extraction step: use POST /v1/batch with the JSON body from step 3 and your two x-rapidapi-* headers. Then add an IF node with the pay / hold rule.

If you use AI agents, the API is also an MCP server at /mcp, so Claude or Cursor can call the validators as tools.

Questions or edge cases (odd country formats, VIES quirks)? Drop them in the comments.

Extracting invoices from Gmail with n8n and Gemini, then checking the IBAN before anyone pays it

Mohamed Zaghloul — Sun, 18 Oct 2026 08:41:33 +0000

Most invoice automations stop at extraction: an LLM pulls the vendor, total and bank details out of a PDF, and the numbers go straight into a spreadsheet or ledger. The risky part is that an LLM can read a perfectly formatted IBAN with one wrong digit, and nothing downstream notices.

This post walks through a free n8n template that handles the extraction, and a small API I built that handles the checksum part. They're separate pieces: the template doesn't call the API out of the box. I'll show how to add it.

Part 1: the free n8n template

Nodes, in order:

Gmail Trigger: polls for unread mail labelled invoices with a PDF attached.
Split PDF Attachments (Code node): one item per PDF, so an email with three invoices produces three rows.
Extract From File: reads the PDF's text layer. Scanned image-only PDFs need OCR first.
Gemini + Structured Output Parser: returns vendor, invoice number, PO number, dates, currency, subtotal, tax, total, line items and a confidence score as strict JSON. Temperature 0, retries on rate limits.
Normalize Invoice (Code node): cleans numbers and dates, checks that subtotal + tax = total, and marks low-confidence rows needs_review.
Google Sheets: appends the row. Then Gmail marks the email read so it's never processed twice.

Setup takes about 10 minutes: a Gmail label and filter, a sheet with an Invoices tab, and three credentials (Gmail, Sheets, a free Gemini key from Google AI Studio).

Two things that made it more reliable:

Strict schema over free text. The Structured Output Parser rejects anything that isn't the expected JSON, so a malformed response fails loudly instead of writing garbage.
Check the math in code, not in the prompt. Asking Gemini "does this add up?" is slower and less reliable than one line of arithmetic in the Normalize node.

Part 2: validating payment IDs with an API

Extraction confidence doesn't tell you whether an IBAN is real. The ISO 13616 mod-97 checksum does. I put that, plus a few related checks, in a Cloudflare Worker:

IBAN: country, length, mod-97 checksum, SEPA membership
BIC: format and country
EU VAT: format, plus an optional live lookup against the EU's VIES service
UK VAT: format and checksum
US EIN: format and IRS prefix (EINs have no check digit, so that's as far as honest validation goes)

Up to 50 identifiers per batch request. Here's a real response from the keyless demo endpoint:

curl "https://invoice-validator.mohamed-bahaa53000.workers.dev/demo/v1/iban?value=DE89370400440532013000"

{
  "iban": "DE89370400440532013000",
  "country": "DE",
  "check_digits": "89",
  "bban": "370400440532013000",
  "valid": true,
  "formatted": "DE89 3704 0044 0532 0130 00",
  "sepa": true,
  "errors": []
}

The demo allows 10 calls per IP per day without a key.

Why a Worker rather than an n8n Code node? Mostly reuse: the same check is useful from Make, Zapier or a backend, and it's stateless, so it runs at the edge in milliseconds. It also isolates the slow part. VIES throttles by member state, so when it's unavailable the API returns checked: false with a reason instead of a false "invalid".

Part 3: wiring them together yourself

After Normalize Invoice, add an HTTP Request node:

Method: POST, to the /v1/batch endpoint from the RapidAPI listing
Headers: X-RapidAPI-Key and X-RapidAPI-Host
Body:

{
  "items": [
    { "id": "INV-1042-iban", "type": "iban", "value": "DE89370400440532013000" },
    { "id": "INV-1042-vat", "type": "eu_vat", "value": "DE123456789", "live": true }
  ]
}

In n8n, map each value with an expression from the previous node, for example $json.invoice_number, $json.vendor_iban and $json.vendor_vat.

You'll need to add vendor_iban and vendor_vat to the Gemini schema first. Then treat valid: false as "needs review", and checked: false from VIES as "verify manually", not as a rejection. A person sees exactly why a row was flagged.

What I took away

Fail open to a human, never closed to the ledger. Anything that can time out should route to review.
Batch external checks. One call per identifier multiplies latency and quota.
Keep each piece honest about what it does. The template extracts; the API validates; you decide how strict to be.