DEV Community: Aurelix Solutions

Turn any website into clean Markdown chunks for RAG

Aurelix Solutions — Thu, 05 Nov 2026 05:41:45 +0000

Most RAG quality problems start before the embedding model. If you feed it raw HTML, or text extracted with navigation, cookie banners and footers still attached, every chunk carries the same boilerplate. Retrieval then matches "Privacy policy | Contact us | © 2026" as often as it matches the answer.

This post covers the two steps that fix most of it: converting each page to clean Markdown of its main content, and chunking that Markdown along its headings so each chunk is a coherent section with a citation.

Disclosure: I'm Aurelix, and I built the Website to Markdown Actor used for the crawl step. It's a paid tool (prices in the Cost section). The chunking code works on Markdown from any source.

The general approach

Crawl within a scope. Usually one docs section (/docs/...), not the whole domain. Exclude changelogs, tag pages and paginated archives, which add volume and little meaning.
Keep the main content only. Use the page's <article> or <main> element when present, otherwise the largest content block. Strip nav, headers, footers, sidebars, cookie banners and scripts.
Convert to Markdown, not plain text. Headings, lists, tables and code blocks carry meaning. Markdown keeps them; plain text flattens them.
Chunk by structure first, then by size. Split at headings so a chunk is a section, then split oversized sections at paragraph boundaries. Never split inside a code block.
Carry metadata on every chunk: source URL, page title and the heading path (Page > Section > Subsection). It improves retrieval and lets the model cite its source.
Be polite. Respect robots.txt and crawl slowly. It's someone else's server.

Step 1: Crawl to Markdown

The Website to Markdown Actor takes:

startUrls (required): pages to start from; links are followed within the scope.
maxPages (default 50, up to 10,000): stop after this many pages are converted.
scope: "path" (only URLs under the start URL's folder) or "domain" (the whole site of each start URL).
includeGlobs / excludeGlobs: URL patterns, where ** matches anything and * anything except /.

Each page becomes one record:

{
  "url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
  "language": "en",
  "last_modified": "2026-11-04T07:17:58Z",
  "markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\n\n## Beginner courses\n\n...",
  "headings": [
    {"level": 1, "text": "Apify Academy"},
    {"level": 2, "text": "Beginner courses"}
  ],
  "links": [
    "https://docs.apify.com/academy/scraping-with-ai",
    "https://docs.apify.com/academy/scraping-basics-javascript"
  ],
  "word_count": 231,
  "crawl_depth": 0,
  "http_status": 200,
  "source_url": "https://docs.apify.com/academy",
  "scraped_at": "2026-11-04T11:25:08Z"
}

Links inside the Markdown are made absolute, and code blocks keep their language tag, which helps the chunker below.

Step 2: Chunk by headings

pip install apify-client

import json
import os
import re

from apify_client import ApifyClient

MAX_CHARS = 2000
MIN_CHARS = 200
HEADING = re.compile(r"^(#{1,6})\s+(.*)$")
FENCE = re.compile(r"^\s*(`{3,}|~{3,})")


def crawl(start_url, max_pages=100):
    client = ApifyClient(os.environ["APIFY_TOKEN"])
    run = client.actor("aurelix/website-to-markdown").call(
        run_input={
            "startUrls": [start_url],
            "maxPages": max_pages,
            "scope": "path",
            "excludeGlobs": ["**/changelog/**"],
        }
    )
    return list(client.dataset(run["defaultDatasetId"]).iterate_items())


def split_sections(markdown):
    """Yield (heading_path, text) per section; ignores '#' inside code fences."""
    path, lines, in_code = [], [], False
    for line in markdown.splitlines():
        if FENCE.match(line):
            in_code = not in_code
        match = None if in_code else HEADING.match(line)
        if match:
            if any(l.strip() for l in lines):
                yield list(path), "\n".join(lines).strip()
            level = len(match.group(1))
            path = path[: level - 1] + [match.group(2).strip()]
            lines = [line]
        else:
            lines.append(line)
    if any(l.strip() for l in lines):
        yield list(path), "\n".join(lines).strip()


def split_blocks(text):
    """Paragraph blocks, keeping each code fence whole."""
    blocks, current, in_code = [], [], False
    for line in text.splitlines():
        if FENCE.match(line):
            in_code = not in_code
        if not line.strip() and not in_code:
            if current:
                blocks.append("\n".join(current))
                current = []
        else:
            current.append(line)
    if current:
        blocks.append("\n".join(current))
    return blocks


def pack(text):
    if len(text) <= MAX_CHARS:
        return [text]
    chunks, current = [], ""
    for block in split_blocks(text):
        if current and len(current) + len(block) + 2 > MAX_CHARS:
            chunks.append(current)
            current = block
        else:
            current = f"{current}\n\n{block}" if current else block
    if current:
        chunks.append(current)
    return chunks


def chunk_page(page):
    chunks = []
    for path, text in split_sections(page.get("markdown") or ""):
        for part in pack(text):
            if chunks and len(part) < MIN_CHARS and chunks[-1]["heading_path"] == path:
                chunks[-1]["text"] += "\n\n" + part
                continue
            chunks.append({"heading_path": path, "text": part})
    for i, c in enumerate(chunks):
        c.update({
            "id": f"{page['url']}#{i}",
            "url": page["url"],
            "title": page.get("title"),
            "language": page.get("language"),
            "last_modified": page.get("last_modified"),
            "embed_text": " > ".join(c["heading_path"]) + "\n\n" + c["text"],
        })
    return chunks


def main():
    pages = crawl("https://docs.apify.com/academy", max_pages=100)
    total = 0
    with open("chunks.jsonl", "w", encoding="utf-8") as f:
        for page in pages:
            for chunk in chunk_page(page):
                f.write(json.dumps(chunk, ensure_ascii=False) + "\n")
                total += 1
    print(f"{len(pages)} pages -> {total} chunks in chunks.jsonl")


if __name__ == "__main__":
    main()

Why these choices

Heading path in the embedded text. A chunk that reads "Set the timeout option to 60" means little alone. Prefixing it with Crawler configuration > Timeouts gives the embedding model the context a human gets from the page layout. Store the bare text separately for display.
Characters, not tokens. 2,000 characters is roughly 400 to 500 tokens of English prose, a common range for retrieval chunks. If your embedding model has a hard token limit, swap len() for a tokenizer count; the logic doesn't change.
Code fences are never split. Half a code example is worse than none. A single code block longer than MAX_CHARS stays one oversized chunk, which is usually the right trade.
Tiny trailing chunks get merged into the previous chunk of the same section, so you don't embed one-line orphans.
Stable IDs (url#index) let you re-crawl and upsert. Pair them with last_modified to re-embed only pages that changed.

From chunks.jsonl, loading into a vector store is one loop: embed embed_text, store text, url, title and heading_path as metadata, and return url with each answer as the citation.

Getting the crawl scope right

Start at the section you need: https://example.com/docs/ with scope: "path" crawls only /docs/....
Versioned docs: exclude old versions with excludeGlobs: ["**/v1/**"], or the same answer appears three times with three different version numbers.
Paginated listings: excludeGlobs: ["**?page=*"] skips archive pages that are lists of links.
Check word_count after a test run of 20 pages. Pages with very few words are often index pages; filter them out before chunking if they add nothing.

Limits

No JavaScript rendering. It reads the HTML the server sends, which covers most docs sites, blogs, knowledge bases and server-rendered React or Next.js. Client-side-only pages (an empty app shell) are skipped, listed in the log and not charged.
HTML only. PDFs, images and other files are skipped.
One site per start URL. www. and the bare domain count as one site; other subdomains aren't followed.
Speed. At most two new requests per second per site, slower when robots.txt sets a Crawl-delay. Conversion is CPU-bound: the README measures about 1,000 large documentation pages an hour at 256 MB, so the default one-hour run fits roughly 1,000 to 3,000 pages. The run stops itself before its timeout and keeps the pages done so far.
Pages over 2 MB of HTML are skipped and not charged.
Main-content detection is automatic, so on unusual layouts a little boilerplate can remain or a side block can be dropped. Spot-check a few pages.
Robots.txt is always obeyed and can't be turned off. Pages behind logins, paywalls or bot challenges aren't accessible.
Crawl only sites you have the right to use this way. Copyright in the content stays with its owner.

Why this tool over the alternatives

Compared with the leading alternatives on the Apify Store (the full table is on each Store page):

A flat $1.50 per 1,000 pages: 34% under Apify's RAG Web Browser, and no compute-unit billing at 8 GB like Apify's Website Content Crawler.
Crawls a whole site at 256 MB, robots.txt always obeyed, failed, duplicate and empty pages never charged.
Not offered: JavaScript rendering, which both Apify Actors have.

Cost

At the time of writing, the README lists $1.50 per 1,000 pages on Apify's free plan and $1.05 per 1,000 on paid plans (Bronze, Silver, Gold), plus $0.001 per run. Skipped pages (robots.txt, errors, non-HTML, duplicate URLs or identical Markdown, JavaScript-only pages, pages with no main content) aren't charged.

The 100-page example costs at most $0.15. Apify's free plan includes $5 of credit a month, which the README puts at about 3,300 pages.

Monitor your competitors' Facebook and Instagram ads with Python

Aurelix Solutions — Thu, 05 Nov 2026 05:40:32 +0000

Meta publishes the ads businesses run on Facebook and Instagram in its Ad Library. Anyone can open it in a browser without logging in, which makes it a free and direct source of competitive ad intelligence.

The problem is that checking it by hand doesn't scale. You open a competitor, scroll, try to remember what was there last week, and repeat for the next competitor. This post shows how to turn it into one weekly "new competitor ads" file with Python.

Disclosure: I'm Aurelix, and I built the Apify Actor used below. It is a paid tool; prices are in the Cost section. The approach is the same whatever tool fetches the data.

The general approach

Decide what to watch. A keyword, a Facebook page ID, or an Ad Library search URL copied from your browser.
Fetch with a stable ad ID. The Ad Library gives every ad an ID. That is what lets you tell "new ad" from "ad I already saw".
Keep a memory between runs. Either the fetcher remembers what it returned last time, or you store IDs yourself. The tool below does the first; the script also writes a local CSV so you keep history.
Keep a few useful columns. Advertiser, ad ID, start date, format, headline, text, link and media are enough to sort and skim.
Schedule weekly or daily. Most ad changes are not urgent; weekly is usually enough.

The data source

The Facebook Ad Library Actor takes queries: each entry is a keyword ("meal kit"), a page ID ("20409006880" or "page:20409006880"), or an Ad Library URL copied from your browser. Other inputs:

country: two-letter ISO code, or "ALL".
activeStatus: "active", "inactive" or "all".
mediaType: "all", "image", "video", "meme" or "none".
maxResultsPerQuery: an exact cap per query.
incremental and stateKey: return only ads not returned by a previous run with the same queries.
slackWebhookUrl and webhookUrl: optional digest of each run's results, sent to Slack or any URL (for example a Zapier, Make or n8n webhook).
includeEuTransparency: optional EU reach, reach by country, age and gender, targeting, and payer and beneficiary for ads delivered in the EU.

Political and issue ads, and ads from personal pages, are excluded. An ad record (lists and long text shortened, dates shown as their format):

{
  "ad_id": "4477357355914906",
  "page_id": "1811034678977534",
  "page_name": "Coohom",
  "page_url": "https://facebook.com/coohom",
  "page_categories": ["Business"],
  "page_like_count": 39638,
  "is_active": true,
  "start_date": "YYYY-MM-DDThh:mm:ssZ",
  "end_date": "YYYY-MM-DDThh:mm:ssZ",
  "platforms": ["facebook", "instagram", "audience_network"],
  "search_country": "ALL",
  "reached_countries": [],
  "ad_text": "From design to production, all in one connected workflow. ...",
  "title": "AI-Powered Design to Production",
  "link_description": "Create designs and renders in minutes.",
  "caption": "fb.me",
  "cta_text": "Get quote",
  "cta_type": "GET_QUOTE",
  "link_url": "http://fb.me/",
  "display_format": "DCO",
  "image_urls": ["https://scontent-phl2-1.xx.fbcdn.net/..."],
  "video_urls": ["https://video-phl2-1.xx.fbcdn.net/..."],
  "cards": ["..."],
  "variants_count": 1,
  "collation_id": "1087839470334171",
  "spend_range": null,
  "impressions_range": null,
  "impressions_text": null,
  "reach_estimate": null,
  "currency": null,
  "ad_library_url": "https://www.facebook.com/ads/library/?id=4477357355914906",
  "source_url": "https://www.facebook.com/ads/library/?id=4477357355914906",
  "scraped_at": "YYYY-MM-DDThh:mm:ssZ"
}

The script

pip install apify-client

import csv
import os
from datetime import date
from pathlib import Path

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

RUN_INPUT = {
    "queries": ["meal kit"],
    "country": "US",
    "activeStatus": "active",
    "mediaType": "all",
    "maxResultsPerQuery": 100,
    "incremental": True,
    "stateKey": "meal-kit-watch",
}

COLUMNS = ["advertiser", "ad_id", "first_seen", "format", "headline", "text", "url", "media"]


def fetch_ads():
    run = client.actor("aurelix/facebook-ad-library-scraper").call(run_input=RUN_INPUT)
    return list(client.dataset(run["defaultDatasetId"]).iterate_items())


def to_row(ad):
    return {
        "advertiser": ad.get("page_name"),
        "ad_id": ad.get("ad_id"),
        "first_seen": (ad.get("start_date") or "")[:10],
        "format": ad.get("display_format"),
        "headline": ad.get("title"),
        "text": (ad.get("ad_text") or "")[:300],
        "url": ad.get("ad_library_url"),
        "media": (ad.get("video_urls") or ad.get("image_urls") or [None])[0],
    }


def main():
    rows = [to_row(a) for a in fetch_ads()]
    rows.sort(key=lambda r: r["first_seen"], reverse=True)

    out_dir = Path("ad_watch")
    out_dir.mkdir(exist_ok=True)
    weekly = out_dir / f"new_ads_{date.today().isoformat()}.csv"
    with weekly.open("w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=COLUMNS)
        writer.writeheader()
        writer.writerows(rows)

    history = out_dir / "all_ads.csv"
    new_file = not history.exists()
    with history.open("a", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=COLUMNS)
        if new_file:
            writer.writeheader()
        writer.writerows(rows)

    by_advertiser = {}
    for r in rows:
        by_advertiser[r["advertiser"]] = by_advertiser.get(r["advertiser"], 0) + 1
    for advertiser, n in sorted(by_advertiser.items(), key=lambda kv: -kv[1]):
        print(f"{n:4} new  {advertiser}")


if __name__ == "__main__":
    main()

On the first run, incremental returns everything that matches, which becomes your baseline. From then on each run returns only ads that weren't returned before, so the dated CSV is literally "what's new since last time", and all_ads.csv grows into a history you can analyse later.

Reading the results

Some things that are cheap to compute from these rows and useful in practice:

Long-running ads. Compare start_date with today. Advertisers tend not to keep paying for ads that fail, so long runners are a hint (not proof) of what converts.
Format shifts. A competitor that suddenly moves from image to video (display_format, video_urls) is changing strategy.
Landing pages and offers. link_url, cta_text and the cards of carousel ads show where the traffic is sent and what the offer is.
Ad variants. Meta groups near-identical versions; each version is its own row with variants_count, so a burst of rows with the same text often means A/B testing.

Limits

Data comes from the public Ad Library web pages, not Meta's official API. A Meta layout change can break it until it is fixed (a daily health run checks for this).
Meta rate-limits datacenter addresses quickly, so the Actor starts on datacenter proxy and switches to residential proxy, included in the price, after the first rate limit.
Spend, impressions and reached countries are filled only where Meta publishes them, which is mostly EU and political ads, and political ads are excluded, so expect those fields to be empty most of the time.
The personal-page filter is a heuristic on page categories and names, so an occasional edge case can slip through either way.
If Meta keeps rate-limiting a query, that query stops early with a PARTIAL status message; you keep (and pay only for) the ads already saved.

Why this tool over the alternatives

Compared with the two leading alternatives on the Apify Store (the full table is on the Store page):

$0.70 per 1,000 ads vs $5.80 for Apify's own Facebook Ads Scraper and $0.75 for the most-used community scraper.
Exact limits (ask for 100, pay for at most 100), plain keywords as input, and an only-new-ads mode with a Slack/webhook digest that neither offers.
No reliability claim: both alternatives are above 99% run success.

Cost

At the time of writing, from the README's pricing section: $0.70 per 1,000 ads on the free plan and $0.60 on paid Apify plans, plus $0.003 per run start. EU transparency data, when turned on, adds $0.20 per 1,000 ads that get it. Excluded political and personal-page ads and duplicates are not charged.

For the example above, the worst case is 100 ads per run, about $0.07 plus the start fee. Incremental runs after the first usually return far fewer.

Find new construction projects in Canadian cities from open permit data

Aurelix Solutions — Thu, 05 Nov 2026 05:09:29 +0000

If you're a contractor, trade or building-supply company, the earliest public sign of a project is usually a development application or a building permit. A rezoning in Toronto, a new detached garage permit in Calgary, a renovation permit in Montréal: each one is a project that will need people and materials, and each one is published by the city.

The data is open. The problem is that every city publishes it differently: different portal software (CKAN, ArcGIS and others), different field names, different date meanings, French in Montréal. This post shows the general approach for turning that into one weekly list of leads, and the Python to run it.

Disclosure: I'm Aurelix, and I built the Canada Development Applications Actor used below. It's a paid tool (prices in the Cost section). Everything it reads is open data, so you can also build this yourself from the city portals.

The general approach

Know which datasets exist. Not every city publishes applications. Some publish only issued permits, which come later in the process. Both are useful, but they answer different questions: applications tell you a project is being planned; issued permits tell you work is about to start.
Normalise to one schema. Address, type, status, the key dates, value, and the builder company where published.
Filter for your trade. By type ("new", "commercial", "addition"), by keywords in the description ("storey", "mixed-use", "basement"), by ward, and by project value.
Fetch only what's new or changed. A weekly job should surface new applications and the ones whose status moved (Under Review to Approved is often the moment to call).
Respect privacy. Permit data can name owners and applicants. Keep companies, drop individuals.

The datasets

The Canada Building Permits Scraper Actor reads 10 official open-data datasets, chosen with the cities input:

`cities` value	What it is
`toronto`	Development applications: rezoning, official plan, site plan, subdivision, condo, part lot
`toronto-building`	Building permit applications and issued permits
`calgary`	Development permits
`calgary-building`	Issued building permits
`edmonton`	Development permits
`edmonton-building`	Issued building permits
`vancouver`	Issued building permits
`winnipeg`	Issued development permits
`kitchener`	Issued building permits (refreshed weekly)
`montreal`	Issued building, alteration and demolition permits (refreshed weekly)

Other inputs:

daysBack (0 to 365, default 60): keep records dated on or after today minus this many days. Which date depends on the dataset: submitted date for Toronto and Calgary development permits, decision date for Edmonton development permits, issue date for the rest.
applicationTypes, statuses, wards, keywords: case-insensitive "contains" filters. Empty means all.
maxResultsPerCity (default 1,000; 0 = no limit), newest first.
incremental and stateKey: return only records not returned before plus records whose status changed.

A record:

{
  "app_id": "calgary-building:BP2026-14894",
  "city": "Calgary",
  "province": "AB",
  "application_number": "BP2026-14894",
  "application_type": "Residential Improvement Project",
  "category": "1113 - Detached Garage Dwelling Unit - New",
  "address": "83G MARBANK WY NE",
  "addresses": ["83G MARBANK WY NE"],
  "postal_code": null,
  "ward": null,
  "neighbourhood": "MARLBOROUGH",
  "description": "Additional Dwelling Unit",
  "status": "Issued Permit",
  "submitted_date": "2026-08-07",
  "decision_date": null,
  "issued_date": "2026-11-02",
  "project_value": 113393.31,
  "land_use": null,
  "permit_category": null,
  "contractor_company": "AIM BUILDER & DEVELOPMENT",
  "latitude": 51.06397737432121,
  "longitude": -113.97820309651591,
  "url": "https://data.calgary.ca/resource/c2es-76ed.json?permitnum=BP2026-14894",
  "source_url": "https://data.calgary.ca/resource/c2es-76ed.json?permitnum=BP2026-14894",
  "scraped_at": "2026-11-04T18:47:17Z"
}

Two fields matter most for contractors: project_value (published for Toronto, Calgary and Edmonton building permits, Vancouver and Kitchener) and contractor_company (Toronto and Calgary building permits, Kitchener). A builder is returned only when it's a company, never a named individual.

The weekly script

pip install apify-client

import csv
import os
from collections import Counter
from datetime import date

from apify_client import ApifyClient

MIN_VALUE = 250_000

RUN_INPUT = {
    "cities": ["toronto", "toronto-building", "calgary-building", "kitchener"],
    "daysBack": 14,
    "keywords": ["storey", "mixed-use", "addition", "new"],
    "maxResultsPerCity": 1000,
    "incremental": True,
    "stateKey": "contractor-weekly",
}

COLUMNS = [
    "city", "application_number", "status", "application_type", "category",
    "address", "neighbourhood", "ward", "submitted_date", "issued_date",
    "project_value", "contractor_company", "description", "url",
]


def main():
    client = ApifyClient(os.environ["APIFY_TOKEN"])
    run = client.actor("aurelix/canada-building-permits-scraper").call(run_input=RUN_INPUT)
    records = list(client.dataset(run["defaultDatasetId"]).iterate_items())

    def keep(r):
        value = r.get("project_value")
        return value is None or value >= MIN_VALUE

    leads = [r for r in records if keep(r)]
    leads.sort(key=lambda r: r.get("project_value") or 0, reverse=True)

    out = f"projects_{date.today().isoformat()}.csv"
    with open(out, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=COLUMNS, extrasaction="ignore")
        writer.writeheader()
        writer.writerows(leads)
    print(f"{len(records)} new or changed records, {len(leads)} written to {out}")

    by_city = Counter(r["city"] for r in leads)
    for city, n in by_city.most_common():
        print(f"  {city:12} {n}")

    builders = Counter(r["contractor_company"] for r in records if r.get("contractor_company"))
    if builders:
        print("\nMost active builders this period:")
        for name, n in builders.most_common(10):
            print(f"  {n:3}  {name}")


if __name__ == "__main__":
    main()

What it does:

14 days back with incremental on. A weekly schedule with a two-week window tolerates a skipped week; incremental mode stops you seeing the same permit twice, but still re-surfaces it if its status changes.
Keeps records without a value. Toronto development applications don't publish one, and a rezoning is often the most interesting lead of all. Filtering them out by value would throw away the earliest signals.
Counts active builders. If you're a supplier or subcontractor, the companies pulling the most permits in your area are your prospects. If you're a general contractor, they're your competition.

Tuning it for your trade

Roofing, windows, exteriors: keywords: ["addition", "new", "replace"] on the building-permit datasets.
Commercial fit-outs: applicationTypes: ["commercial"], or keywords like "tenant improvement" and "change of use". Restaurant and retail openings often show up as changes of use.
Multi-unit residential: Toronto development applications with applicationTypes: ["zoning", "site plan"] and keywords: ["storey", "mixed-use"]. These are months or years ahead of construction, which is exactly when developers pick consultants and trades.
One part of a city: wards takes a number or name (only Toronto development applications, Calgary and Edmonton development permits and Winnipeg publish a ward), or filter on latitude/longitude in Python for a radius around your yard.
Montréal: records are in French as published, so use French keywords that match how the city words its permits.

Limits

Seven cities only. Toronto Committee of Adjustment, Mississauga and Ottawa aren't covered because they have no current open dataset.
Applications vs permits. Vancouver, Kitchener and Montréal are issued building permits and Winnipeg is issued development permits, not applications. Edmonton development permits publish only a decision date. Toronto building permits include applications not yet issued (empty issued_date) and one record per permit revision.
Toronto lag. Toronto adds applications to its dataset some weeks after submission, so very short look-backs can return nothing for Toronto. The default is 60 days for this reason; my 14-day window works only because incremental mode carries earlier finds forward. If you start fresh, run once with 60 days.
Refresh rates differ. Toronto, Calgary, Edmonton, Winnipeg and Vancouver refresh daily; Kitchener and Montréal about weekly, so look back at least 8 days if you rely on them.
Coordinates and links. Toronto coordinates are converted from the city's MTM grid and accurate to about 20 m; Toronto building permits have no coordinates. Most links point to the open-data record rather than a city web page.
Look-back is capped at 365 days.
Descriptions are free text. Emails and phone numbers are removed; the rest is as published, which can be terse or inconsistent.
Owner, applicant and planner names are never returned. If you contact a builder company, keep it relevant and individual and follow CASL.

Why this tool over the alternatives

Compared with the leading alternatives on the Apify Store (the full table is on each Store page):

$3.00 per 1,000 records vs $3.50 and $5.00 for the other Canadian permit scrapers.
The only one with development applications (rezonings, site plans) and a ward filter, across 7 Canadian cities against 5 and 4.

Cost

At the time of writing, the README lists $3.00 per 1,000 applications on Apify's free plan and $2.10 per 1,000 on paid plans (Bronze, Silver, Gold), plus $0.00005 per run. Apify's free plan includes $5 of credit a month, which the README puts at about 1,600 applications.

Because you pay per record returned after filters, narrow keywords and incremental mode keep a weekly run cheap: 300 new or changed records a week would be $0.90.

Track Amazon best-seller rank changes over time with Python

Aurelix Solutions — Thu, 05 Nov 2026 05:09:28 +0000

Amazon's Best Sellers lists are a public signal, updated through the day, of what is selling in each category. One snapshot tells you what is popular. A series of snapshots tells you what is changing: which products are climbing, which just entered the top 100, which dropped out, and whose price moved before they did.

Amazon doesn't keep that history for you. This post shows how to build it: take a daily snapshot, store it as a file, and diff any two snapshots in plain Python.

Disclosure: I'm Aurelix, and I built the Amazon Best Sellers Actor used for the data step. It's a paid tool (prices in the Cost section). The snapshot-and-diff part works with data from any source.

The general approach

Pick lists, not searches. Each category and subcategory has its own Best Sellers list of up to 100 ranks (for example amazon.com/gp/bestsellers/kitchen).
Key every row by ASIN. The ASIN is Amazon's product ID and stays stable while titles get edited.
Snapshot on a fixed schedule. Lists change through the day, so compare like with like: the same time each day.
Store raw snapshots, compute diffs later. Disk is cheap; you'll want to ask questions you didn't think of on day one.
Diff by ASIN: present in both (rank moved?), only in the new one (new entry), only in the old one (dropped out).

Getting the data

The Amazon Best Sellers Actor takes:

categories (required): a slug from the Best Sellers URL ("kitchen", "pet-supplies", "beauty") or a full list URL, including subcategory URLs.
listType: "bestsellers" or "new-releases" (for bare slugs).
maxPages: 1 = top 50, 2 = top 100, which is Amazon's maximum.
onlyNewEntries and stateKey: covered below.

A record looks like this:

{
  "list_type": "bestsellers",
  "category_name": "Kitchen & Dining",
  "category_url": "https://www.amazon.com/gp/bestsellers/kitchen",
  "rank": 1,
  "asin": "B0BZYCJK89",
  "title": "Owala FreeSip Stainless Steel Water Bottle 24 oz Denim",
  "product_url": "https://www.amazon.com/dp/B0BZYCJK89",
  "image_url": "https://images-na.ssl-images-amazon.com/images/I/61Gvw0-b0eL._AC_UL300_SR300,200_.jpg",
  "price": 29.99,
  "price_max": 29.99,
  "price_text": "$29.99",
  "currency": "USD",
  "rating": 4.6,
  "reviews_count": 144563,
  "page": 1,
  "previous_rank": 1,
  "rank_change": 0,
  "new_entry": false,
  "source_url": "https://www.amazon.com/gp/bestsellers/kitchen",
  "scraped_at": "2026-11-04T11:24:07Z"
}

The Actor already compares each run with your previous run of the same list and fills previous_rank, rank_change (previous minus current, so positive means it climbed) and new_entry. That covers "what moved since yesterday". For "what moved since last Monday", or across any two dates you pick, you want your own snapshots. That's what the rest of this post builds.

Step 1: Save a daily snapshot

pip install apify-client

import json
import os
from datetime import date
from pathlib import Path

from apify_client import ApifyClient

CATEGORIES = ["kitchen", "pet-supplies"]
SNAPSHOT_DIR = Path("snapshots")


def slug_of(item):
    return item["category_url"].rstrip("/").split("/gp/")[-1].replace("/", "_")


def take_snapshot():
    client = ApifyClient(os.environ["APIFY_TOKEN"])
    run = client.actor("aurelix/amazon-best-sellers").call(
        run_input={
            "categories": CATEGORIES,
            "listType": "bestsellers",
            "maxPages": 2,
            "stateKey": "daily-snapshot",
        }
    )
    by_list = {}
    for item in client.dataset(run["defaultDatasetId"]).iterate_items():
        by_list.setdefault(slug_of(item), []).append(item)

    today = date.today().isoformat()
    for slug, items in by_list.items():
        folder = SNAPSHOT_DIR / slug
        folder.mkdir(parents=True, exist_ok=True)
        path = folder / f"{today}.json"
        path.write_text(json.dumps(items, indent=1), encoding="utf-8")
        print(f"{path}: {len(items)} products")


if __name__ == "__main__":
    take_snapshot()

Run it daily with cron (0 9 * * * python3 snapshot.py) or an Apify schedule. You end up with one snapshots/bestsellers_kitchen/<date>.json file per day. The stateKey gives this job its own "previous run" memory, so testing the Actor by hand doesn't disturb the built-in rank_change values of the scheduled runs.

Step 2: Diff two snapshots

import json
import sys
from pathlib import Path


def load(path):
    items = json.loads(Path(path).read_text(encoding="utf-8"))
    return {item["asin"]: item for item in items}


def diff(old_path, new_path, min_move=10):
    old, new = load(old_path), load(new_path)

    entered = [new[a] for a in new.keys() - old.keys()]
    dropped = [old[a] for a in old.keys() - new.keys()]

    moved = []
    repriced = []
    for asin in new.keys() & old.keys():
        before, after = old[asin], new[asin]
        delta = before["rank"] - after["rank"]
        if abs(delta) >= min_move:
            moved.append((delta, after))
        if before.get("price") is not None and after.get("price") is not None:
            if before["price"] != after["price"]:
                repriced.append((before["price"], after["price"], after))

    print(f"New in top list ({len(entered)}):")
    for item in sorted(entered, key=lambda i: i["rank"]):
        print(f"  #{item['rank']:>3}  {item['asin']}  {item['title'][:70]}")

    print(f"\nDropped out ({len(dropped)}):")
    for item in sorted(dropped, key=lambda i: i["rank"]):
        print(f"  was #{item['rank']:>3}  {item['asin']}  {item['title'][:70]}")

    print(f"\nMoved {min_move}+ places ({len(moved)}):")
    for delta, item in sorted(moved, key=lambda m: -m[0]):
        arrow = "up" if delta > 0 else "down"
        print(f"  {arrow:4} {abs(delta):>3} -> #{item['rank']:>3}  {item['title'][:60]}")

    print(f"\nPrice changed ({len(repriced)}):")
    for was, now, item in sorted(repriced, key=lambda r: r[1] - r[0]):
        print(f"  {was:>8.2f} -> {now:>8.2f}  #{item['rank']:>3}  {item['title'][:55]}")


if __name__ == "__main__":
    diff(sys.argv[1], sys.argv[2])

Usage:

python3 diff.py snapshots/bestsellers_kitchen/<earlier>.json snapshots/bestsellers_kitchen/<later>.json

A few details in there are deliberate:

Rank delta is old - new, the same sign convention as the Actor's rank_change: positive means the product climbed.
A min_move threshold. Ranks near the bottom of a list jitter by a few places every day. A threshold of 10 cuts the noise; lower it for the top 20, where small moves matter.
Prices are compared only when both snapshots have one. Some products show no price on the list (for example when they're sold only through other offers), and treating a missing price as zero would invent price drops.

Step 3: Build a rank history

Once you have a few weeks of files, turning them into one long table is a few lines:

import csv
import json
from pathlib import Path

rows = []
for path in sorted(Path("snapshots/bestsellers_kitchen").glob("*.json")):
    day = path.stem
    for item in json.loads(path.read_text(encoding="utf-8")):
        rows.append({
            "date": day,
            "asin": item["asin"],
            "rank": item["rank"],
            "price": item.get("price"),
            "reviews_count": item.get("reviews_count"),
            "title": item["title"],
        })

with open("kitchen_history.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["date", "asin", "rank", "price", "reviews_count", "title"])
    writer.writeheader()
    writer.writerows(rows)

Load that into a spreadsheet or pandas and pivot on asin × date. Two useful views:

Days in the top 100 per ASIN: sustained demand rather than a one-day spike.
reviews_count growth next to rank: a product climbing while gaining reviews quickly is selling; one climbing with flat reviews may be riding a promotion.

Only want the newcomers?

If all you need is "what joined the list", skip the snapshots and set onlyNewEntries: true. The run then returns only products that weren't on the same list at your previous run (the first run returns everything). Rank changes are filled in either way.

Limits

Amazon.com (US) only in this version. Other Amazon sites' URLs are rejected with a clear message.
Movers & Shakers isn't supported: Amazon serves those pages empty to automated visitors. Daily Best Sellers snapshots plus rank_change cover the same question.
Up to 100 ranks per list, Amazon's own maximum. Amazon sometimes repeats a product at two ranks while the list updates; the repeat is dropped, so a list can return 98 or 99 products. Your diff will see that as a "drop-out" for a day, which is another reason for a threshold.
Some products show no price on the list; price is then empty, never guessed.
A run is a snapshot of the moment it ran. Lists change through the day.
If Amazon starts answering with robot checks mid-run, the products already read are kept and the run status says "Partial". Check the product count before trusting a diff against that day.

Why this tool over the alternatives

Compared with the leading alternatives on the Apify Store (the full table is on each Store page):

$1.80 per 1,000 products, 69% under the most-used Best Sellers scraper ($5.90), and $0.38 on paid plans, the lowest of the three.
10 marketplaces including amazon.ca, plus rank change and new entries since your last run. Not offered: Movers & Shakers and Most Wished For.

Cost

At the time of writing, the README lists $1.80 per 1,000 products on Apify's free plan and $0.38 per 1,000 on paid plans (Bronze, Silver, Gold), plus $0.001 per run. Unknown categories and unsupported pages return nothing and cost nothing beyond the start fee.

Two top-100 lists a day is about 200 products, or $0.36 a day on the free plan. Apify's free plan includes $5 of credit a month, which the README puts at about 2,700 products: roughly 27 days of one top-100 list.

Export new LinkedIn jobs to Google Sheets every day with Python

Aurelix Solutions — Thu, 05 Nov 2026 04:38:22 +0000

If you run a niche job board, a hiring newsletter, or you are just job hunting in one country, you probably want the same thing: one spreadsheet that gains a row for every new posting that matches your search, every day, with no duplicates and no manual copy-paste.

This post walks through how to build that with Python. The general pattern works with any job source. I use a LinkedIn scraper for the data step, and the rest is plain Python and the Google Sheets API.

Disclosure: I'm Aurelix, and I built the LinkedIn Jobs Actor used below. It is a paid tool on the Apify Store; the pricing is in the Cost section, and the approach works with other sources too.

The general approach

A "daily export" sounds simple, but three details decide whether the sheet stays useful after a month:

A stable ID per job. Titles and companies repeat. You need an ID that is the same every time you see the same posting. On LinkedIn that is the number in /jobs/view/<id>/.
Incremental fetching. Each run should return only jobs you have not seen. You can do this on the source side (remember what was returned last time) or on your side (compare against IDs already in the sheet). Doing both is cheap insurance.
Append, never rewrite. People add notes and status columns to a shared sheet. If your script clears and rewrites the sheet, it destroys their work. Append new rows at the bottom and leave existing rows alone.

Then you schedule it: cron on a server, a GitHub Actions workflow, or a schedule on the platform that runs the scraper.

Step 1: Get the jobs

The LinkedIn Jobs Actor reads LinkedIn's public job pages (no login, no cookies) and returns one row per posting. The inputs that matter for a daily feed:

keywords: one search per entry, for example ["data engineer", "product manager"].
country: a two-letter code such as "CA" or "US". The Actor maps it to LinkedIn's exact country id and drops results whose location names another country.
datePosted: "any", "past24h", "pastWeek" or "pastMonth".
jobTypes: any of fullTime, partTime, contract, temporary, internship, volunteer, other.
maxResultsPerSearch: an exact cap per keyword (1 to 1,000, default 50).
incremental: when true, a run returns only jobs not returned by a previous run with the same searches.
stateKey: an optional name for that "seen before" memory, so two different feeds don't share it.

A full record looks like this (description fields shortened):

{
  "job_id": "4466544261",
  "title": "Data Engineer I",
  "company": "Docebo",
  "company_url": "https://www.linkedin.com/company/docebo",
  "location": "Toronto, Ontario, Canada",
  "countries": ["CA"],
  "posted_date": "2026-09-15",
  "posted_ago": "1 month ago",
  "job_url": "https://www.linkedin.com/jobs/view/4466544261/",
  "seniority": "Entry level",
  "employment_type": "Full-time",
  "industries": "Software Development",
  "applicants": "Over 200 applicants",
  "salary_text": "CA$68,600.00/yr - CA$91,500.00/yr",
  "description_text": "Artificial Intelligence. Actual Impact. ...",
  "description_html": "<strong>Artificial Intelligence. Actual Impact.<br/><br/></strong>...",
  "search_country": "CA",
  "source_url": "https://www.linkedin.com/jobs/view/4466544261/",
  "scraped_at": "2026-11-04T11:17:58Z"
}

job_id is the stable key we need.

Step 2: Prepare the Google Sheet

Create a Google Cloud project, enable the Google Sheets API, and create a service account. Download its JSON key.
Create a sheet and share it with the service account's email address (it looks like name@project.iam.gserviceaccount.com) with Editor access.
Copy the sheet ID from its URL: https://docs.google.com/spreadsheets/d/<SHEET_ID>/edit.

Install the two libraries:

pip install apify-client gspread

Step 3: The script

import os

import gspread
from apify_client import ApifyClient

APIFY_TOKEN = os.environ["APIFY_TOKEN"]
SHEET_ID = os.environ["SHEET_ID"]
GOOGLE_CREDS = os.environ.get("GOOGLE_CREDS", "service_account.json")

COLUMNS = [
    "job_id",
    "posted_date",
    "title",
    "company",
    "location",
    "seniority",
    "employment_type",
    "salary_text",
    "applicants",
    "job_url",
    "scraped_at",
]

RUN_INPUT = {
    "keywords": ["data engineer", "product manager"],
    "country": "CA",
    "datePosted": "pastWeek",
    "jobTypes": ["fullTime"],
    "maxResultsPerSearch": 100,
    "incremental": True,
    "stateKey": "sheet-ca-data-pm",
}


def fetch_new_jobs():
    client = ApifyClient(APIFY_TOKEN)
    run = client.actor("aurelix/linkedin-jobs").call(run_input=RUN_INPUT)
    return list(client.dataset(run["defaultDatasetId"]).iterate_items())


def to_row(job):
    return ["" if job.get(col) is None else str(job.get(col)) for col in COLUMNS]


def main():
    jobs = fetch_new_jobs()

    gc = gspread.service_account(filename=GOOGLE_CREDS)
    ws = gc.open_by_key(SHEET_ID).sheet1

    if not ws.row_values(1):
        ws.append_row(COLUMNS)

    known_ids = set(ws.col_values(1)[1:])
    rows = []
    for job in jobs:
        job_id = str(job.get("job_id"))
        if job_id in known_ids:
            continue
        known_ids.add(job_id)
        rows.append(to_row(job))

    rows.sort(key=lambda r: r[1], reverse=True)
    if rows:
        ws.append_rows(rows, value_input_option="RAW")

    print(f"{len(jobs)} jobs returned, {len(rows)} appended")


if __name__ == "__main__":
    main()

A few choices worth explaining:

datePosted: "pastWeek" plus incremental: True. The first run fills the sheet with the last week. After that, the incremental memory means each run only returns postings that weren't returned before, so a missed day (server down, laptop asleep) doesn't leave a gap, which a strict past24h window would.
A second dedupe against the sheet. If someone changes the stateKey, or you run the script from two places, the known_ids check still stops duplicate rows. Reading one column is cheap.
value_input_option="RAW". Values are stored as text, exactly as returned. Without it, Sheets may turn job IDs into numbers in scientific notation, and then your dedupe check stops matching.
No description column. Full descriptions run to thousands of characters and make a sheet hard to scan. Keep the job_url; if you want the text, add description_text to COLUMNS (a single Sheets cell holds up to 50,000 characters).

Step 4: Schedule it

On any Linux box, a cron entry is enough:

# every day at 07:00
0 7 * * * cd /opt/jobs-sheet && APIFY_TOKEN=... SHEET_ID=... /usr/bin/python3 export_jobs.py >> export.log 2>&1

If you don't want to keep a machine running, a scheduled GitHub Actions workflow works the same way: store APIFY_TOKEN, SHEET_ID and the service-account JSON as repository secrets and run the script on a schedule: trigger.

There is also a no-code route: schedule the Actor itself in Apify Console with incremental on, and connect its Google Sheets integration. That's fine for a single feed; the script above gives you control over columns, sorting and the extra dedupe.

Variations

Several countries. country takes one value per run. Call the Actor once per country with a different stateKey each time, and write each country to its own worksheet (gc.open_by_key(SHEET_ID).worksheet("US")).
Paste a search you already built. If you have a LinkedIn job search URL with filters set in the browser, put it in searchUrls instead of keywords. Its own filters are used as they are.
A cheaper "links only" feed. Set includeDetails to false to skip opening each job page. You get title, company, location, date and link only, and keyword matching is then checked against the title only.

Limits

Being honest about what this setup will and won't do:

LinkedIn shows at most about 1,000 jobs per search. For broad searches, split by keyword or location.
The Actor uses Apify's datacenter proxy only, and LinkedIn rate-limits datacenter addresses. On HTTP 429 it backs off and switches proxy sessions, but a search can still stop early; the jobs already found are kept and the run's status message names the search that stopped. For a daily feed of a few hundred jobs this matters less than for one big backfill.
posted_date is a date only. LinkedIn's public page shows no time.
salary_text is filled only where LinkedIn shows a salary on the public page, which is often not the case.
Workplace type (remote, hybrid, on-site) is sent to LinkedIn as a filter but can't be re-checked, because the public job page doesn't show it.
Keyword matching uses whole words, simple plurals and a few synonyms (for example RN for nurse). Very loose titles can be skipped.
Recruiter and poster names are never collected, and personal emails are removed from descriptions. Use the data for job search and hiring research, not for bulk outreach.

Why this tool over the alternatives

Compared with the leading alternatives on the Apify Store (the full table is on each Store page):

25% under the most-used LinkedIn jobs scraper on the free plan ($1.50 vs $2.00 per 1,000).
LinkedIn's exact country id with other-country results dropped, job type and level filters re-checked on each posting, and filtered-out jobs not charged. One rival is cheaper per job ($0.40).

Cost

At the time of writing, the README lists pay-per-result pricing: $1.50 per 1,000 jobs on Apify's free plan and $1.05 per 1,000 jobs on paid plans (Bronze, Silver, Gold), plus a start fee of $0.00005 per run. Jobs your filters drop and duplicates are not charged.

For the example above, the worst case is two keywords times 100 jobs = 200 jobs per run, or about $0.30 on the free plan. With incremental mode on, later runs usually return far fewer than the cap. Apify's free plan includes $5 of credit a month, which the README puts at about 3,300 jobs.