DEV Community: Ben

Merge YouTube exports without losing where each value came from

Ben — Fri, 11 Sep 2026 04:17:24 +0000

One video appeared in two of my September 10 exports. YouTube search returned 7,493,319 views and a duration of 7,341 seconds for Programming with Mosh's Python Full Course for Beginners. A later metadata lookup returned the same title, but both numbers were null.

An ordinary dictionary update would have erased the useful values. Replacing those nulls with zero would have invented a collapse in views.

I maintain the six Apify Actors used below. This guide combines their existing successful cloud exports into a small local index, retaining the source of every observation. You can reproduce the example offline with Python 3.11 or later; it needs no account, token or installed package.

The actual overlap

Both observations refer to video K5KVEU3aaeQ. These are dated measurements, not current view counts.

Observation on September 10, 2026, UTC	View count	Duration, seconds	Source detail
Search, 17:53:13	7,493,319	7,341	Query: `python tutorial`
Metadata, 18:33:17	`null`	`null`	`metadata_source: oembed`

The metadata Actor's cloud response supplied basic oEmbed information. Detailed metadata had worked locally, but that did not establish cloud access to those fields. Keep the source marker in any downstream export that uses this Actor.

The index groups observations by YouTube video ID. It deliberately leaves both records intact, so a report can choose the value appropriate to its purpose without rewriting the evidence.

Choose the export for the question

These six existing runs produced 15 rows across 14 distinct videos. I read their datasets back and used selected metadata columns for the downloadable sample; it contains no caption text or full descriptions.

Source and tested example	Rows	Useful distinction
YouTube Search: Python tutorial queries	4	Retain the query and `search_rank`. The cap applies per query, with duplicate IDs and exclusions reducing output.
Playlist Extractor: Corey Schafer's Python playlist	3	Keep `playlist_id` and `playlist_index`; playlist position is different from search rank.
Channel RSS: recent Linus Tech Tips uploads	3	Use the source publication date for a recent-upload feed. RSS does not provide a complete channel archive.
Shorts Scraper: Apify's Shorts tab	3	This export calls its identifier `short_id`. The index maps that identifier to the same video-ID namespace while preserving the original fields.
Video Metadata: one known Mosh video	1	The cloud response used oEmbed and left detailed metrics null. It cannot fill every gap in discovery output.
Transcript Scraper: one known video with English captions	1	Caption availability, language and generation status describe a track. Translated-track delivery remains unverified.

The search sample ran on 1.0.3 and the caption sample on 1.0.12; their later 1.0.4 and 1.0.13 releases changed schema declarations without changing those runtimes. The other sample builds are Playlist/Shorts 1.0.4, Channel RSS 1.0.9 and Video Metadata 1.0.6. These independent examples illustrate the record contracts; they are not a single chained discovery-to-caption run.

The six recorded runs used about $0.00467 of platform resources, excluding builds. That is an owner-test resource measurement, not the amount a customer would pay. Each Actor's Pricing tab governs customer fees. No new cloud run is needed to reproduce the local example.

Run the local index

Download these three files from the existing workflow Gist:

youtube_observation_index.py
test_youtube_observation_index.py
youtube-observations-sample.json

Keep them in one directory, then run:

python test_youtube_observation_index.py
python youtube_observation_index.py youtube-observations-sample.json > first-index.json
python youtube_observation_index.py youtube-observations-sample.json youtube-observations-sample.json > repeat-index.json

The check exercises the real search/oEmbed overlap, missing values, Shorts IDs and repeat imports. It also checks deliberately invalid inputs and a synthetic successful empty run; those checks are separate from the cloud examples.

Confirm the repeat output and inspect the overlap:

import json
from pathlib import Path

assert Path("first-index.json").read_bytes() == Path("repeat-index.json").read_bytes()
index = json.loads(Path("first-index.json").read_text())
assert len(index["runs"]) == 6
assert len(index["videos"]) == 14
assert sum(len(v["observations"]) for v in index["videos"]) == 15

video = next(v for v in index["videos"] if v["video_id"] == "K5KVEU3aaeQ")
for observation in video["observations"]:
    print(observation["source"], observation["fields"].get("view_count"))

The output includes 7493319 for search and None for metadata. It does not choose a single authoritative view count for you.

Import your own successful exports

Download a run's dataset as JSON. Wrap it in a list containing an object with these fields: source (one of the six Actor names above), run_id, observed_at, status and items. Set items to the downloaded array. Use the run's actual ID and its timezone-bearing finish time; observed_at is the collection checkpoint, not the video's publication date.

Only SUCCEEDED exports enter this index. Inspect failures and partial datasets separately before deciding whether they are usable. A successful export with zero rows remains in the runs list, because silence from a bounded search does not prove that videos were removed.

Supply all the export files you want to combine as arguments. The script builds a fresh index from those files; it does not read an existing index, start Actors, send alerts or maintain a scheduled monitor. Keep the source files as your archive and write each comparison to a new output filename. Shell redirection can truncate an existing file before Python validates the inputs.

The same source/run ID with identical contents is harmless to import twice. Conflicting contents under that identity cause an error, which catches accidentally relabelled or incomplete downloads. Rows within one export remain separate observations, including repeated appearances of a video. Their export_row is simply the one-based row number in that downloaded file.

Deciding which value to display

For a research reading list, a title and canonical video link may be enough. A trend chart needs compatible measurements over time. Mixing an old search view count with a new metadata timestamp would make a number look fresher than it is, even if you correctly ignored the null.

Keep that decision in the report: select an observation whose field is present, show its collection time and identify its source. Playlist position belongs to a particular playlist; search rank belongs to a particular query. Neither is a permanent property of the video.

The downloadable script retains all fields you supply, so choose what you store. For captions, this sample keeps availability and language metadata without redistributing the text. Use content you have permission to process, and respect source restrictions. The in-memory index suits small research exports; a larger archive needs durable storage and an explicit retention policy.

Track Airbnb stay prices without mistaking missing results for bookings

Ben — Thu, 10 Sep 2026 22:47:44 +0000

A room can disappear from a short Airbnb search export even when nothing about its availability has changed. Search order changes, and a three-result limit only captures three observations. Treating every missing room as booked would turn that sampling limit into a false alert.

This Python and SQLite workflow keeps price observations for one fixed stay. It compares a room only with its earlier observation for the same destination, dates, guest counts and currency. Missing rooms keep their history. A missing numeric price stays null.

I maintain the Airbnb Search Scraper used here. The code, offline check, input and dated sample are available together. The script needs Python 3.11 or newer and uses the standard library.

What the two real captures showed

On September 10, 2026, I captured three public search results for Berlin, September 18–20, two adults, EUR, then repeated that exact search about five minutes later. Both runs used the existing Actor build 1.0.6. Each returned three records and stopped at the result limit.

Room ID	First observed total	Second observed total	Interpretation
`721527807549293451`	€239	€239	Same observed price
`19976281`	€257	€257	Same observed price
`26180749`	€240	Not in this export	Keep the earlier observation
`23848408`	Not in the first export	€286	First observation

The database contains six observations across four room IDs. No price change was observed in this small live example. The offline check separately exercises a price decrease, missing prices, invalid stay data and failed imports; those checks are not additional market observations. The download airbnb-stay-sample.jsonl contains the six dated observations and their comparison fields.

The source labels these prices “for 2 nights.” They are observed search totals, not nightly prices or confirmed checkout quotes. Fees, availability and the final payable amount can change. This sample is dated evidence, not a current booking recommendation.

Keep the stay fixed

Download airbnb_stay_watch.py, test_airbnb_stay_watch.py and airbnb-stay-input.json from the Gist into the same directory. The input used above is:

{
  "location": "Berlin, Germany",
  "checkIn": "2026-09-18",
  "checkOut": "2026-09-20",
  "adults": 2,
  "children": 0,
  "infants": 0,
  "pets": 0,
  "currency": "EUR",
  "maxResults": 3,
  "maxPages": 1
}

Choose future dates before your first capture. Use a new output folder when changing any input, including the page or result limit. Rolling dates such as “seven days from now” make successive prices describe different stays, so this workflow requires explicit dates.

Run the check before using your token:

python test_airbnb_stay_watch.py

It runs offline and starts no Actor. Set APIFY_TOKEN through your environment or secret manager; keep it out of the input file and downloaded code. Then capture your chosen stay:

python airbnb_stay_watch.py capture berlin-stay --input airbnb-stay-input.json

Keep the input file outside berlin-stay. The folder contains stays.sqlite3, stay-history.jsonl and completed run receipts. Back it up if the history matters. Run commands sequentially against one folder.

Later, repeat the same capture command. That starts a new paid Actor run for the same search and adds observations for its returned rooms. The second real capture printed:

{
  "new_observations": 3,
  "comparisons": {"same_price": 2, "first_observation": 1},
  "source_stop_reason": "result_limit",
  "stored_observations": 6,
  "stored_runs": 2
}

Read the history without inventing changes

Each JSONL line contains the source listing, run ID, comparison status, previous total and price delta. The statuses are:

Status	Meaning
`first_observation`	This room has no previous observation in this folder
`same_price`	Both numeric totals are present and equal
`price_changed`	Both totals are present and differ; `price_delta` is current minus previous
`unpriced`	At least one of the two observations lacks a numeric total; no numeric delta is inferred

The importer checks the room ID and URL, destination, dates, all guest counts, currency and source price labels before writing. It uses decimal arithmetic for price differences and rejects duplicate room IDs within one run. An invalid row rejects the entire import, preserving the earlier database history.

A missing room creates no replacement record. Its earlier price remains a historical observation, not a claim that it is still available. Even an unchanged numeric price says nothing about cancellation terms, minimum-stay rules or checkout availability.

Regenerate the JSONL file from SQLite without making an API request:

python airbnb_stay_watch.py export berlin-stay

Recover a completed run before starting another

In the first live capture, the finished run's dataset export was not immediately consistent with its summary. The importer rejected the incomplete read and kept the pending run receipt. Once the export was available, I resumed that same run, without starting another.

The script now retries dataset reads up to three times, two seconds apart. It never automatically repeats a start request. If the export still is not ready, use the run ID printed by the script:

python airbnb_stay_watch.py capture berlin-stay --input airbnb-stay-input.json --run-id YOUR_EXISTING_RUN_ID

This path only reads the existing run. It validates the saved input before importing anything. Reimporting an already imported run adds zero observations; the real second run was resumed this way and its JSONL export remained byte-identical.

If a start request times out before a run ID is received, pending.json prevents another automatic start. Inspect your Apify runs for that request's time and exact input, then resume the matching run ID. Do not delete the pending receipt and blindly repeat the start. Failed runs leave the history intact and need inspection before a new capture.

Cost and scope

At the September 10 Free-tier price, three exported records cost $0.015, plus $0.00005 for a 512 MB start: $0.01505 per capture. The two demonstrated captures have a calculated Free-tier Actor-price equivalent of $0.03010. Repeat observations are still billable results; SQLite prevents duplicate imports, not charges for fresh source runs. Check the Actor's Pricing tab for your current tier.

Every start made by this example sets a $0.06 maximum Actor charge, 512 MB and a 300-second timeout. The wrapper accepts at most ten results and two source pages. A spending or result limit may leave the search incomplete. The cap applies to Actor charges; separate platform or local storage costs are outside it. These demonstrations were owner tests, not customer sales.

This workflow does not book rooms, send notifications or create a recurring schedule. For long-term rental change alerts, the Zumper workflow covers a different source and monitoring contract. Use the fixed-stay workflow here when dates and occupancy are part of the price you need to compare.

Join LinkedIn jobs to their company pages with Python

Ben — Thu, 10 Sep 2026 17:07:55 +0000

LinkedIn job records include an employer URL. You can use that URL to attach a company's public industry, website and size range to a job feed, keeping the source identity through both steps. This is useful when the employer's displayed name is abbreviated, shared by several organizations or belongs to a recruiting agency.

I maintain the LinkedIn Jobs and LinkedIn Company Actors used here. This example joins their exports with Python and SQLite. It saves the jobs before attempting company lookups, so a company failure leaves a recoverable job export.

What the live sample produced

On September 10, 2026, a bounded search for Python jobs in Berlin returned three postings. Their employer URLs led to three verified company records:

Posted employer	Company ID	Public company-size range	LinkedIn-associated members
Flix	2726149	1,001–5,000 employees	2,631
Nelly Solutions	74522753	51–200 employees	206
Berliner Verlag	1178870	201–500 employees	100

These are dated source observations. Nelly's size range and associated-member count disagree because they are different LinkedIn fields; the workflow keeps both. Neither establishes an independently verified workforce size. The employer on a job advertisement can also be an agency, so this join doesn't identify an undisclosed end client.

The first capture stored three jobs, including their complete public descriptions. Enrichment attached three company profiles. A fresh job run returned the same IDs; the database added zero jobs and the enrichment step made zero API calls. The resulting JSONL file matched byte for byte.

The Jobs run IDs were vM96cCZQDC8v24ltV and 4AyEwm8YDWjMvrQ2v; the Company run was uNaVz3GdaNyn4gXUP. All three used build 1.0.4 of their respective Actor and ended successfully. This sample covers three returned postings, not every Python vacancy in Berlin.

Run the two steps

Download linkedin_hiring_feed.py, test_linkedin_hiring_feed.py and linkedin-hiring-input.json from the public workflow files. Keep the Python files together. They require Python 3.11 or newer and use its standard library.

python3.11 -m venv venv
source venv/bin/activate
python test_linkedin_hiring_feed.py

The checks run without credentials or network access. They exercise repeated imports, exact employer matching, invalid rows, a failed company run and recovery from the saved jobs. Set APIFY_TOKEN through your environment or secret manager before the live commands; keep it out of the input JSON and downloaded scripts.

The supplied search is small:

{
  "keywords": "python",
  "location": "Berlin, Germany",
  "maxResults": 3,
  "maxPages": 2,
  "timeRange": "week",
  "sortBy": "date",
  "strictKeywords": true,
  "onlyNew": false,
  "proxyConfiguration": {"useApifyProxy": false}
}

Capture the jobs, then enrich the pending employer URLs:

python linkedin_hiring_feed.py capture linkedin-hiring-input.json berlin-hiring
python linkedin_hiring_feed.py enrich berlin-hiring

Each command that starts an Actor fixes memory at 512 MB, timeout at 300 seconds and the maximum Actor charge at $0.05. The two starts have separate caps. The example accepts at most ten jobs and five search pages per capture; each enrichment command requests at most ten pending companies. Inspect the printed counts before increasing your search within those bounds.

The folder contains hiring.sqlite3, the persistent database; hiring-feed.jsonl, the joined export; and runs/, the run receipts and successfully downloaded source rows. Keep the input file outside this folder. A changed input requires a different folder, which prevents accidentally mixing two search definitions.

Run the same two commands sequentially on later days. capture stores each job ID once. enrich requests only eligible employer URLs without a saved company snapshot, so several jobs from the same employer share one lookup. The second command exits without contacting Apify when nothing is pending.

Read the joined record

Every JSONL line has an outer job_id, the original job, a company_profile and an enrichment_status. These selected fields come from the saved Nelly record; the full file also retains the job description and the remaining company fields:

{
  "job_id": "4434332752",
  "job": {
    "title": "Senior Software Engineer, Python",
    "company": "Nelly Solutions",
    "company_url": "https://de.linkedin.com/company/nelly-solutions"
  },
  "company_profile": {
    "company_id": "74522753",
    "url": "https://www.linkedin.com/company/nelly-solutions",
    "industry": "Software Development",
    "company_size": "51-200 employees",
    "linkedin_employee_count": 206,
    "identity_verified": true
  },
  "enrichment_status": "verified"
}

The join normalizes country-specific LinkedIn hostnames, case, trailing slashes and tracking parameters. It then requires the Company Actor's verified requested URL to equal that normalized employer URL. Company names never serve as a fallback key. Related organizations or a profile returned for a different URL fail validation.

Missing or unsupported employer links keep their jobs in the export with missing_company_url or unsupported_company_url. A supported URL awaiting a successful lookup has pending. Only a matched, verified company row receives verified; the other states have a null company_profile. The Company Actor currently accepts named company slugs containing letters, digits, underscores and hyphens. Numeric redirect URLs and other forms need separate source review; this example doesn't rewrite them into guessed company pages.

Use the outer job_id as the destination key when importing into a sheet or database. Retain job.scraped_at and company_profile.scraped_at so readers can see when each observation was collected. The saved profile also has the source-published website, when available.

Recover without collecting the jobs again

The first live capture stopped because a separate dataset-statistics read disagreed with the item response. A follow-up read confirmed all three jobs. The released script validates the total from the same paginated response as the items, removing that separate statistics dependency. I resumed the existing successful run:

python linkedin_hiring_feed.py capture linkedin-hiring-input.json berlin-hiring --run-id vM96cCZQDC8v24ltV

For your own recovery, substitute your own run ID. The script checks the Actor identity, successful terminal status, input and complete bounded export before importing it. A failed run cannot pass as a successful empty feed. If a start request times out before returning an ID, inspect your Apify Runs page before starting another; the server may already have accepted it. The script never retries a start automatically.

When a company batch fails, the captured jobs remain saved and their supported employer URLs remain pending. Inspect that run and the source issue before retrying enrich; no new Jobs run is needed. A successful Company run whose input still matches the pending batch can also be imported with enrich berlin-hiring --run-id YOUR_COMPANY_RUN_ID. Failed batches are not partially imported. A successful spending-limited batch can leave companies pending, and the printed count makes that visible.

SQLite rolls back an import if a later row has a duplicate or mismatched identity. The JSONL file replaces its predecessor only after writing finishes. If export is interrupted, regenerate it locally:

python linkedin_hiring_feed.py export berlin-hiring

That command makes no network requests. Avoid overlapping commands on one folder, and retain the database as the source of truth.

Costs and observation limits

At the verified Free-tier rates, Jobs costs $0.002 per exported posting and Company costs $0.004 per exported company. Each run at this memory setting adds a $0.00005 start event.

Step	Calculated Actor charge
Initial three-job capture	$0.00605
Three company profiles	$0.01205
Fresh three-job repeat	$0.00605
Repeat enrichment with no pending companies	$0.00000
Total for the demonstrated sequence	$0.02415

These are customer-price equivalents calculated from the recorded events, not revenue from my owner tests. Apify's current pricing panel is authoritative, and plan discounts may apply. Repeated job captures still export and charge for the snapshot; local ID deduplication prevents duplicate destination records. An existing-run import makes API reads but starts no new Actor run.

This example keeps the first observation of each job and the first successful company snapshot per URL. Later captures won't update their descriptions or company metrics. It doesn't detect job closure, refresh company headcount, send alerts or activate a schedule. The dates inside the saved records remain meaningful precisely because they're preserved. For a fresh research snapshot, use a new folder; that starts a separate history and incurs fresh lookups.

The source search and text filtering bound which jobs enter this feed. An absent result does not establish that a vacancy closed, and a public profile may become unavailable. Exports load the local feed into memory; use batched or streaming exports if the accumulated archive becomes large. The workflow contains no personal-profile collection or employee enumeration. Follow source terms and your organization's data-use requirements.

You can inspect the Berlin Jobs example and the separate verified Company example before running the workflow in your own account. If you need an account, this Apify signup link is a referral link; I may earn a commission at no extra cost to you.

Refresh website Markdown without losing unchanged pages

Ben — Thu, 10 Sep 2026 16:33:51 +0000

A documentation refresh can return a page with markdown: null because its content hasn't changed. If an importer treats that row as a replacement document, it erases the useful text it collected earlier. The fix is to keep the previous content and compare hashes before updating it.

I maintain Website Markdown Crawler on Apify. This walkthrough turns its exports into a local documentation corpus using Python's standard library and SQLite. It includes a runner that supplies the previous hashes automatically, so you can repeat a crawl without copying state between files.

What two real refreshes returned

On September 10, 2026, I ran the same five-page Python tutorial crawl twice on build 1.0.2. The first run imported the content; the second supplied the hashes from that local database.

Observation	First refresh	Repeat refresh
Exported page records	5	5
Markdown body bytes, UTF-8	90,019	0
Content versions written locally	5	0
Complete pages retained locally	5	5
Calculated Free-tier Actor charge	$0.00505	$0.00505

The local JSONL corpus and hash file matched byte for byte after the repeat. Zero Markdown bytes describes those five response fields; the repeat still transfers metadata and still fetches the source pages. Both runs charge for five exported records. The benefit is preserving the corpus and letting a downstream index skip unchanged content, not eliminating the crawl fee.

The run IDs are Mr1xmbQyMpwugttds and 6O33lSVE0q6ve33uS. Both stopped at the requested page limit with 12 discovered URLs pending. Five successful pages do not mean the entire Python tutorial was collected.

Where the nulls come from

The crawler hashes the extracted Markdown. On the next run, the runner builds previousHashes from SQLite, keyed by the final source URL. A matching hash produces a record like this excerpt from the repeat:

{
  "url": "https://docs.python.org/3/tutorial/",
  "change_status": "unchanged",
  "content_hash": "b216133cb14e3dfcc3d354ef3a83bd77b1d1100d955064e1477e2eac4be3928e",
  "markdown": null,
  "text": null
}

The importer checks that it already holds content with that exact URL and hash. If it does, it keeps the stored document. If it doesn't, it rejects the import; a hash alone cannot reconstruct a page. New and changed rows must contain text and Markdown whose SHA-256 matches the supplied hash.

Each stored document keeps url, title, markdown, text and content_hash from its last imported content version. SQLite's URL primary key prevents duplicate documents. A malformed row rolls back that import, including valid rows encountered earlier in the same array.

Try the five-page workflow

Download website_markdown_archive.py, website_markdown_refresh.py and website-markdown-input.json from the public workflow files. Keep the two Python files together. They need Python 3.11 or newer and no additional packages.

The supplied input is also available as a public Apify example Task:

{
  "startUrls": ["https://docs.python.org/3/tutorial/"],
  "maxPages": 5,
  "maxDepth": 1
}

Create a local environment and run the checks, which need no credentials or network access:

python3.11 -m venv venv
source venv/bin/activate
python website_markdown_archive.py --self-test
python website_markdown_refresh.py --self-test

Set APIFY_TOKEN through your shell environment or secret manager, then run:

python website_markdown_refresh.py website-markdown-input.json docs-archive

The runner starts one Actor run at 512 MB with a 300-second timeout and a $0.02 maximum Actor charge. It prints the run ID, waits for a successful terminal status, checks the coverage/export counts, and imports the result. It doesn't retry the start request automatically.

The archive folder contains docs.sqlite3, the persistent source of truth; corpus.jsonl, one complete stored document per line; and hashes.json, the current URL-to-hash map. Keep this folder between refreshes. The runner reads its baseline directly from SQLite, so the exported hash file isn't a separate state you need to maintain.

Repeat the same command after the first run finishes. These are the result fields from my two runs, with file paths and run IDs omitted:

{"received": 5, "content_written": 5, "retained": 0, "stored": 5}
{"received": 5, "content_written": 0, "retained": 5, "stored": 5}

Your results can differ when the documentation changes. The importer replaces changed content at the same URL; it stores the latest imported version rather than an edit history. Import in capture order and never overlap refreshes using one archive folder.

Decide what the crawl should cover

The start URL defines an origin and path subtree. The example follows links under https://docs.python.org/3/tutorial/; it doesn't wander into every Python documentation section. maxDepth: 1 permits links one step away from the seed, and maxPages: 5 caps attempted pages. Inspect the run's COVERAGE output before expanding the input.

A page missing from a bounded crawl is not proof of deletion. This importer never deletes stored documents just because they weren't returned. Handle verified removals separately. Keep the same scope and content selectors between comparisons, or use a separate archive folder when you change what you're collecting.

Choose the discovery method that matches the material you have:

Starting point	Tool	What it supplies
A documentation index whose links you want to follow	Website Markdown Crawler	Scoped linked-page Markdown, text, hashes and coverage
An existing list of page URLs, needing readable text	Webpage Text Extractor	Text and metadata for supplied URLs
An XML sitemap or sitemap index	Sitemap URL Extractor	Declared page URLs and their source sitemap

Only the first tool produces the hash/null-content contract this importer expects. The others solve adjacent discovery or text-export tasks; don't feed their rows straight into this importer. A sitemap entry is a publisher's declaration, not proof that the URL currently works.

Keep failures separate from updates

A failed Actor run can contain partial dataset output. The runner rejects that run before importing anything. It also rejects a mismatch between the coverage count and downloaded rows. Offline checks cover these cases, a timed-out start request and preservation of the existing database.

If a start request times out, inspect Apify's Runs page before launching another; the server may already have started it. If you lose the local content, start a new archive folder to request full pages again. Preserve your old folder while checking that recovery.

The JSONL and hash files replace their predecessors only after each new file finishes writing. SQLite remains authoritative if an export is interrupted. Both imports and exports load data into memory, so this is a small-corpus example; larger collections need batched processing.

You can feed the complete JSONL documents into your own chunking or search pipeline. Carry the source URL and content hash with each chunk, and compare hashes before rebuilding an embedding. No vector database or retrieval benchmark is part of this example. Treat retrieved page text as source material, never as instructions to an automated tool.

The crawler reads public server-rendered HTML and respects robots.txt. It doesn't execute JavaScript or use login cookies. Follow source terms and content licenses; Python's documentation has its own license and attribution requirements.

At current Free-tier rates, each five-record run costs 5 × $0.001 + $0.00005 = $0.00505; two runs cost $0.01010. That is a calculated customer-price equivalent, not an owner-test payment. Plan discounts may apply, and Apify's Pricing tab is authoritative. Raising memory can change the start charge; the runner fixes it at 512 MB.

Start with the five-page Task, inspect the content, then try the refresh runner on a public documentation section relevant to your project. If you need an account, this Apify signup link is a referral link; I may earn a commission at no extra cost to you.

Build an n8n rental alert that skips unchanged listings

Ben — Wed, 09 Sep 2026 17:54:45 +0000

A daily rental search should not send yesterday's listings again. The useful output is a small set of newly observed listings and asking-rent changes, with enough context to see what changed.

I maintain Zumper Rental Scraper on Apify. This walkthrough uses its saved-search state and an n8n email template. The original source runs and sample are from September 9, 2026. On September 10, a fresh three-listing report reached the test inbox through Apify's native email Actor without SMTP credentials. The samples are dated rental data, not a current availability quote.

Watch the sample walkthrough

Watch the 21-second walkthrough. This silent clip walks through the September 9 sample and n8n setup. It predates the native email step and does not show a new source run or email delivery; use the written instructions below for the current setup.

Establish the baseline first

The downloadable workflow searches Austin apartments with at least one bedroom and a maximum asking-rent filter of $3,000:

{
  "mode": "search",
  "location": "austin-tx",
  "propertyType": "apartments-for-rent",
  "maxPrice": 3000,
  "minBeds": 1,
  "maxListings": 25,
  "includePhotos": false,
  "useApifyProxy": true,
  "onlyChanges": true,
  "monitorId": "austin-rentals-under-3000",
  "firstRunMode": "seedOnly"
}

seedOnly stores the initial snapshot without exporting the backlog. Later runs with the same monitor ID return newly observed or changed listings. Use emitAll if you want the first snapshot in the dataset too.

Keep one monitor ID for one search and avoid overlapping runs. Changing cities or filters deserves a new ID. Leaving resetState enabled would repeatedly discard the baseline and defeat the monitor.

What the live check returned

For the bounded demonstration, I used maxListings: 3 on build 1.1.23. The initial emitAll snapshot contained:

Listing	Minimum asking rent	Source location
Seabrook Square	$1,206/month	3515 Manor Rd, Austin
The Rail	$1,300/month	2921 E 17th St, Austin
The Rhone	$1,749/month	2450 Wickersham Ln, Austin

These can be building-level rent ranges, not individual units available at the minimum price. The Actor does not establish complete Austin market coverage or verify lease terms.

Run QDtz5Vvr2bRKoFbQs exported those three new rows. The next run, AdjAk7J4AZNMiTJrB, returned zero duplicates. A separate seed-only monitor returned zero rows on both its initial run and repeat. The earlier Austin case study provides additional background; the downloadable sample records this newer test.

There was also a failure worth keeping in the record: the first source attempt returned HTTP 403 through a residential route. That run failed before updating the baseline. The next attempt succeeded. A failed source request is not evidence that no rentals changed; check the run status before interpreting an empty dataset.

Wire the n8n workflow

Import zumper-rental-alerts.json from the Gist. Its five nodes run a daily schedule, call the Actor, build a text digest, send through Apify and check the mail run's status. The exported workflow is inactive and contains no credentials.

Create an HTTP Header Auth credential with header name Authorization and value Bearer YOUR_APIFY_TOKEN, then select it in both HTTP nodes. In Email rental alert, replace you@example.com in the JSON body expression with your Apify account email. No SMTP credential or custom sender address is needed. Check your n8n timezone before enabling the 08:00 schedule.

Apify Send Email allows Free accounts to send reports to their registered address; paying accounts can send to up to 20 different recipients. It is a reporting tool, not a promotional-mail service. The mail run uses 256 MB and a 60-second timeout, and its platform usage is separate from the rental Actor's charges.

The request uses 512 MB, a 240-second Actor timeout and a $0.75 maximum run charge. The formatter accepts both an array response and one n8n item per row. It returns no items when the dataset is empty, so the email node receives no digest on an unchanged run. Invalid rows raise an error instead of producing a misleading empty message.

I exercised the exact formatter code with the real three-row export and checked empty output, invalid rows and repeated inputs. That also caught literal \n sequences in the older template; the updated formatter produces actual line breaks. The September 10 delivery check used fresh output from build 1.1.24, passed that same formatter's text to Send Email 0.3.20, and verified one received Gmail message with an identical subject and body plus Apify's standard footer.

I also imported the workflow into n8n 2.38.6 and executed two manual checks. A saved copy of the fresh three-row export passed through every downstream node with Send Email in mock mode. A second execution called the live Zumper Actor with the same monitor ID, returned zero changes and stopped before the digest or email nodes. These were isolated test executions; no recurring schedule was activated. The inbox receipt was verified separately from those mock-mail checks.

The last node throws if the mail Actor fails or has not finished. Inspect the mail run before retrying a delivery: a request timeout can occur after a message was sent. Retry from the saved digest in that n8n execution; restarting the entire rental search can return zero rows because its baseline already advanced.

You can run the formatter checks without an Apify token:

git clone https://gist.github.com/e28968a940fe4195ffb5a997fe309503.git apify-workflows
cd apify-workflows
node test_zumper_rental_alerts.mjs

Separate rent cuts from other changes

The current Actor uses change_type: new, price_changed or details_changed. It does not emit a price_drop event. To keep only reductions in the minimum asking rent, filter change rows with:

row.change_type === 'price_changed' && row.price_change_amount < 0

The formatter's controlled test uses a simulated $2,000 → $1,800 change to check the “was” label. That is a formatting test, not an observed market price cut. A change in the maximum rent alone can have a zero minimum-rent delta. Missing listings are not labeled rented or withdrawn, because disappearance from a capped search window is not proof of either.

Know the event cost

On Free, the Actor charges $0.015 per emitted listing, plus $0.01 when that returned listing has a phone. The start event is $0.00005 at 512 MB. Plan discounts apply; the pricing was not changed for this workflow.

The three-row sample had two phones, so its Free-equivalent event cost is $0.06505. It was an owner test with no customer payment collected. A seed-only run or an unchanged repeat has no result or phone charge, but the start event remains. At the template's 25-row cap, even 25 rows with phones fit within its $0.75 charge limit at these prices.

Use the real sample to inspect the digest, then start with a small search for your own market. Keep the monitor ID stable, investigate failed runs, and preserve the source links when sending the changes onward.

Build a citable Reddit corpus with SQLite and repeat imports

Ben — Wed, 09 Sep 2026 16:16:26 +0000

A weekly Reddit archive refresh can return posts you've already collected. Appending every export creates duplicate documents; reading only selftext drops the comments. This example keeps one document per Reddit identity and preserves the source link you need for a citation.

I maintain Reddit Archive Scraper on Apify. The export below comes from a small cloud run on September 9, 2026. The importer uses Python's standard library and SQLite, so you can try the supplied sample locally before running an Actor.

Watch the sample walkthrough

Watch the 18-second walkthrough. This silent clip walks through the dated sample and repeat-import check below; it does not record a new cloud run. The written instructions contain the complete commands and output.

Start with a dated sample

Open the public Python archive example. It requests posts created on January 1, 2024, with a cap of three posts and two comments per post:

{
  "subreddits": ["Python"],
  "searchQuery": "",
  "afterDate": "2024-01-01",
  "beforeDate": "2024-01-02",
  "sortOrder": "oldest",
  "maxPosts": 3,
  "includeComments": true,
  "maxCommentsPerPost": 2
}

Build 1.0.20, run aQVygX8ydEuuDuwjM, returned three posts and six comments. One post was “Monday Daily Thread: Project ideas!”, ID 18vkgtu, created at 2024-01-01T00:00:08+00:00. Its citation is the original Reddit thread.

That proves the bounded request returned usable records. The three-post limit cannot establish complete coverage of the day. The Actor reads PullPush and Arctic Shift, whose coverage and ingestion lag can vary.

Keep the identity and the text together

Use post:18vkgtu as the document key. Prefix comment IDs with comment: because posts and comments have separate ID namespaces. The importer stores that key as SQLite's primary key:

CREATE TABLE IF NOT EXISTS documents (
    record_key TEXT PRIMARY KEY,
    document TEXT NOT NULL
);

INSERT INTO documents VALUES (?, ?)
ON CONFLICT(record_key) DO UPDATE
SET document = excluded.document
WHERE document != excluded.document;

An identical import leaves the existing document alone. If a later export contains changed text, the same key updates in place. The database holds the latest imported version; it doesn't retain edit history or compare source versions to prevent an older export overwriting a newer one. Import exports in capture order.

Each JSONL document contains record_key, type, id, post_id, parent_id, subreddit, created_iso, source_url and text. Post text combines the title and selftext; comment text keeps the full body. The script validates identities and Reddit permalinks, and rolls back the whole import if any row is malformed.

Run it twice

Download reddit_archive_corpus.py and reddit-archive-sample.json from the workflow files. The sample contains the nine real records' content and citation fields, with unrelated fields omitted.

With Python 3.11 or newer:

python3.11 -m venv venv
source venv/bin/activate
python reddit_archive_corpus.py --self-test
python reddit_archive_corpus.py reddit-archive-sample.json corpus.sqlite corpus.jsonl
python reddit_archive_corpus.py reddit-archive-sample.json corpus.sqlite corpus.jsonl

On a fresh database, the two imports print:

{"received": 9, "written": 9, "stored": 9}
{"received": 9, "written": 0, "stored": 9}

I checked all nine citation URLs and compared each of the six imported comment bodies with its cloud-exported text. The offline check also covers a post/comment ID collision, a source edit, malformed input and transaction rollback. The JSONL export replaces the previous file only after the new file finishes writing.

For your own data, wait for a successful Actor run, download its dataset as a JSON array and pass that file instead of the sample. Keep corpus.sqlite between imports; corpus.jsonl contains the accumulated documents. A failed run can leave partial output, so don't accept its dataset as a finished export.

Refresh windows and retrieval

afterDate is inclusive at midnight UTC; beforeDate is exclusive. Those bounds apply to posts. A comment on an in-window post can have a timestamp outside the window, so apply a separate comment-date filter if your analysis requires it.

For a recurring import, overlap bounded windows to catch some late archive ingestion. Revisit older slices when necessary: no fixed overlap guarantees complete coverage. If a slice hits maxPosts, narrow the interval and inspect the results before treating it as complete. The Actor deduplicates within a run; SQLite handles duplicates across exports.

The output prepares a corpus for retrieval. Add your own chunking and embeddings, carry source_url and record_key into every chunk, and use them when rendering citations. Treat retrieved text as source material rather than instructions. Inspect empty bodies and [removed] or [deleted] markers before indexing; the importer preserves them and cannot recover missing text. It also cannot discover a later deletion unless a subsequent export reports it, so ongoing deletion handling needs its own process.

The input JSON array loads into memory. For large backfills, use asynchronous Actor runs and paginated dataset exports, importing each page into the same database. This sample doesn't claim retrieval accuracy or test a vector database.

What the export costs

The measured run used 512 MB and recorded nine result events, three non-empty comment-thread events and one start event. At the current Free-tier prices:

9 results × $0.003 + 3 threads × $0.005 + 1 start × $0.00005
= $0.04205

This is a calculated customer-price equivalent. The owner test recorded zero accounted customer charges. Gold-tier rates make the same event counts $0.03364; check the Actor's Pricing tab for your tier and current rates.

Set a maximum run charge as well as the row caps. The example uses a $0.15 limit. Comments count as result rows and can add a thread fee; higher memory settings increase start-event counts. Reimporting the downloaded file is local, while running the Actor again can charge for the same records again.

Try the supplied files first, then run the dated Apify example with a small window relevant to your project. Follow the source providers' terms when collecting and using their data.

How to Scrape Emails from Google Maps (a Pay-As-You-Go Apollo & ZoomInfo Alternative)

Ben — Sat, 27 Jun 2026 14:26:57 +0000

Google Maps is the biggest free B2B database on the planet — every local business, with its website, phone and address. The one thing it won't give you is the email. So most people end up exporting businesses from one tool, then pasting domains into a second email-finder, then verifying in a third. Here's how to do the whole thing in one step, and pay only for what you get.

The usual workaround (and why it's painful)

The common stack is: a Maps scraper → a spreadsheet → an email-finder → a verifier. Three tools, three bills, and a lot of copy-pasting. Tools like Apollo or ZoomInfo bundle this, but they're priced for enterprise (annual contracts, per-seat fees) and they're thin on local SMEs, especially outside the US.

The one-step approach

The Google Maps Email Scraper does the search and the email enrichment in a single run. Give it a niche + a city, and for every business it:

Scrapes the listing — name, phone, address, website, rating, categories.
Visits that business's website — homepage, contact/Kontakt page and Impressum — and pulls the published email address.
If nothing is public, tests common business inboxes (info@, sales@, kontakt@…) and validates them with MX and SMTP checks.

You get one clean row per lead with a primary_email, a confidence score, and every email it found:

{
  "name": "Studio Berlin Marketing GmbH",
  "phone": "+49 30 1234567",
  "website": "https://www.studio-berlin.de",
  "primary_email": "info@studio-berlin.de",
  "email_confidence": 95,
  "email_source": "impressum",
  "rating": 4.8,
  "lead_score": 92
}

Why it's a strong Apollo / ZoomInfo alternative

Pay-as-you-go — a small fee per lead, plus a premium only when an email is actually found. No annual contract, no per-seat tax, no credit rationing.
You own the data — export to CSV/JSON or straight to your CRM via API, Make, Zapier or n8n.
Fresh and targeted — sourced live for your exact niche + city, not a stale shared database.
Especially strong in the DACH market — German sites are legally required to publish a contact email in their Impressum, so hit rates on German, Austrian and Swiss businesses are very high. Most US-centric tools are thin here.

Example: build a prospect list in one run

{
  "mode": "search",
  "query": "dentists",
  "location": "Munich, Germany",
  "maxResults": 100,
  "requireEmail": true
}

requireEmail: true means you only get leads that actually have an email — you're not charged the email fee for the ones without. Already have a list of companies? Switch to websites mode and paste your domains to enrich them directly.

Honest trade-offs

This finds business contact emails (the info@/sales@/Impressum kind), not a specific named person's inbox — for that, pair it with a name-based email finder. Published addresses are real and high-confidence; role-based guesses are validated by MX (and SMTP where the network allows), so filter by email_confidence if you only want the surest ones. And use the data for legitimate B2B outreach — honor GDPR/CAN-SPAM and opt-outs.

FAQ

Do I need an API key? No — a search query + location is enough.

Will it work outside Germany? Yes, anywhere Google Maps covers; the Impressum edge just makes DACH hit rates exceptionally high.

Can I schedule it? Yes — recurring runs and full API/Make/Zapier/n8n integration.

How much does it cost? Per result, with the email premium charged only when an email is found — far below an enterprise seat for the same verified-lead output.

Build verified B2B lead lists straight from Google Maps with the Google Maps Email Scraper.

ImmoScout24 Alternative: German Real-Estate Data Without the Blocks

Ben — Sat, 27 Jun 2026 13:31:22 +0000

ImmoScout24 is Germany's #1 property portal — and one of the hardest to scrape, with
aggressive blocking and pricey API access. The good news: most of the same German
real-estate data is available from sources that are far more accessible, including
private-landlord inventory ImmoScout under-represents. Here are three ImmoScout24
alternatives for actual data, by use case.

1. Private-landlord rentals & sales → Kleinanzeigen

Kleinanzeigen is where private owners list directly — often cheaper and absent from
the big portals. The
Kleinanzeigen Immobilien Scraper
parses rooms, living space, Kaltmiete/Warmmiete, deposit, address and PLZ into clean
fields, with city/radius search.

2. Agent-listed properties → Immowelt

For broader agent-listed inventory, the
Immowelt Scraper pulls
rental and sale listings with price, size, rooms, location and details — a strong
ImmoScout substitute for market analysis.

3. Shared flats & rentals (WGs) → WG-Gesucht

For the shared-apartment and rental market (huge in German cities), the
WG-Gesucht Scraper extracts
room/flat listings with price, location and availability — inventory you won't find on
ImmoScout at all.

{ "locationCode": "münchen", "radiusKm": 20, "maxResults": 200 }

Why these beat fighting ImmoScout24

Accessible — they don't wall you off the way ImmoScout does, so runs finish.
Different inventory — private-landlord and WG listings ImmoScout under-covers.
Pay-as-you-go — per-result pricing, no portal API contract.
Clean, typed fields — rooms, m², rent, deposit, address — ready for analysis.

Honest note

These are alternative sources of German property data, not ImmoScout24 scrapers. If
you specifically need ImmoScout's listings, you'll need ImmoScout. For market
analysis, private-landlord leads, rentals and WGs, this trio is cheaper, more
reliable, and covers inventory the big portal misses. (For Austria, add
willhaben.)

FAQ

Do I need an API key? No — give a city/PLZ and your filters.

Rent and buy? Yes — Kleinanzeigen and Immowelt cover both; WG-Gesucht covers shared/rentals.

Can I search by city? Yes — city names auto-resolve; radius search supported.

Is it legal? These read publicly available listing data. Use responsibly and within
each site's terms and applicable law.

German real-estate data without the ImmoScout wall: Kleinanzeigen Immobilien, Immowelt, WG-Gesucht.

Phantombuster Alternative: Pay-As-You-Go Social Media Data Extraction

Ben — Sat, 27 Jun 2026 13:27:32 +0000

Phantombuster is a handy social-automation suite — but it's subscription-based with
execution-time limits, and a lot of people only use it for one thing: pulling social
media data (posts, profiles, followers). If that's you, you can do the data side
pay-as-you-go, with no monthly seat and no time budget to ration.

What most people actually use it for

Extracting structured data from social platforms — a creator's posts, a profile's
stats, a follower list, hashtag/keyword streams. That's a scraping job, and you can
price it per result instead of renting hours.

The pay-as-you-go data stack

Pick the network:

Bluesky Scraper — posts, profiles, followers, search (no login needed for most).
Mastodon Scraper — hashtags, accounts, trends from any instance.
Instagram Intelligence Scraper — profiles, posts, comments, hashtags + lead scoring.
TikTok Profile Scraper — public profile stats at scale.
Reddit Scraper + Lemmy Scraper for discussion data.

Each returns clean JSON (text, engagement counts, author, media) ready for analysis or
an LLM:

{ "mode": "search", "searchTerms": ["#yourbrand"], "maxItems": 500 }

Why this beats a subscription

Per-result pricing — pay for the data you pull, no monthly seat or execution-time cap.
Own the data — export to CSV/JSON, no lock-in.
One consistent shape across networks — easy to pipe into a warehouse or LLM.
Emerging networks covered — Bluesky/Mastodon/Lemmy, where competition is thin.

Honest trade-offs

Phantombuster also does automation (auto-connect, auto-message — mostly LinkedIn
growth flows). These actors don't automate actions on your account; they extract
data. If you need account automation, that's a different (and ToS-riskier) job. For
clean social-data extraction and listening, pay-as-you-go scraping is cheaper and
simpler.

FAQ

Do I need logins? Mostly no (Bluesky/Mastodon/Lemmy public data); keyword search on
some networks works best with an optional app token.

Can I schedule it? Yes — recurring runs + API/Make/Zapier/n8n.

Which networks? Bluesky, Mastodon, Lemmy, Instagram, TikTok, Reddit today.

Is it legal? It reads publicly available data via public APIs/pages. Use it
responsibly and within each platform's terms.

Extract social data pay-as-you-go: Bluesky, Mastodon, Instagram, TikTok, Reddit.

ZoomInfo Alternative: Build B2B Contact Data Without the Enterprise Contract

Ben — Sat, 27 Jun 2026 13:23:52 +0000

ZoomInfo is powerful — and priced for enterprises, with annual contracts that start in
the five figures. If what you actually need is targeted company + contact data
(businesses in a vertical and region, with phone, website and email), you can source
it yourself from live public data for a tiny fraction of the cost, and own every row.

What you're really paying ZoomInfo for

A giant prebuilt contact database with intent signals and org charts, billed annually
per seat. That's overkill (and over-budget) if your job is building targeted
prospect lists for outbound — the most common use case by far.

The build-your-own stack

Find the companies. Pick your source by market:

Google Maps Business Scraper — any niche + city (name, address, phone, website, rating) with a lead score.
Gelbe Seiten Scraper (Germany) and local.ch Scraper (Switzerland) for DACH B2B directories.

Get the emails. Feed the companies' domains into the
Smart Email Finder & Verifier —
it tests common patterns, runs SMTP/MX checks, flags catch-all domains and returns a
confidence score, so your list is deliverable before you send.

{ "search": "logistics companies", "location": "Hamburg, Germany", "maxResults": 500 }

Why this beats an enterprise seat

Pay-as-you-go — per result, no annual contract or per-seat tax.
You own the data — export to CSV/CRM, no usage caps or credit rationing.
Fresh & targeted — sourced live for your exact ICP, not a stale shared DB.
DACH-strong — most US-centric tools are thin on German/Swiss SMEs; directory scrapers aren't.

Honest trade-offs

ZoomInfo's edge is scale, intent data and org hierarchy. If you need buying-intent
signals or a pre-enriched database of millions, that's a different product. For
building targeted, verified outbound lists on demand — especially in DACH — live
sourcing + verification is dramatically cheaper and just as actionable.

FAQ

Is this GDPR-compliant? You're collecting publicly listed business contact data;
use it for legitimate B2B outreach and honor opt-outs and local law.

Do I need an API key? No for the directory/Maps scrapers or the email finder.

Can I automate it? Yes — schedule runs and pipe results to your CRM via API, Make,
Zapier or n8n.

How accurate are the emails? SMTP/MX-verified with a confidence score; drop the
low-confidence ones before sending.

Build B2B lists on demand: Google Maps Business Scraper + Smart Email Finder & Verifier, plus Gelbe Seiten and local.ch for DACH.

mobile.de Alternative: Get Private-Seller German Used-Car Data

Ben — Sat, 27 Jun 2026 11:19:18 +0000

mobile.de is the default for German used-car data — but it's dealer-dominated,
heavily defended, and you're seeing the same inventory every other dealer and tool
sees. The biggest blind spot it leaves is private sellers, who list on
Kleinanzeigen (formerly eBay Kleinanzeigen). That's where the arbitrage, the cheaper
cars and the motivated sellers are — and it's far more accessible to pull.

Why private-seller data is the edge

Different inventory — private owners list cheaper and earlier than dealers; it's the half of the market mobile.de under-represents.
Arbitrage & sourcing — dealers and flippers buy private and resell; this is the sourcing side.
Cleaner economics — Kleinanzeigen is easier to access reliably than mobile.de's defenses.

The approach

The Kleinanzeigen Autos Scraper
parses the German car attributes most tools drop — Marke (make), Modell,
Erstzulassung (year), Kilometerstand (mileage), Kraftstoff (fuel), Getriebe
(gearbox), Leistung (power), price — into clean, typed fields, with city/PLZ +
radius search.

{ "locationCode": "berlin", "maxPrice": 20000, "minYear": 2016, "maxMileage": 120000 }

Output is one tidy row per car — make, model, year, mileage, fuel, transmission,
power, price, location, URL — ready for price analysis, arbitrage screening or a
search product.

Pair it for full DACH coverage

Used cars are one slice of the DACH market. The same engine powers the
Kleinanzeigen Immobilien Scraper
(real estate) and Kleinanzeigen Jobs Scraper
(local jobs), plus willhaben
for Austria — so you can cover the German-speaking market with one consistent data shape.

Honest note

This complements mobile.de rather than replacing it: mobile.de still has the deepest
dealer inventory. For private-seller cars, market-price analysis and sourcing,
Kleinanzeigen is the better, more accessible source — and it's the half most people miss.

FAQ

Do I need an API key? No — give a location and optional filters.

Can I filter by make/model? Best-effort by title keyword; for exact make filtering,
paste a Kleinanzeigen search URL with the make facet selected.

What fields do I get? Make, model, year, first registration, mileage, price, fuel,
transmission, power, condition, color and location, where the listing provides them.

Is it legal? It reads publicly available listing data. Use it responsibly and
within applicable laws and Kleinanzeigen's terms.

Get the private-seller side of the German car market with the Kleinanzeigen Autos Scraper — a data complement to mobile.de.

Zillow Alternative: Free Real-Estate Data Sources That Don't Block You

Ben — Sat, 27 Jun 2026 11:15:59 +0000

Everyone tries to scrape Zillow first — and hits a wall of CAPTCHAs, bans and a $300–
$800/month lead bill. The smarter move is to pull comparable real-estate data from
sources that are far easier to access and often carry inventory Zillow doesn't.
Here are three Zillow alternatives for actual data, by use case.

1. Off-market & FSBO leads → Craigslist

Zillow's leads are bought by every agent in your zip. Craigslist is full of
for-sale-by-owner sellers with no agent — exclusive, direct-contact leads that
aren't on the MLS. The
Craigslist Real Estate Scraper
pulls price, beds, baths, sqft, photos, contact availability and a 0–100 lead score,
with an owner-only filter for pure FSBO.

2. US homes for sale, rent & sold comps → Redfin

For structured listing data and sold comps (the part Zillow gates hardest), Redfin
is more accessible. The
Redfin Scraper returns price,
beds, baths, sqft, address and photos across for-sale, rental and sold — clean enough
for valuation models and market dashboards.

3. International / UK property → OnTheMarket

Zillow is US-only. For the UK, the
OnTheMarket Scraper pulls
for-sale and to-rent listings with price, beds, type, address, agent and even
latitude/longitude — without the heavy blocking Rightmove and Zoopla throw up.

Why these beat fighting Zillow

They don't ban you on sight — Craigslist, Redfin and OnTheMarket are far more scrapable than Zillow, so runs actually finish.
Different, exclusive inventory — FSBO and off-market deals Zillow never shows.
Pay-as-you-go — per-result pricing instead of a $300–800/month lead contract.
Own the data — export to CSV/Excel/JSON for your CRM, model or product.

Honest note

These aren't Zillow scrapers — they're alternative sources of comparable data
(listings, comps, leads). If you specifically need Zillow's Zestimate, you'll need
Zillow. For everything else — leads, listings, comps, market trends — these are
cheaper, more reliable, and often richer.

FAQ

Do I need an API key? No — give a city/location and your filters.

Can I get sold comps? Yes — via the Redfin scraper (for-sale, rent and sold).

FSBO only? Yes — the Craigslist scraper has an owner-only filter.

Is it legal? These read publicly available listing data. Use responsibly and
within each site's terms and applicable law.

Real-estate data without the Zillow wall: Craigslist Real Estate (FSBO), Redfin (US comps), OnTheMarket (UK).