DEV Community: Ben

How to Scrape Local Job Ads from Kleinanzeigen (Germany) — Python + No-Code

Ben — Thu, 25 Jun 2026 20:48:25 +0000

Most job-scraping guides point you at corporate ATS boards — Greenhouse, Lever,
Ashby. Those only carry mid-to-large-company tech and office roles. The entire
local German labour market — trades, care, hospitality, retail, mini-jobs — is
posted somewhere else entirely: the Jobs (Stellenangebote) section of
Kleinanzeigen. If you want hyper-local hiring data, that's the source.

Why Kleinanzeigen for jobs?

Hidden inventory — small businesses, shops, restaurants and private employers post here directly; these roles are absent from corporate boards and aggregators.
Hyper-local — search any city or PLZ with a radius.
Structured fields — employment type, job category, hourly/monthly pay and experience level, once you parse the detail page.

Parsing the German job attributes

The key insight is what each German attribute actually means — it's easy to get
these backwards:

Arbeitszeit → employment type (Vollzeit / Teilzeit / Minijob)
Art → job category (Reinigungskraft, Verkäufer/-in, …)
Stundenlohn → hourly wage
Berufserfahrung → experience level

def parse_job(attrs: dict):
    a = {k.lower(): v for k, v in attrs.items()}
    def find(*keys):            # keys tried in priority order
        for k in keys:
            for label, val in a.items():
                if k in label:
                    return val
    return {
        "employment_type": find("arbeitszeit", "anstellung"),
        "job_category": find("art"),
        "experience": find("berufserfahrung", "erfahrung"),
        "salary": find("stundenlohn", "gehalt"),
    }

The attribute rows come off the detail page the same way as any Kleinanzeigen
section (read the .addetailslist--detail--value node, strip it from the row text
to get the label), and you'll want a residential DE proxy plus the
location-autocomplete endpoint to turn a city name into the internal location ID.

One handy quirk: titles contain the role

Unlike cars (where titles often omit the make), job titles reliably contain the
role — "Reinigungskraft (m/w/d)", "LKW Fahrer in Vollzeit". So a client-side
title filter on a keyword like fahrer or pflegekraft works well, on top of the
location browse.

The catch: scale and upkeep

Paging results, fetching each detail page politely, mapping the German attributes
correctly, extracting pay from messy free text, and rotating residential IPs — then
maintaining it — adds up fast.

The no-code option

The Kleinanzeigen Jobs Scraper
on Apify does it for you: enter a role and a location (or leave the role blank for
everything in the area), click Run, get clean rows.

{
  "keyword": "fahrer",
  "locationCode": "münchen",
  "radiusKm": 20,
  "employmentType": "Vollzeit",
  "maxResults": 100
}

Output is one clean row per ad — title, job category, employment type, salary +
period, experience, company, city, PLZ, posting date and the listing URL — ready
for a spreadsheet, a database, or an LLM.

Common use cases

Recruiting & sourcing — find local hires and employers outside the big boards.
Labour-market analysis — track local demand, wages and employment types by region.
Job-board aggregation — enrich your board with hyper-local German listings.
Lead generation — small employers actively hiring are strong B2B leads.

FAQ

How is this different from an ATS scraper? Corporate boards carry mid/large-company
roles only. Kleinanzeigen carries local, hourly, trade, care and mini-jobs — a
completely different, hyper-local inventory.

Do I need an API key? No — a residential DE proxy is used automatically.

Is this legal? You're reading publicly available listing data. Use it responsibly
and within applicable laws and Kleinanzeigen's terms.

Building something with German job data? The Kleinanzeigen Jobs Scraper handles the scraping so you can focus on the product.

How to Scrape Used-Car Listings from Kleinanzeigen (Germany) — Python + No-Code

Ben — Thu, 25 Jun 2026 20:47:39 +0000

If you want used-car data from Germany, everyone reaches for mobile.de — but
that's the dealer market. The private-seller market lives on
Kleinanzeigen Autos, and it's where the deals (and the arbitrage) are. This post
shows how to pull structured car listings — make, model, year, mileage, fuel,
gearbox, power and price — out of it.

Why Kleinanzeigen for cars?

Private sellers — typically cheaper than dealer platforms; ideal for deal-hunting and market analysis.
Different inventory — a complement to mobile.de, not a duplicate.
Rich attributes — once you parse the detail page you get the full German spec sheet, not just a title and a price.

Parsing the German car attributes

Kleinanzeigen stores the spec sheet as label/value rows. Map the German labels to
clean English keys, with a regex fallback on the title for year and mileage when a
field is missing:

import re
from bs4 import BeautifulSoup

YEAR = re.compile(r"\b(?:19|20)\d{2}\b")
KM = re.compile(r"([\d.]+)\s*km\b", re.I)

def parse_car(attrs: dict, title: str):
    a = {k.lower(): v for k, v in attrs.items()}
    def find(*keys):
        for label, val in a.items():
            if any(k in label for k in keys):
                return val
    er = find("erstzulassung") or ""
    ym = YEAR.search(er) or YEAR.search(title)
    return {
        "make": find("marke"),
        "model": find("modell"),
        "year": int(ym.group()) if ym else None,
        "mileage_km": find("kilometerstand"),
        "fuel": find("kraftstoff"),
        "transmission": find("getriebe"),
        "power": find("leistung"),
    }

The attribute rows themselves come off the detail page like this:

soup = BeautifulSoup(html, "lxml")
attrs = {}
for li in soup.select(".addetailslist--detail"):
    val = li.select_one(".addetailslist--detail--value")
    if not val:
        continue
    v = val.get_text(" ", strip=True)
    label = li.get_text(" ", strip=True).replace(v, "", 1).strip()
    attrs[label] = v

As with any Kleinanzeigen section, you'll want a residential DE proxy and the
location-autocomplete endpoint to turn a city name into the internal location ID.

One gotcha: make is a facet, not free text

Kleinanzeigen treats the car make as a structured filter, and many listing titles
omit the brand (e.g. "320d Touring, Automatik"). So a naive keyword search on
"BMW" misses cars. The robust approach is to browse the Autos category by location
and filter make/model client-side from the parsed make field — or paste a search
URL with the make facet already selected.

The catch: scale and upkeep

Paging hundreds of results, fetching each detail page politely, normalizing the
German spec sheet, handling the make-facet quirk, and rotating residential IPs —
then keeping it alive as the site changes — is real, ongoing work.

The no-code option

The Kleinanzeigen Autos Scraper
on Apify does it for you: pick a location (and optional price/year/mileage filters),
click Run, get clean rows.

{
  "locationCode": "berlin",
  "maxPrice": 20000,
  "minYear": 2016,
  "maxMileage": 120000,
  "maxResults": 100
}

Output is one clean row per car — make, model, year, first registration, mileage,
price, fuel, transmission, power, condition, color, city and the listing URL —
ready for a spreadsheet, a database, or an LLM.

Common use cases

Deal-hunting & arbitrage — spot underpriced private-seller cars fast.
Price & market analysis — track asking prices by make/model/region over time.
Dealers & resellers — source private inventory and leads.
Automotive apps — power a search product with structured listings.

FAQ

How is this different from mobile.de? Kleinanzeigen is dominated by private
sellers and budget cars — a different inventory, useful for arbitrage and broader
coverage.

Do I need an API key? No — a residential DE proxy is used automatically.

Is this legal? You're reading publicly available listing data. Use it responsibly
and within applicable laws and Kleinanzeigen's terms.

Building something with German car data? The Kleinanzeigen Autos Scraper handles the scraping so you can focus on the product.

How to Scrape Real Estate Listings from Kleinanzeigen (Germany) — Python + No-Code

Ben — Thu, 25 Jun 2026 20:46:40 +0000

Kleinanzeigen (formerly eBay Kleinanzeigen) is Germany's largest classifieds site,
and its Immobilien section is one of the richest sources of private-landlord
rental and sale listings anywhere in the DACH region — the kind of inventory that
never makes it onto the big paid portals. If you're building a rental-market
dashboard, a relocation tool, or just hunting for underpriced flats, this is where
the data is.

The problem: Kleinanzeigen renders server-side German HTML, blocks datacenter IPs,
and hides the structured attributes (Zimmer, Wohnfläche, Kaltmiete) inside the
detail page. Here's how to get clean data out of it.

Why Kleinanzeigen for real estate?

Private landlords — many listings are direct from owners, often below market and absent from Immowelt/ImmoScout.
Hyper-local — search any PLZ or city with a radius, down to the district.
Structured fields — rooms, living space, cold/warm rent, deposit, availability date and more, once you parse the detail page.

The hard part: location IDs and German attributes

Two things trip up a naive scraper:

Location resolution. Kleinanzeigen doesn't search by city name directly — it uses internal location IDs (e.g. Berlin → l3331). You resolve them through its autocomplete endpoint:

import httpx

q = "münchen"
r = httpx.get("https://www.kleinanzeigen.de/s-ort-empfehlungen.json",
              params={"query": q}, timeout=15)
# -> {"_6411": "München", ...}  the digits after the underscore are the location id
print(r.json())

Attribute parsing. On the detail page each attribute is a label/value pair without a clean class on the label, so you read the value node and strip it back out of the row text:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
attrs = {}
for li in soup.select(".addetailslist--detail"):
    val = li.select_one(".addetailslist--detail--value")
    if not val:
        continue
    value = val.get_text(" ", strip=True)
    label = li.get_text(" ", strip=True).replace(value, "", 1).strip()
    attrs[label] = value
# attrs -> {"Wohnfläche": "72 m²", "Zimmer": "3", "Kaltmiete": "1.150 €", ...}

You also need a residential DE proxy — datacenter IPs get a soft block fast.

The catch: scale and upkeep

One listing is easy. A useful dataset needs paging through hundreds of results,
fetching every detail page politely, normalizing German fields into clean keys
(rooms, living_space_sqm, cold_rent_eur), handling umlauts in city slugs, and
rotating residential IPs — then staying alive as the markup shifts. That's days of
maintenance.

The no-code option

The Kleinanzeigen Immobilien Scraper
on Apify does all of it: enter a city or PLZ (names auto-resolve to the right
location ID) and an optional price/room filter, click Run, and get clean rows.

{
  "locationCode": "münchen",
  "radiusKm": 20,
  "maxResults": 200
}

Output is one tidy row per listing — title, rooms, living space, cold/warm rent,
deposit, address, PLZ, images and the listing URL — ready for a spreadsheet, a
database, or an LLM. Residential DE proxies and attribute parsing are built in.

Common use cases

Rental-market analysis — track asking rents by district and room count over time.
Deal-hunting — surface underpriced private-landlord flats the moment they post.
Relocation & proptech apps — power search with structured German listings.
Lead generation — reach private landlords directly.

FAQ

Do I need an API key? No — a residential DE proxy is used automatically.

Can I search by city name? Yes — city names auto-resolve to Kleinanzeigen's
internal location IDs; you can also pass a PLZ or paste a full search URL.

Is this legal? You're reading publicly available listing data. Use it responsibly
and within applicable laws and Kleinanzeigen's terms.

Building something with German property data? The Kleinanzeigen Immobilien Scraper handles the scraping so you can focus on the product.

Pushshift Alternative: Scrape Historical Reddit Posts & Comments (2026)

Ben — Tue, 23 Jun 2026 19:49:50 +0000

Pushshift was the de facto way to pull historical Reddit data — until access got heavily restricted and it stopped being the easy public option it once was. If you need posts and comments older than what Reddit's API will give you, here's a working alternative in 2026.

Why the official Reddit API isn't enough

Reddit's API caps listings at roughly 1,000 items per endpoint, so you can't page back through a large subreddit's full history. That's fine for recent data, useless for backfilling months or years — which is exactly why people used Pushshift.

The alternative: Reddit Archive Scraper

The Reddit Archive Scraper on Apify pulls historical Reddit posts and comments from the community PullPush archive (Pushshift's successor) — by subreddit, date range, and keyword — reaching data the official API can't. For current/live data, the Reddit Scraper returns posts and comments as clean markdown for AI/RAG.

{ "subreddits": ["datascience"], "since": "2020-01-01", "until": "2022-12-31", "keyword": "career" }

Pushshift vs. this

	Pushshift (today)	Reddit Archive Scraper
Access	Restricted / limited	Open, run on demand
History depth	Was full archive	Years, via PullPush
Setup	API + auth hurdles	No API key, run + export
Output	Raw JSON	Structured JSON / CSV
Live data	No	Pair with Reddit Scraper

Use cases

AI / RAG datasets — years of real Q&A and discussion as training context.
Academic & social research — longitudinal analysis of communities.
Trend & sentiment analysis — track how opinion shifted over time.
Brand/competitor history — every historical mention in a subreddit.

FAQ

What replaced Pushshift? The PullPush archive is the community successor; the Reddit Archive Scraper wraps it so you can query by subreddit, date, and keyword without managing access.

Why can't I just use the Reddit API for old posts? It caps listings at ~1,000 items, so deep history isn't reachable through it.

What format is the output? Structured JSON/CSV; the live Reddit Scraper outputs AI-ready markdown with token counts.

Is it legal? It reads publicly available Reddit data. Use responsibly and within applicable terms.

Need Reddit history beyond the API's limit? The Reddit Archive Scraper pulls years of posts by subreddit, date, and keyword.

Hunter.io Alternative: Find & Verify Business Emails by Domain (Pay-As-You-Go)

Ben — Tue, 23 Jun 2026 19:48:10 +0000

Hunter.io is a popular email finder, but its plans are monthly subscriptions with credit caps — overkill if you just need to enrich a list now and then. If you want to find and verify business emails without a recurring subscription, here's a pay-per-result alternative.

The job to be done

Given a person's name and their company domain, you want the most likely verified work email — for sales outreach, recruiting, or CRM enrichment. The mechanics are the same everywhere: generate the common corporate patterns, then verify which one actually receives mail (MX + SMTP), while detecting catch-all domains so you don't get false positives.

The alternative: Smart Email Finder & Verifier

The Smart Email Finder & Verifier on Apify does exactly that — from a name + domain it generates ~14 patterns, verifies via SMTP where possible, detects catch-all domains, and returns a confidence score. You pay per verified email found, with no monthly fee.

{ "contacts": [ { "first_name": "Jane", "last_name": "Doe", "domain": "acme.com" } ] }

Hunter.io vs. pay-per-use finder

	Hunter.io	This finder
Pricing	Monthly subscription + credit caps	Pay per verified email, no subscription
Verification	Yes	MX + SMTP + catch-all detection
Bulk	Yes (plan-limited)	Yes — array or CSV
Confidence score	Yes	Yes (0–100)
Best for	Always-on teams	Lists, projects, on-demand enrichment

Use cases

Cold outreach — turn a list of names + companies into reachable contacts.
Recruiting — reach candidates directly.
CRM enrichment — fill missing email fields in bulk.
PR & partnerships — find the right person at a target company.

Do it responsibly

Use verified business emails for legitimate, consented outreach; follow CAN-SPAM / GDPR and always include an opt-out. Don't target personal data.

FAQ

Is there a cheaper Hunter.io alternative? Yes — a pay-per-result finder avoids monthly subscriptions; you only pay for emails actually found.

How accurate is it? High for companies with standard formats and non-catch-all domains (SMTP-verified). Big-tech domains block SMTP from datacenters, so you get a pattern-based best guess with a lower confidence score.

Can I run a whole list? Yes — pass an array of contacts or CSV for bulk enrichment.

Do I need an API key? No — give it a name and domain.

Turning a name list into reachable contacts? The Smart Email Finder & Verifier finds and verifies professional emails by domain — bulk, pay-as-you-go.

BuiltWith Alternative: Detect Any Website's Tech Stack via API (No Subscription)

Ben — Tue, 23 Jun 2026 19:46:37 +0000

BuiltWith is the go-to for "what is this site built with" — but its paid plans start around $295/month, which is a lot if you just need to enrich a lead list or check a few hundred domains now and then. If you don't need an enterprise subscription, here's a pay-as-you-go alternative that returns the same kind of data via a simple API.

What you actually need

Most people reaching for BuiltWith want one of three things:

Sales intelligence — "does this company use Shopify / HubSpot / Salesforce?"
Lead scoring — filter a list by the tech a prospect runs
Competitive research — see a competitor's analytics, CMS, and hosting

For all of these, you need to point a tool at a list of domains and get back the technologies — not pay for a yearly seat.

The alternative: Website Tech Stack Detector

The Website Tech Stack Detector (and its sibling Tech Stack Detector) on Apify detect 100+ technologies on any site — frameworks, CMS, analytics, hosting, payment, marketing tools — and return clean JSON. You pay per site checked, with no monthly subscription.

{ "urls": ["https://example.com", "https://shopify-store.com"] }

Output (per site): detected technologies grouped by category, plus the domain and a success flag — ready for a spreadsheet, CRM, or your own scoring model.

BuiltWith vs. pay-per-use detector

	BuiltWith (paid plans)	This detector
Pricing	Subscription (~$295+/mo)	Pay per site, no subscription
Best for	Large, ongoing programs	Lists, enrichment, occasional checks
Access	Web app + API (higher tiers)	API + no-code UI out of the box
Output	Reports	Clean JSON / CSV / Excel
Lock-in	Annual plans	None

If you run BuiltWith daily across millions of domains, their subscription may be worth it. If you enrich lists or check domains on demand, pay-per-use wins.

Use cases

Outbound sales — find every prospect on Shopify/Webflow/HubSpot before you pitch.
Lead enrichment — append tech stack to a CRM list in bulk.
Competitive & market research — map what tools a segment uses.
Recruiting/agencies — qualify clients by their stack.

FAQ

Is there a free or cheaper BuiltWith alternative? Yes — a pay-per-use detector like the one above avoids BuiltWith's monthly subscription; you only pay for sites you check.

Does it detect the same technologies? It covers 100+ common technologies across frameworks, CMS, analytics, hosting, payments, and marketing tools.

Can I check a bulk list of domains? Yes — pass an array of URLs and get one structured record per site.

Do I need an API key? No — run it via API or the no-code UI.

Need it for a list of domains? The Website Tech Stack Detector checks them in one run — no subscription, just pay per site.

How to Find Anyone's Professional Email from Name + Company (2026 Guide)

Ben — Mon, 22 Jun 2026 14:15:08 +0000

Finding a verified work email from just a name and a company domain is the backbone of cold outreach, recruiting, and sales prospecting. Here's how the pattern-and-verify approach works, how to do it in Python, and a no-code option that does the whole thing for you.

The idea: permute, then verify

Most companies use one of a handful of email formats. Given Jane Doe at acme.com, the likely candidates are:

first, last, domain = "jane", "doe", "acme.com"
patterns = [
    f"{first}.{last}@{domain}",   # jane.doe@
    f"{first}{last}@{domain}",    # janedoe@
    f"{first[0]}{last}@{domain}", # jdoe@
    f"{first}@{domain}",          # jane@
    f"{last}{first[0]}@{domain}", # doej@
]

That's the easy part. The hard part is verification — confirming which candidate actually receives mail without sending a test email. That's done with MX lookups and SMTP handshake checks (and you'll hit greylisting, catch-all domains, and rate limits along the way).

The Python reality

You can script MX + SMTP checks with dnspython and smtplib, but at any volume you'll need IP rotation (many mail servers block repeated probes), catch-all detection, and back-off logic — which is why most teams use a service.

The no-code option

The Smart Email Finder & Verifier generates the common patterns and verifies them (SMTP where possible) with a confidence score, from just a name + domain:

{
  "contacts": [
    { "fullName": "Jane Doe", "domain": "acme.com" }
  ]
}

You get the most likely address, the patterns tried, and a confidence rating — ready to drop into your CRM or outreach tool.

Use cases

Sales prospecting — turn a lead list of names + companies into reachable contacts.
Recruiting — reach candidates directly.
PR & partnerships — find the right person at a target company.
CRM enrichment — fill missing email fields in bulk.

A note on doing this responsibly

Use verified business emails for legitimate, consented outreach, follow CAN-SPAM / GDPR rules, and always include an opt-out. Don't scrape or target personal data.

FAQ

How accurate is pattern + verify? For companies with standard formats and non-catch-all domains, high. Catch-all domains can't be verified by SMTP — the confidence score tells you when.

Do I need an API key? Not for the Smart Email Finder & Verifier — give it a name and domain.

Can I run a whole list? Yes — pass an array of contacts for bulk enrichment.

Turning a name list into reachable contacts? The Smart Email Finder & Verifier finds and verifies professional emails by domain — bulk, with confidence scores.

How to Scrape Reddit Posts & Comments for AI / RAG (Python + No-Code)

Ben — Mon, 22 Jun 2026 14:13:38 +0000

Reddit is one of the richest sources of real human opinion on the internet — which makes it gold for RAG pipelines, sentiment analysis, and market research. Here's how to pull Reddit posts and comments in 2026, the limits to know about, and a no-code option that outputs clean markdown ready for embeddings.

Option 1: the official Reddit API (PRAW)

import praw

reddit = praw.Reddit(client_id="...", client_secret="...", user_agent="my-app")
for post in reddit.subreddit("dataengineering").hot(limit=25):
    print(post.title, post.score, post.url)
    post.comments.replace_more(limit=0)
    for c in post.comments.list()[:10]:
        print("  ", c.body[:120])

This works, but note the catches: you need an app + OAuth credentials, the free tier is rate-limited, and listings are capped at ~1000 items — so you can't page back through a large subreddit's full history.

Option 2: historical archives (beyond the 1000-item cap)

For data older than what the API will return, the community archive PullPush (the successor to Pushshift) lets you query historical posts and comments by subreddit, date range, and keyword — useful for backfilling years of data.

Option 3: no-code / markdown output for AI

If your goal is an AI/RAG dataset, you mostly want clean text, not JSON plumbing. The Reddit Scraper returns posts, comments, and user data as AI-ready markdown (with word/token counts), no API keys to manage:

{
  "subreddits": ["MachineLearning", "LocalLLaMA"],
  "sort": "top",
  "maxPosts": 200,
  "includeComments": true
}

For deep history beyond the API's cap, there's a companion Reddit Archive Scraper that pulls years of posts from the archive by date range.

Which should you use?

Small, live pulls? The official API via PRAW is free and fine.
AI/RAG datasets or historical backfill? A managed scraper saves you OAuth, rate-limit handling, and the markdown conversion.

Use cases

RAG / fine-tuning datasets — real Q&A and discussion as training context.
Sentiment & trend analysis — track opinion on products, tickers, or topics.
Market research — mine niche subreddits for pain points and feature requests.
Community monitoring — watch mentions of your brand or competitors.

FAQ

Do I need Reddit API credentials? For PRAW, yes. The managed scraper handles access for you.

Why only ~1000 posts from the API? Reddit caps listing pagination. Use an archive source for deeper history.

What format is best for AI? Markdown or plain text with token counts — which is what the Reddit Scraper outputs.

Building an AI app on Reddit data? The Reddit Scraper gives you posts and comments as clean markdown — no API keys, no rate-limit juggling.

How to Scrape Yahoo Finance Stock Data in 2026 (Python + No-Code API)

Ben — Mon, 22 Jun 2026 13:50:22 +0000

Yahoo Finance is still the most accessible source of free stock market data — real-time quotes, historical prices, fundamentals, analyst ratings, and news. This guide covers the practical ways to pull it in 2026, the gotchas, and a no-code option when you don't want to babysit a scraper.

Option 1: yfinance (Python)

The yfinance library wraps Yahoo's endpoints:

import yfinance as yf

t = yf.Ticker("AAPL")
print(t.fast_info["last_price"])          # real-time-ish quote
hist = t.history(period="1mo")            # historical OHLCV
print(hist.tail())
print(t.news[:3])                         # latest news

It's great for notebooks and quick analysis. The downsides at scale: Yahoo rate-limits aggressive polling, the unofficial endpoints change without notice, and you'll be maintaining retry/back-off logic and proxies once you go beyond a handful of tickers.

Option 2: hit the JSON endpoints directly

yfinance is really calling endpoints like https://query1.finance.yahoo.com/v8/finance/chart/AAPL. You can call them yourself, but you'll quickly run into consent cookies, crumb tokens, and 429s — which is why most people either use the library or a managed API.

Option 3: no-code / API (no maintenance)

If you want clean JSON without maintaining any of the above, the Yahoo Finance Scraper on Apify returns quotes, company info, historical prices, financials, analyst recommendations, and news for any list of tickers — via API or a simple UI, no key on your side:

{
  "symbols": ["AAPL", "MSFT", "NVDA"],
  "dataTypes": ["quote", "history", "news"]
}

You get one structured record per ticker, ready for a dashboard, a backtest, or an LLM. Because it runs on managed infrastructure, the rate-limit and rotation problems are handled for you.

Which should you use?

Exploring in a notebook? yfinance is perfect and free.
Building a product or pulling many tickers on a schedule? A managed API saves you the cat-and-mouse of cookies, crumbs, and rate limits.

Use cases

Trading dashboards & alerts — live quotes and historical series.
Backtesting — clean OHLCV history across many tickers.
Research & screeners — fundamentals and analyst data at scale.
AI/LLM finance apps — structured market data as context.

FAQ

Is Yahoo Finance data free? For personal and most research use, yes. Respect Yahoo's terms for redistribution.

How do I avoid rate limits? Back off on 429s and cache; or use a managed API that handles rotation.

Can I get fundamentals and news too? Yes — yfinance exposes them, and the Yahoo Finance Scraper returns them in one structured response.

Pulling Yahoo Finance at scale? The Yahoo Finance Scraper gives you quotes, history, fundamentals, and news as clean JSON — no key, no rate-limit headaches.

How to Scrape Jobs from Greenhouse, Lever & Ashby (Free Python + No-Code)

Ben — Mon, 22 Jun 2026 13:48:35 +0000

If you've ever tried to build a job board, a sourcing tool, or a hiring-trends dashboard, you've hit the same wall: the big job aggregators are stale, rate-limited, and wrapped in anti-bot defenses. The fix most people miss is that thousands of companies publish their open roles through public ATS APIs — Greenhouse, Lever, and Ashby all expose JSON endpoints that anyone can read, no key required.

This post shows the free Python way to pull jobs from each, and a one-click no-code option if you'd rather skip the maintenance.

Why scrape ATS endpoints instead of job boards?

First-party and fresh — you read the same data that powers the company's careers page, the moment a role opens.
Working apply links — every job links to the real application.
No blocking — these are public JSON APIs, so no proxies or headless browsers.

Greenhouse jobs API (Python)

Greenhouse exposes a board endpoint per company token:

import httpx

company = "stripe"  # the slug in boards.greenhouse.io/<slug>
url = f"https://boards-api.greenhouse.io/v1/boards/{company}/jobs?content=true"
jobs = httpx.get(url, timeout=30).json()["jobs"]
for j in jobs[:5]:
    print(j["title"], "—", j["location"]["name"], "—", j["absolute_url"])

content=true includes the full HTML job description.

Lever jobs API (Python)

import httpx

company = "openai"  # the slug in jobs.lever.co/<slug>
url = f"https://api.lever.co/v0/postings/{company}?mode=json"
for j in httpx.get(url, timeout=30).json()[:5]:
    cats = j.get("categories", {})
    print(j["text"], "—", cats.get("location"), "—", j["hostedUrl"])

Ashby jobs API (Python)

import httpx

org = "notion"  # the slug in jobs.ashbyhq.com/<slug>
url = f"https://api.ashbyhq.com/posting-api/job-board/{org}"
for j in httpx.get(url, timeout=30).json()["jobs"][:5]:
    print(j["title"], "—", j["location"], "— remote:", j.get("isRemote"))

The catch: scale and upkeep

One company is easy. But a useful product needs dozens of companies across all three ATS, normalized into one schema, de-duplicated, filtered by keyword, location, and remote status — and kept running as companies come and go. That's where a maintained tool saves you days.

The no-code option

The ATS Jobs Aggregator on Apify does exactly this: it pulls live jobs from Greenhouse, Lever, Ashby, SmartRecruiters, and Recruitee in one run, normalized into a single dataset, with keyword/location/remote filters and de-duplication built in. It ships with a curated list of well-known companies so it returns results on the first click, and you can drop in your own company slugs to track exactly the employers you care about.

{
  "sources": ["greenhouse", "lever", "ashby"],
  "keywords": ["engineer", "data"],
  "remoteOnly": true,
  "maxResults": 500
}

Output is one clean row per job — title, company, location, department, employment type, remote flag, apply URL, and posted date — ready for a spreadsheet, a database, or an LLM.

Common use cases

Niche job boards — power a regional or role-specific board with fresh listings.
Sourcing & recruiting — see who's hiring for a role across dozens of companies at once.
Talent & market intelligence — track hiring velocity and locations by company.
Sales signals — companies hiring for a function often signal budget and growth.

FAQ

Do I need an API key? No — all three endpoints are public.

How do I find a company's slug? It's in their careers URL: boards.greenhouse.io/<slug>, jobs.lever.co/<slug>, jobs.ashbyhq.com/<slug>.

Is this legal? You're reading publicly published job data. Use it responsibly and within applicable terms.

Building something with job data? The ATS Jobs Aggregator handles the scraping so you can focus on the product.

Best Game & Media Data Scrapers and APIs in 2026

Ben — Tue, 02 Jun 2026 11:55:38 +0000

Building a game-deal site, a music database, an anime app, or a media catalog? Here are the best no-code ways to pull structured data about games, music, anime and apps in 2026 — with honest pros and cons.

TL;DR: For game prices and metadata use the Steam Scraper. For music, podcasts and apps, the iTunes Scraper. For vinyl and releases, the Discogs Scraper. For anime, the Anime Scraper.

What to look for in a media data tool

Official/keyless source — avoids breakage and captchas.
Rich fields — price, rating, genre, release date, artwork, IDs.
Search + filters — by keyword, category, country store.
Clean output — flat JSON/CSV you can load straight into an app or DB.

1. Steam Scraper — game prices, ratings & metadata

Searches the Steam store by keyword and returns name, price, discount, Metacritic score, genres, developers, publishers, release date, description, recommendations and platforms.

Pros: keyless and fast; great for price tracking and deal aggregators; rich metadata via the appdetails enrichment.
Cons: Steam store catalog only (not other game stores).
Best for: game-deal sites, price trackers, and games databases.

➡️ Steam Scraper

2. iTunes Scraper — music, podcasts, apps & more

Wraps Apple's official iTunes Search API. One tool covers music, podcasts, apps, audiobooks, movies, TV shows and ebooks — returning title, artist, price, genre, rating, artwork and preview/view URLs.

Pros: official Apple data; keyless; covers many media types and country stores in one input.
Cons: Apple ecosystem data (not Spotify/Google Play).
Best for: media catalogs, podcast/music discovery tools, and app research.

➡️ iTunes Scraper

3. Discogs Scraper — vinyl, CDs & music releases

Searches the Discogs database — the world's largest catalog of music releases — by artist, album, genre or year. Returns title, year, country, genre, style, format, label, catalog number, barcode and cover image.

Pros: unmatched release/pressing detail; keyless; great for collectors and marketplaces.
Cons: music-only; rate-limited for very large pulls.
Best for: record collecting, discography databases, and music marketplace analytics.

➡️ Discogs Scraper

4. Anime Scraper — MyAnimeList data

Pulls anime data from MyAnimeList via the public Jikan API: title, type, episodes, status, score, rank, popularity, studios, genres, synopsis, season/year and artwork. Search by title or fetch top-ranked lists.

Pros: rich anime metadata; keyless; supports top-N lists for leaderboards.
Cons: anime-specific (by design).
Best for: anime databases, recommendation tools, and Discord bots.

➡️ Anime Scraper

5. Raw official APIs (Steam, iTunes, Jikan, Discogs)

You can call each underlying API yourself.

Pros: free, full control.
Cons: different formats, pagination, and rate limits per source; you build and maintain the glue code.
Best for: developers who want to script it themselves.

Quick comparison

Scraper	Domain	Key fields	Keyless	No-code
Steam	Games	price, discount, Metacritic, genres	Yes	Yes
iTunes	Music/podcasts/apps	artist, price, genre, artwork	Yes	Yes
Discogs	Vinyl/music	year, label, format, cat#	Yes	Yes
Anime	Anime	score, rank, studios, genres	Yes	Yes
Raw APIs	Varies	Varies	Yes	No

How to start (no code)

Open any scraper above on Apify.
Enter a keyword (and country store or category where relevant).
Set maxResults and run.
Export JSON/CSV, or schedule recurring runs to keep data fresh.

Conclusion

For media and game data in 2026, no-code scrapers over official sources are the sweet spot — clean output, no key setup, no maintenance:

Games → Steam Scraper
Music/podcasts/apps → iTunes Scraper
Vinyl/releases → Discogs Scraper
Anime → Anime Scraper

How to Build a Research Paper Dataset for RAG & LLMs (No Code, 2026)

Ben — Tue, 02 Jun 2026 11:54:22 +0000

Grounding an LLM or running a literature review? You need a clean corpus of papers — titles, abstracts, authors, citations, PDF links. Here's how to build one in minutes without writing a scraper, pulling from arXiv, OpenAlex and PubMed.

What you'll build: a structured JSON dataset of academic papers on your topic, ready to drop into a vector database, a notebook, or a RAG pipeline.

Why not just hit the APIs directly?

arXiv, OpenAlex and PubMed are all free and open — but each returns a different format (Atom XML, nested JSON, E-utilities), with its own pagination and rate limits. Wiring that up and flattening it is a half-day of glue code you'll have to maintain.

Instead, we'll use three no-code scrapers that wrap those official APIs and return clean, flat JSON:

arXiv Scraper — CS/ML/physics preprints
OpenAlex Scraper — 250M+ works across all fields, with citations
PubMed Scraper — 37M+ biomedical citations

Step 1 — Pick your source by field

Machine learning / CS / physics → arXiv
Anything, with citation counts → OpenAlex
Medicine / life sciences → PubMed

For an AI/ML RAG corpus, start with arXiv.

Step 2 — Run the arXiv Scraper

Open the arXiv Scraper and set:

{
  "allFields": "retrieval augmented generation",
  "sortBy": "newest",
  "maxResults": 200
}

Or use the advanced query syntax for precision:

{ "searchQuery": "cat:cs.CL AND abs:retrieval augmented", "maxResults": 500 }

Each result includes the abstract and a direct PDF link:

{
  "arxiv_id": "2605.30351v1",
  "title": "VideoMLA: Low-Rank Latent KV Cache...",
  "authors": ["Hidir Yesiltepe", "Jiazhen Hu"],
  "abstract": "Long-rollout causal video diffusion...",
  "primary_category": "cs.CV",
  "pdf_url": "https://arxiv.org/pdf/2605.30351v1"
}

Step 3 — Add breadth and citations with OpenAlex

To rank by impact or cover non-arXiv venues, run the OpenAlex Scraper:

{
  "searchQuery": "retrieval augmented generation",
  "sortBy": "citations",
  "fromYear": 2020,
  "maxResults": 200
}

You get cited_by_count, is_open_access and PDF links — perfect for prioritising the most influential papers in your corpus.

Step 4 — Export and load into your pipeline

From the Storage tab, export each dataset as JSON (or pull it via the API). Then in your RAG pipeline:

Use the abstract (and the pdf_url if you want full text) as your documents.
Keep title, authors, doi, arxiv_id as metadata for citations.
Chunk, embed, and load into your vector DB (Pinecone, Weaviate, pgvector, etc.).

Because the output is already flat JSON with consistent field names across runs, there's no per-source parsing — you merge the three datasets and you're done.

Step 5 — Keep it fresh

Schedule the arXiv Scraper with sortBy: newest to append new papers on your topic each week, so your RAG index stays current without re-scraping everything.

Bonus — add real-world signal

Papers tell you what researchers say; forums tell you what practitioners say. To enrich a dataset with public discussion, add the Reddit Archive Scraper to pull historical threads from subreddits like r/MachineLearning by date range and keyword.

Wrap up

Three no-code scrapers, three short inputs, and you've got a clean, multi-source research corpus — no XML parsing, no rate-limit juggling. Start with the arXiv Scraper and layer in OpenAlex and PubMed as you need breadth.