DEV Community: g4dr

Scrape Kalshi Prediction Markets: Odds, Volume and Weather Data via API

g4dr — Tue, 20 Oct 2026 13:33:00 +0000

Kalshi is the regulated US prediction market — election odds, Fed decisions, economic prints, even weather markets. Its public API exists but is rate-limited and awkward for bulk history. This tutorial shows the Apify-actor pipeline: markets, order books, and cross-platform arbitrage with Polymarket.

🤖 Try it live — chat with a Poe bot that generates the exact pipeline for your use case: Lead List Generator · Job Postings Miner · Amazon Product Miner · SERP Rank Tracker · Social Lead Finder

The stack

Apify — Kalshi actors for markets, odds, volume, profiles. Free monthly credits included.
Python — 20 lines to a live odds feed.

Step 1 — Get your API token

Create an Apify account (free credits included)
Console → Settings → Integrations → copy your API token
export APIFY_TOKEN=your_token

Step 2 — Pull market data

curl -X POST "https://api.apify.com/v2/acts/scrapemint~kalshi-prediction-market-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"category": "economics", "maxItems": 100}'

Fields per market: ticker, title, yesPrice, noPrice, volume, openInterest, closeTime, url.

Step 3 — Build an odds feed

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("scrapemint/kalshi-prediction-market-scraper").call(run_input={
    "category": "economics", "maxItems": 200
})

for m in client.dataset(run["defaultDatasetId"]).iterate_items():
    yes = m.get("yesPrice") or 0
    print(f"{m.get('title', '')[:65]:65s} YES={yes:>4.2f} vol={m.get('volume', 0)}")

Schedule it in Apify Console → snapshots → price-movement alerts.

The arbitrage angle

Kalshi and Polymarket price the same events — differently. congism/polymarket-kalshi-arb-finder (835 users) does the cross-platform matching and flags YES+NO < $1.00 pairs. Both are regulated/legit venues, so the spread is real.

Niche gold: weather markets

bigdavidson/kalshi-weather-markets (282 users) pulls temperature/rain odds per city — energy traders, event insurers, and quants pay for this feed. Build it once, sell the dataset.

Useful actors

Actor	Job
`scrapemint/kalshi-prediction-market-scraper`	Markets + live event odds (115 users)
`congism/polymarket-kalshi-arb-finder`	Kalshi↔Polymarket arbitrage pairs
`bigdavidson/kalshi-weather-markets`	Temperature & rain odds API (282 users)
`pixflor/kalshi-markets-scraper`	Markets, odds, volume, order books
`saswave/kalshi-profile-scraper`	Trader profile data

Who uses this

Traders — arb alerts + order-book snapshots before market-moving events
Quants/researchers — odds history beats polling data for event probability studies
Data sellers — cleaned odds/volume feeds = a sellable product

Automation

Hourly schedule → diff → alert when a market's YES moves >5¢. A few cents a day on free credits.

More resources

apify-skills — Agent Skills so Claude Code / Cursor runs this from a prompt: npx skills add g4dr/apify-skills
awesome-web-scraping — curated scraping tools and actors
Related: polymarket-odds-tracker — the Polymarket side of the arb pair

Scrape Polymarket Odds: Build a Prediction Market Tracker (and Find Arbitrage)

g4dr — Tue, 20 Oct 2026 13:32:49 +0000

Prediction markets move billions and their odds are data gold — elections, sports, crypto, weather. Polymarket has an API, but it's rate-limited and incomplete for history and order books. Apify actors fill the gap: live odds, whale trades, and Kalshi-vs-Polymarket arbitrage detection.

🤖 Try it live — free Poe bots that generate the exact pipeline for your use case: Lead List Generator · Job Postings Miner · Amazon Product Miner · SERP Rank Tracker · Social Lead Finder

The stack

Apify — dedicated prediction-market actors, some with 800+ users. Free monthly credits.
Python — odds → CSV, alerts, dashboards.

Step 1 — Token

Create an Apify account → Settings → Integrations → API token.

Step 2 — Pull live market odds

curl -X POST "https://api.apify.com/v2/acts/scrapemint~polymarket-market-monitor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"category": "politics", "maxItems": 100}'

Fields per market: question, outcomes (YES/NO prices), volume, liquidity, endDate, url.

Step 3 — Detect mispriced markets

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("scrapemint/polymarket-market-monitor").call(run_input={
    "category": "crypto", "maxItems": 200
})

for m in client.dataset(run["defaultDatasetId"]).iterate_items():
    prices = [o.get("price", 0) for o in m.get("outcomes", [])]
    # binary market where YES+NO != $1 = inefficiency
    if prices and abs(sum(prices) - 1.0) > 0.03:
        print("GAP:", m["question"][:70], "| prices:", prices,
              "| vol:", m.get("volume"))

The arb play

congism/polymarket-kalshi-arb-finder (800+ users — the proven demand) compares the same event on both platforms. If Kalshi says YES at $0.52 and Polymarket says NO at $0.45, buying both locks ~3% risk-free. The actor outputs the pairs; you decide.

Whale watching

jy-labs/polymarket-whale-tracker surfaces the largest positions — when a wallet that called the last three elections moves $50k, that's a signal worth having before the odds move.

Who uses this

Traders — arb alerts, momentum on whale entries.
Journalists/analysts — prediction odds beat polls for breaking stories.
Data sellers — cleaned odds history = a sellable dataset.

Schedule the monitor hourly → diff snapshots → odds-movement alerts. Cost: cents.

More tools: awesome-web-scraping. Skills: npx skills add g4dr/apify-skills (repo).

Which markets are you tracking — politics, crypto, or sports?

Best Web Scraping Tools in 2026: Apify vs Bright Data vs ScraperAPI (Honest Comparison)

g4dr — Tue, 20 Oct 2026 08:22:09 +0000

Picking a scraping platform is a commitment — you'll build pipelines on it, so switching costs real time. After running lead-gen, price-monitoring, and SERP pipelines on all three, here's the honest breakdown.

TL;DR

Apify — best all-rounder: 4,000+ ready-made actors, free tier, agent-friendly. Where I'd start.
Bright Data — best raw proxy network and enterprise compliance. Pricier, heavier.
ScraperAPI — simplest "give me HTML" endpoint. Good for one-off projects, thin on coverage.

Apify

Model: actor marketplace + platform credits. You rent pre-built scrapers ("actors") or run your own on their infra.

Strengths

4,000+ maintained actors — Google Maps, Amazon, LinkedIn, Instagram, SERP... someone already built it
Free tier with monthly credits — genuinely enough to test real pipelines
Built-in scheduling, datasets, webhooks, proxies included
Best fit for AI agents — actors are basically APIs with documented inputs

Weaknesses

Credit pricing can confuse beginners (costs vary per actor)
Heavy custom crawls need some platform learning

Verdict: if you need leads, products, reviews, or SERP data this week, Apify's free tier gets you there fastest. It's what I use for the pipelines below.

Bright Data

Model: premium proxy network + dedicated scraper APIs + datasets.

Strengths

Largest residential/mobile proxy pool — unmatched geo coverage
Enterprise-grade compliance documentation (KYC, legal review)
Web Unlocker handles brutal targets

Weaknesses

Pricing is enterprise-oriented — minimum commits sting for solo builders
More moving pieces; steeper setup
Overkill if your target isn't heavily protected

Verdict: the serious option when you're scraping at volume against hard targets, or when legal/compliance sign-off matters.

ScraperAPI

Model: single endpoint — send a URL, get HTML. They rotate proxies and render JS for you.

Strengths

Dead simple — one API call
Predictable per-request pricing
Quick to bolt onto existing code

Weaknesses

You still write and maintain the parsing logic
No ready-made extractors; weaker for JS-heavy marketplaces
Costs climb on protected targets

Verdict: great for DIY scrapers who just want the unblocking layer. Less great if you want structured data out-of-the-box.

Head-to-head

	Apify	Bright Data	ScraperAPI
Ready-made scrapers	4,000+ actors	API per site	none
Free tier	✅ monthly credits	trial	trial
Proxy network	included	best-in-class	included
Fit for AI agents	✅ skills + API	partial	partial
Best for	leads, ecommerce, social, SERP	enterprise scale	DIY unblocking

What I'd pick

Solo dev / lead-gen / affiliate projects: Apify — the marketplace saves weeks, and the free tier is real.

Enterprise with compliance requirements: Bright Data.

You already have parsers and just need unblocking: ScraperAPI.

Try it yourself

Concrete pipelines I run on Apify, all documented with copy-paste code:

Scrape Google Maps leads — phones, emails, websites at scale
Scrape job postings — hiring signals as sales leads
Amazon affiliate feed — live prices for affiliate sites
Google SERP tracker — rank tracking without $100/mo SaaS
Website → markdown for RAG — LLM-ready pipelines

More tools, proxies, and niche actors in awesome-web-scraping, plus agent skills that run these pipelines from a prompt.

What are you scraping? Comments open.

Build an Amazon Affiliate Feed with Live Prices in 2026

g4dr — Tue, 20 Oct 2026 07:29:28 +0000

Static prices kill affiliate conversions. A "best standing desks" table that says $299 while Amazon shows $349? Your reader bounces. Live data converts — here's how to build a self-updating product feed with Apify actors.

🤖 Try it live — chat with a Poe bot that generates the exact pipeline for your niche: Amazon Product Miner · Lead List Generator · Job Postings Miner · SERP Rank Tracker · Social Lead Finder

The stack

Apify — apify/amazon-crawler handles Amazon's anti-bot wall. Free credits to start.
Nightly schedule — fresh prices every morning, zero maintenance.

Step 1 — Token

Create an Apify account → Settings → Integrations → API token → export APIFY_TOKEN=...

Step 2 — Scrape ASINs in batch

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
asins = ["B0ABC12345", "B0DEF67890", "B0GHI11223"]

run = client.actor("apify/amazon-crawler").call(run_input={
    "productUrls": [{"url": f"https://www.amazon.com/dp/{a}"} for a in asins],
    "proxyCountry": "AUTO_SELECT_PROXY_COUNTRY",
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["asin"], item["title"], item.get("price"), item.get("inStock"))

Fields: asin, title, price, listPrice, stars, reviewsCount, inStock, thumbnailImage, url.

Step 3 — Scheduled snapshots → live table

import json, os

DB = "prices.json"
current = {}
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    current[item["asin"]] = {
        "price": item.get("price"),
        "stock": item.get("inStock"),
        "stars": item.get("stars"),
    }

old = json.load(open(DB)) if os.path.exists(DB) else {}
for asin, p in current.items():
    if asin in old and p["price"] and old[asin]["price"]:
        drop = (old[asin]["price"] - p["price"]) / old[asin]["price"]
        if drop > 0.10:
            print(f"PRICE DROP {asin}: {old[asin]['price']} -> {p['price']}")

json.dump(current, open(DB, "w"))

Put the actor on a daily schedule in Apify Console → this script runs via cron/Actions → your comparison pages always show today's price.

Step 4 — Reviews for content angles

junglee/amazon-reviews-scraper with filterByRating: "critical" pulls the worst reviews first — exactly what you need for "pros & cons" sections that readers trust:

curl -X POST "https://api.apify.com/v2/acts/junglee~amazon-reviews-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"productUrls": [{"url": "https://www.amazon.com/dp/B0EXAMPLE"}], "maxReviews": 100, "filterByRating": "critical"}'

Pro tips

proxyCountry must match the marketplace (US → .com, DE → .de) or data comes back wrong/blocked.
Batch ASINs in one run; re-run only failures.
Pair prices with your Associates tag in the outbound URL — ?tag=yourtag-20.

More tools on awesome-web-scraping. Agent users: npx skills add g4dr/apify-skills — the apify-amazon-products skill does this end-to-end (repo).

What niche are you building a feed for?

Scrape Job Postings at Scale: Hiring Signals Are Buying Signals

g4dr — Tue, 20 Oct 2026 07:24:28 +0000

Companies posting jobs are revealing exactly what they need — in real time. A company hiring "SEO specialist" needs SEO help. Five sales openings mean they want pipeline. Hiring a "migration engineer"? They need a dev agency.

Here's how to turn public job boards into a lead-gen machine using Apify actors — no scraper maintenance, no proxy bills.

🤖 Try it live — chat with a Poe bot that generates the exact pipeline for your search: Job Postings Miner · Lead List Generator · Amazon Product Miner · SERP Rank Tracker · Social Lead Finder

The stack

Apify — ready-made actors for Google Jobs, Indeed, and LinkedIn. Free monthly credits included.
Python + apify_client — 20 lines to a clean CSV.

Step 1 — Token

Create an Apify account → Console → Settings → Integrations → copy the API token → export APIFY_TOKEN=...

Step 2 — Run the Google Jobs actor

curl -X POST "https://api.apify.com/v2/acts/apify~google-jobs-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "queries": "remote react developer",
    "countryCode": "us",
    "maxPagesPerQuery": 2
  }'

Output fields: title, companyName, location, salary, description, applyUrl, postedAt.

Step 3 — Hiring-signal filter

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("apify/google-jobs-scraper").call(run_input={
    "queries": "seo specialist\ncontent writer\nppc manager",
    "countryCode": "us",
    "maxPagesPerQuery": 1,
})

signals = {}
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    company = job.get("companyName", "")
    signals.setdefault(company, []).append(job.get("title"))

# Companies hiring multiple roles = hot prospects
for c, titles in signals.items():
    if len(titles) >= 2:
        print(f"{c}: {len(titles)} openings -> {titles[:3]}")

The lead-gen play

Schedule the actor daily in Apify Console
Filter new postings per company
Enrich the company domain with vdrmota/contact-info-scraper (emails, socials)
Outreach: "Saw you're hiring for X — we deliver that as a service today"

Companies hiring are companies with budget. You're not cold-calling — you're showing up exactly when the pain is public.

Extras

More scrapers and proxies on my awesome-web-scraping list. And if you use Claude Code/Cursor: npx skills add g4dr/apify-skills teaches your agent this whole pipeline — repo.

Dedupe on (companyName, title) — boards cross-post the same job everywhere. Cap maxPagesPerQuery at 1-3 to control spend.

What niche are you scraping jobs for? Drop it below — I'll share the run input.

How to Scrape Google Maps into a Lead List in 2026 (No Code Required)

g4dr — Mon, 19 Oct 2026 23:43:41 +0000

You need a list of plumbers in Manchester, dentists in Miami, or roofers in Lyon — with phones, emails, and websites. Doing it by hand takes days. Writing a scraper takes hours... until Google rate-limits you into CAPTCHA hell.

Here's the 2026 way: run a maintained Apify actor that handles proxies, pagination, and anti-blocking — then let your AI agent do the plumbing.

🤖 Try it live — chat with a Poe bot that generates the exact pipeline for your niche: Lead List Generator · Job Postings Miner · Amazon Product Miner · SERP Rank Tracker · Social Lead Finder

The stack

Apify — actor marketplace with ready-made Google Maps scrapers. Free tier includes monthly credits — enough to test this end-to-end.
Your AI agent (Claude Code, Cursor, Codex...) — runs the pipeline via a skill that knows the actor inputs.

No proxies to configure. No headless browser tuning.

Step 1 — Get your API token

Create an Apify account (free credits included)
Console → Settings → Integrations → copy your API token
export APIFY_TOKEN=your_token

Step 2 — Run the lead-extraction actor

The lukaskrivka/google-maps-with-contact-details actor doesn't just grab businesses — it visits each website and extracts emails and social profiles automatically:

curl -X POST "https://api.apify.com/v2/acts/lukaskrivka~google-maps-with-contact-details/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "searchStringsArray": ["dentist"],
    "locationQuery": "Miami, FL",
    "maxCrawledPlacesPerSearch": 100,
    "language": "en"
  }'

You get back a JSON array with title, phone, website, emails, address, totalScore, reviewsCount, categoryName.

Step 3 — Clean lead CSV with Python

from apify_client import ApifyClient
import csv

client = ApifyClient("APIFY_TOKEN")
run = client.actor("lukaskrivka/google-maps-with-contact-details").call(run_input={
    "searchStringsArray": ["plumber"],
    "locationQuery": "Austin, TX",
    "maxCrawledPlacesPerSearch": 200,
})

with open("prospects.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["business", "phone", "email", "website", "rating", "reviews"])
    for p in client.dataset(run["defaultDatasetId"]).iterate_items():
        if p.get("phone") or p.get("emails"):
            w.writerow([
                p["title"], p.get("phone", ""),
                (p.get("emails") or [""])[0],
                p.get("website", ""), p.get("totalScore", ""),
                p.get("reviewsCount", ""),
            ])

Dedupe by website or phone, filter for totalScore >= 3.5 — and you have a clean outreach list.

Bonus: let your agent run it for you

I packaged this (plus 7 other scraping patterns — jobs, Amazon, social, SERP) as Agent Skills:

npx skills add g4dr/apify-skills

After installing, tell Claude Code "get me all roofers in Birmingham" — it picks the right actor, builds the run input, and hands you the data. Full repo: g4dr/apify-skills.

For the bigger picture (proxies, headless browsers, niche actors), I maintain an awesome-web-scraping list.

Compliance note

Google Maps data is public business info — but respect GDPR and outreach laws (CAN-SPAM, opt-outs). Scrape public data, don't republish personal info, and honor platform ToS.

Questions? Drop them in the comments — happy to share run inputs for specific niches.