DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Scraping Kickstarter: goal_usd is a derived number, not a given one

Quick answer

Kickstarter's own listing API doesn't hand you a campaign's funding goal in USD when the campaign isn't priced in USD — it hands you the raw goal in whatever currency the creator picked, plus a separately-converted pledged total. The Kickstarter Campaigns Scraper reconstructs goal_usd from that mismatch rather than pretending Kickstarter ships a clean number, and returns 30 typed fields per campaign at $2.40 per 1,000 results plus a $0.02 run-start charge — $2.42 for 1,000 campaigns.

goal_usd for a non-USD campaign is a derived number, not a given one 💱

Kickstarter's discover/advanced.json listing payload gives you goal and pledged in the campaign's native currency, plus a separately-converted converted_pledged_amount in USD. There's no converted_goal_amount field. Kickstarter converts what people gave, not what the creator asked for.

So for any non-USD campaign we back into the goal's USD value using the ratio Kickstarter already gave us for the pledged amount: goal_usd = goal * (converted_pledged_amount / pledged) — the same implied exchange rate Kickstarter used for the pledge total, applied to the goal.

One honest edge case worth knowing before you build on it: a campaign that just launched, in a non-USD currency, with zero backers, has pledged == 0 — no ratio to derive from. goal_usd then falls back to the raw native-currency number, unconverted. currency always ships raw alongside it, so check it before trusting cross-campaign USD comparisons on day-one campaigns. pledged_usd never has this problem — it's Kickstarter's own usd_pledged field.

percent_funded follows the same "trust Kickstarter's own number first" rule: when the listing payload includes its own percent_funded, we use it verbatim; only when it's absent do we compute pledged / goal * 100 ourselves.

Turning off reward-tier enrichment doesn't just skip rewards — it nulls three other fields too ⚙️

includeRewardTiers defaults to true, and it's not a single-field toggle. Kickstarter's listing endpoint doesn't carry reward tiers, comment counts, update counts, or tags at all — those four only exist on a campaign's detail page, which costs a second HTTP request per campaign and roughly doubles your run's request volume.

Set includeRewardTiers to false for faster, cheaper runs — but reward_tiers comes back null, and comments_count, updates_count, and tags come back null / [] on every row, not just missing. If your pipeline expects a comment count for engagement scoring, that default matters more than the flag's name alone suggests.

reward_tiers is a list because backer distribution is the actual signal 🎯

backers_count on the top-level row tells you how many people backed a campaign in total. It doesn't tell you what they bought. reward_tiers is a list[RewardTier], and each tier carries its own backers_count — the number who picked that specific pledge level — alongside minimum_usd, shipping_enabled, and available.

Flatten that list to a count and you lose the useful part: a campaign funded by a thousand people at the $10 tier and one funded by fifty people at the $500 tier tell two different stories about the product and audience, even at identical total pledged_usd. Tier pricing gets the same USD treatment as the campaign goal, too — minimum_usd prefers Kickstarter's converted_minimum and falls back to raw minimum when a conversion isn't available.

What we handle for you 🛡️

  • We rotate browser fingerprintscurl-cffi impersonation cycling through Chrome, Firefox, and Safari TLS profiles on every retry.
  • We warm the session first. One cookie-collecting GET against Kickstarter's discover page per session, before any listing or detail request goes out.
  • We retry with exponential backoff on 403 / 408 / 429 / 503, up to 5 attempts per page, honoring Retry-After, rotating browser profile and proxy session on every retry.
  • We isolate detail-page failures. Reward-tier enrichment is best-effort — a campaign detail page that fails after retries still ships its listing fields, with enrichment fields left null and a warning logged. One bad detail page never fails the run.
  • We keep the dataset clean — Pydantic-validated rows (extra="forbid"), ISO-8601 timestamps, JSON / CSV / Excel export from the Apify Console.
  • You pay only for results that land — no data, no charge, beyond the small run-start fee.

Full output schema 📦

Thirty fields per campaign, plus up to eight more per reward tier when includeRewardTiers is on:

Field Type Notes
campaign_id int Numeric project ID
name / blurb / slug string Title, one-line pitch, URL slug
state enum live / successful / failed / canceled / suspended
goal_usd / pledged_usd float Normalized to USD (see above)
percent_funded float Kickstarter's own value, else pledged/goal*100
backers_count int Total backers
currency / country string Native currency + country codes
deadline / created_at / launched_at string ISO-8601
staff_pick bool Kickstarter staff feature flag
creator_id / creator_name int / string Creator identity
location_name / location_state / location_country string | null When disclosed
category_id / category_name / category_parent_name int / string / string | null Parent set for subcategories
project_url / image_url string / string | null Canonical URL + cover photo
comments_count / updates_count int | null Detail-page only; null when reward tiers not fetched
tags list[string] Detail-page only; empty when not fetched
reward_tiers list[RewardTier] \ null
scraped_at string ISO-8601 UTC

Who this is for

Category and trend trackingcategory_name, percent_funded, and state across a category slug show which niches actually convert backers, not just launch.

Creator scouting — filter on staff_pick and most_funded/most_backed sort to find campaigns worth reaching out to before a launch cools off.

Reward-strategy benchmarkingreward_tiers across competing campaigns shows how rivals price and structure pledge levels, not just how much they raised.

Crowdfunding market researchstate, goal_usd, pledged_usd, and backers_count across a search term or category, for a dataset of what succeeds versus fails at what funding scale.

Frequently asked questions

Is goal_usd always an exact USD conversion?
For USD-native campaigns, yes — the raw goal. For non-USD campaigns with at least one backer, it's derived from Kickstarter's own pledged-to-USD ratio. For a non-USD campaign with zero backers, it falls back to the unconverted native-currency figure — check currency on those rows.

What happens if I turn off includeRewardTiers?
reward_tiers ships null, comments_count/updates_count ship null, and tags ships an empty list on every row — and the run roughly halves its request volume, since all four come from a second per-campaign detail fetch.

Does percent_funded ever come straight from Kickstarter instead of being calculated?
Yes, when the listing payload includes it. We only compute pledged/goal*100 ourselves when Kickstarter's payload omits it.

Can I combine a category filter with a search term?
Yes — category and searchTerm are independent inputs and combine as an AND filter.

What's the cap per run?
maxItems accepts 1–2,000. For more, run it again with a different sort or filter.

Try it

Live on the Apify Store: Kickstarter Campaigns Scraper.

Point it at a category, a search term, or both, and get back typed campaign rows with USD-normalized funding figures and per-tier reward data when you need it. Pay-per-event, no subscription, no card required to try the free run.


Built by Devil Scrapes — we build scrapers for the targets that fight back.

Top comments (0)