Hipcamp Scraper: Three Listing Types That Never Share a Schema

#webscraping #python #api #datascience

Quick answer

Hipcamp's regional discover pages hold three different inventory types — private land, public campgrounds, and public land — and each arrives from the page's own data in a different shape, under different field names. The Hipcamp Campsite Listings Scraper reads all three out of the page's Next.js bootstrap JSON and normalizes them into one row shape — with USD nightly price, rating, geo-coordinates, and accommodation types — at $1.25 per 1,000 rows.

Why doesn't a Hipcamp scraper just parse the listing cards? 🏕️

Hipcamp is a Next.js app, and the state that renders a discover page's cards is sitting in a <script id="__NEXT_DATA__"> block on the page itself — full structured JSON, not markup to reverse-engineer. That part is the easy half.

The other half is that this JSON isn't one array of listings. It's three: discoverHipcampLands.nodes (privately hosted land), publicCampgrounds.nodes (public campgrounds), and publicLands.nodes (public land). Each array uses its own field names for the same concept — a private-land node's name lives at name, a public campground's sometimes only at parkName; the price block is minPricePerNight on one path and pricePerNight on another; isStarHost only exists on private-land nodes at all. Treat these as one array and you'll either miss two-thirds of a region's inventory or crash the first time a field you assumed was universal isn't there.

This Actor walks all three arrays, in a fixed order, and maps each into the same ResultRow shape — with a listing_type field telling you which of the three it came from — so you get one table instead of three incompatible ones.

What running a region looks like

{
  "regions": [{ "state": "Texas", "city": "Austin" }],
  "maxListingsPerRegion": 20,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("DevilScrapes/hipcamp-campsite-listings-scraper").call(
    run_input={"regions": [{"state": "Colorado"}, {"state": "Oregon", "city": "Bend"}]}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["listing_type"], item["name"], item["price_per_night_usd"])

And one resolved row:

{
  "listing_id": "7rvhj6kr",
  "listing_type": "private_land",
  "name": "Texas Urban Hideout",
  "url": "https://www.hipcamp.com/en-US/land/texas-urban-hideout-7rvhj6kr",
  "city": "Austin",
  "state": "Texas",
  "state_abbr": "TX",
  "latitude": 30.2672,
  "longitude": -97.7431,
  "price_per_night_usd": 72.0,
  "rating_percentage": 98.0,
  "rating_count": 41,
  "total_campsites_count": 3,
  "accommodation_types": ["house"],
  "is_star_host": true,
  "image_url": "https://hipcamp-res.cloudinary.com/images/v1721137420/land-photos/texas-urban-hideout/cover.jpg",
  "scraped_at": "2026-08-24T18:03:11Z"
}

Why is the proxy pinned to the US? 🌐

Hipcamp's inventory is US-only, and a residential proxy exit that lands outside the US — or even in the wrong part of it — can still return a 200 OK with plausible-looking listings that just aren't the right ones. That's a worse failure mode than a block: a block is visible, a wrong-but-plausible dataset isn't, until someone downstream notices the city names don't match. This Actor's default proxy configuration pins the exit country to US specifically to close that gap, rather than leaving country selection to chance.

What we handle 🛡️

Browser fingerprint rotation — curl-cffi impersonates a real Chrome, Firefox, or Safari TLS handshake, rotated on every retry.
Retries with backoff on 408 / 429 / 503, up to 5 attempts, 2 seconds doubling to a 30-second cap, honoring Retry-After.
A fresh session on every block signal. A 403 or 451 doesn't just retry — it forces a different browser profile and a fresh proxy session before the next attempt, not the same fingerprint knocking again.
Region-level fault isolation. One region that can't be reached after retries is skipped and logged; it doesn't fail the other nine in the same run.
Deduped, typed rows across all three listing types, keyed on listing_id and url so the same place can't show up twice even if it were to appear in more than one node array.

What people build with this 💡

Hipcamp hosts benchmarking their own nightly rate against comparable listings in the same region before adjusting pricing.
Glamping and RV-park operators sizing nightly rates and amenity mix across a market they're considering entering.
Travel-data aggregators feeding normalized US outdoor-stay inventory into a broader lodging dataset.
Outdoor-recreation analysts tracking listing supply and pricing across multiple US states in one run, up to 10 regions at a time.

Frequently asked questions

What does 1,000 rows cost?
$1.25 — a $0.05 run-start charge plus $0.0012 per unique listing.

Does this cover a region's entire inventory?
No. Hipcamp's discover page is curated to roughly 40-45 combined listings per region, even when the site reports a much larger total. There's no pagination parameter on this page — broaden coverage by adding more regions (up to 10 per run), not by paging one region deeper.

Do I need a Hipcamp account or API key?
No — this reads Hipcamp's own publicly served discover pages, no login required.

What regions can I search?
Any US state, optionally narrowed to a city — free-text names like "Texas" or "Los Angeles" work directly. Hipcamp's inventory is US-only.

What happens if a region has no listings?
The run still succeeds with zero rows for that region and a status message naming what was searched. A real search that matched nothing is a success, not a failure.

Try it

Live on the Apify Store: Hipcamp Campsite Listings Scraper. Apify gives every new account $5 of free credit, no card required to try it.

Built by Devil Scrapes — three inventory shapes in, one clean table out.

DEV Community