DEV Community: Omar Eldeeb

How to Scrape Reddit Without the API (After the 2023 Price Changes)

Omar Eldeeb — Sun, 31 May 2026 16:11:10 +0000

If you've landed here, you already know the backstory: in 2023 Reddit's API went from free-and-generous to metered-and-expensive, third-party apps shut down, and a lot of data pipelines broke overnight. So the practical question for developers and data folks is no longer "should I use the API?" but how to scrape Reddit without the API at all — cleanly, legally-aware, and without burning hours on requests that silently return 403.

This article walks through what genuinely works in 2026, what looks like it works but doesn't, and the constraints you'll hit no matter which path you choose. The code paths you can verify yourself in a terminal; the rate limits, the ~250 search cap and the Pushshift/terms details are drawn from Reddit's docs and widely-reported community experience (links where it matters), and real-world enforcement is more erratic than any documented figure.

The thing everyone tries first (and why it fails)

The classic "no API" trick is appending .json to any Reddit URL:

https://www.reddit.com/r/programming/.json
https://www.reddit.com/r/programming/comments/<id>/.json

This is a real, undocumented JSON view of the page. The problem is where you call it from.

From a browser (client-side JS): it's CORS-blocked. Reddit doesn't send Access-Control-Allow-Origin for these endpoints, so fetch() from your web app throws before you ever see data. No amount of header tweaking fixes CORS from the browser — it's enforced by the browser, not by your code.
From a datacenter server (AWS, GCP, a VPS): the .json endpoints increasingly return HTTP 403 from datacenter IP ranges. Reddit tightened this after the API changes specifically to stop the "just hit .json from a Lambda" pattern.

So the .json approach dies in the two places people most want to use it: the browser and cheap cloud servers. You can sometimes get it to work from a residential IP with a sane User-Agent, but it's fragile and rate-limited, and it is not a foundation you want a pipeline on.

What actually works: old.reddit.com server-rendered HTML

The most reliable no-API path is the old Reddit interface, old.reddit.com. Unlike the modern React SPA (which hydrates data client-side and is painful to parse), old Reddit ships fully server-rendered HTML, cookie-free. You request a page, you get the listing already in the markup.

Two important nuances I want to be honest about:

Subreddit listings and user-profile pages parse fine and often work even from datacenter IPs. These are the easy wins.
Search results and comment threads are stricter — in practice you'll need residential IPs to fetch them reliably, because Reddit rate-limits and challenges those routes harder.

Here's a minimal, correct example that pulls the front page of a subreddit from old Reddit and extracts post titles and links. It uses requests + BeautifulSoup, with a real User-Agent (Reddit reliably rejects the default python-requests UA):

import requests
from bs4 import BeautifulSoup

HEADERS = {
    # A real, descriptive UA. Reddit blocks the default python-requests UA.
    "User-Agent": "research-bot/1.0 (contact: you@example.com)"
}

def scrape_subreddit(subreddit: str):
    url = f"https://old.reddit.com/r/{subreddit}/"
    resp = requests.get(url, headers=HEADERS, timeout=20)
    resp.raise_for_status()  # 403/429 will surface here

    soup = BeautifulSoup(resp.text, "html.parser")
    posts = []
    for thing in soup.select("div.thing[data-fullname]"):
        title_el = thing.select_one("a.title")
        if not title_el:
            continue
        posts.append({
            "id": thing.get("data-fullname"),
            "title": title_el.get_text(strip=True),
            "permalink": thing.get("data-permalink"),
            "score": thing.get("data-score"),
            "author": thing.get("data-author"),
            "subreddit": thing.get("data-subreddit"),
        })
    return posts

if __name__ == "__main__":
    for p in scrape_subreddit("programming")[:5]:
        print(p["score"], "-", p["title"])

The div.thing element carries most of what you need as data-* attributes — data-fullname (the post ID like t3_abc123), data-score, data-author, data-permalink. That's why old Reddit is so pleasant: the structure is stable and the data is right there in attributes instead of buried in a hydration blob.

Pagination

Old Reddit paginates with a ?count=25&after=<fullname> query string. The "next" button's href gives you the URL directly:

next_btn = soup.select_one("span.next-button a")
next_url = next_btn["href"] if next_btn else None

Follow that link to walk listings. Add a polite delay (1–2 seconds) between requests and reuse a requests.Session so connections are kept alive.

The hard limits you cannot engineer around

Before you build anything ambitious, internalize these constraints. They're properties of Reddit, not of your scraper.

Search caps at ~250 results (observed). In practice Reddit's search — whether via the API or the HTML interface — appears to return roughly the top ~250 matches for a query and then stops, with no deep pagination past that. It's widely-observed behavior rather than an officially documented number, but it's consistent enough to plan around. If your use case is "give me every post ever mentioning X," search alone will not deliver it.

Comment indexing is weak. Reddit search indexes post titles and bodies far better than it indexes comments. A keyword that lives only in comment threads will frequently not surface in search at all. This trips up sentiment and brand-monitoring projects constantly.

Pushshift is gone for you (probably). Pushshift used to be the answer for historical, full-text, deep Reddit search. Since 2023 it has been restricted to verified subreddit moderators. Unless you're a mod with approved access, treat Pushshift as unavailable.

The official Data API is metered and commercial-use-restricted. For completeness: the official route allows roughly 100 requests/minute with OAuth (about 10/minute unauthenticated), and Reddit's terms restrict commercial use without a separate licensing/paid agreement. So even if you go "official," you're capped and legally boxed in for anything revenue-adjacent.

Put together: there is no magic endpoint that gives you unlimited, deep, full-text Reddit history for free. Anyone who tells you otherwise is selling something or about to get blocked.

A sane workflow: build the query first, then export

A mistake I see often is jumping straight to code, then discovering the query was wrong after burning a bunch of requests. Because search is capped at ~250 results and comment indexing is weak, the precision of your query matters more than the speed of your scraper.

So the workflow I'd recommend:

Compose and preview the query before you fetch anything. A free, no-signup helper for this is the Reddit Search Builder. It lets you assemble a precise Reddit query (subreddit filters, time windows, sort, exact-phrase syntax) and previews the result schema so you know exactly which fields you'll get back before committing to a run. Getting the query right up front is the single biggest lever given the 250-result ceiling.
Run small from a residential context to validate the HTML parser against real markup (selectors drift; verify before scaling).
Scale the export with proper IP rotation. This is where a DIY scraper gets painful — you need datacenter IPs for cheap subreddit/user listings, residential IPs for search and comments, retry/backoff on 403/429, and dedup across pages. Maintaining that yourself is a real project.

If you'd rather not run and maintain the proxy + retry + parsing stack, the Reddit Scraper Pro actor on Apify is the do-this-at-scale option I built around exactly the constraints above (disclosure: it's my actor). It runs five modes (subreddit posts, search, comment threads, user profiles, and a monitor mode) and handles datacenter-first with residential fallback so the easy routes stay cheap and the hard routes still work, with retry/backoff on 403/429 to keep success rates high. Pricing is $0.0025 per post with 10 free per run, so you can validate output on a real query before spending anything. It's the same old.reddit.com strategy described here, just with the IP rotation, backoff, and schema normalization already wired up.

A quick decision guide

Need a few subreddit or user listings, occasionally? The old.reddit.com + BeautifulSoup snippet above is genuinely enough. Run it from a residential IP, be polite, done.
Need search results or comment trees at any volume? Plan for residential IPs and accept the ~250-result search ceiling. Build your query carefully first.
Need scale, reliability, or scheduled monitoring? Either invest serious time in a rotating-proxy pipeline, or hand it to a managed actor and spend your time on the analysis instead of the plumbing.

One honest closing note

Whatever path you pick, respect the source. Reddit's terms prohibit unauthorized commercial use of its data, the official API is rate-limited for a reason, and aggressive scraping gets IPs and projects banned. Scrape conservatively, identify your bot honestly in the User-Agent, cache what you fetch so you don't re-hammer the same pages, and don't republish content in ways that violate users' or Reddit's rights. "Without the API" is a technical choice — it isn't a license to ignore the terms behind it. Build accordingly, and your pipeline will outlast the next round of changes.

How to Export Google Patents to CSV (Honest Guide to Every Real Path)

Omar Eldeeb — Sun, 31 May 2026 16:02:23 +0000

If you've ever needed to pull a few thousand patents into a spreadsheet — every filing by a competitor, every patent citing your portfolio, the legal status of an entire technology cluster — you've probably searched how to export Google Patents to CSV and found a maze of half-answers. This guide cuts through it. I'll show you exactly what works, what's capped, and what's quietly impossible, with verified facts and a runnable example.

Let me start with the single most important thing, because it shapes every decision below:

Google Patents has no public REST API. There is no documented, supported HTTP endpoint you can hit to query patents programmatically. This is the root cause of nearly every frustration people run into.

With that established, here are the three real paths, from simplest to most powerful.

Path 1: The built-in CSV download (fast, but capped at 1,000)

Google Patents does have an export button, and for small jobs it's perfect. Run a search at patents.google.com, then look for the Download (CSV) link near the results.

It works. But it has a hard ceiling:

The built-in CSV export returns only the top 1,000 results. If your query matches 40,000 patents, you get the first 1,000 by relevance and nothing more.

The exported columns are also fairly thin — typically id, title, assignee, inventor, priority/filing/publication/grant dates, and result link. You do not get the abstract, the claims text, the full citation graph, or detailed legal-status events. For a quick competitor snapshot, this is fine. For analysis, it's a teaser.

A practical tip: tighten your query so the 1,000 you get are the 1,000 you want. Combine fields:

(assignee:"Tesla") AND (inventor:"Straubel") AND before:priority:20200101

Google Patents supports field-qualified search — assignee:, inventor:, before:/after: with priority/filing/publication, country codes, CPC classifications, and free text. Narrowing first is the difference between a useful 1,000-row export and a useless one.

Path 2: BigQuery — the only Google-supported bulk path

When 1,000 rows isn't enough, there is exactly one path Google itself supports for bulk patent data, and it's a good one: the patents-public-data dataset on Google BigQuery.

This is a genuinely first-class resource. The main table, patents-public-data.patents.publications, contains bibliographic information on tens of millions of patent publications worldwide, with structured fields for assignees, inventors, titles, abstracts, claims, CPC/IPC classifications, citations, and priority/filing/publication dates — far richer than the CSV button.

Two things to know before you commit:

It requires SQL. There's no point-and-click here. You write queries.
Pricing is generous but real. On-demand BigQuery gives you the first 1 TiB of query data processed free every month; beyond that, queries are billed per TiB scanned (Google has historically documented the patents dataset access at $5/TB, and current general on-demand US pricing is $6.25/TiB — check the official BigQuery pricing page for the rate that applies to you). The patents tables are large, so a careless SELECT * can chew through your free tier in a single query. Always select only the columns you need and filter early.

Here's a real, runnable example. It pulls US patents matching an assignee, flattens the repeated fields (titles and assignees are nested arrays in this schema), and writes a clean CSV. You'll need a Google Cloud project and pip install google-cloud-bigquery pandas db-dtypes.

from google.cloud import bigquery

client = bigquery.Client(project="your-gcp-project-id")

# title_localized and assignee_harmonized are REPEATED records, so UNNEST them.
# Filter by country and date FIRST to limit the bytes scanned (and the cost).
query = """
SELECT
  pub.publication_number,
  title.text          AS title,
  assignee.name       AS assignee,
  pub.filing_date,
  pub.publication_date,
  pub.grant_date
FROM `patents-public-data.patents.publications` AS pub,
  UNNEST(pub.title_localized)      AS title,
  UNNEST(pub.assignee_harmonized)  AS assignee
WHERE pub.country_code = 'US'
  AND title.language = 'en'
  AND assignee.name LIKE '%TESLA%'
  AND pub.filing_date BETWEEN 20150101 AND 20231231
"""

# Dry run FIRST — see how many bytes this will scan before you pay a cent.
dry = client.query(query, job_config=bigquery.QueryJobConfig(dry_run=True))
print(f"This query will scan {dry.total_bytes_processed / 1e9:.2f} GB")

# If that looks acceptable, run it for real.
df = client.query(query).to_dataframe()
df.to_csv("tesla_patents.csv", index=False)
print(f"Exported {len(df)} rows to tesla_patents.csv")

The dry_run step is the habit that saves your bill. It returns the exact byte count without running the query, so you always know the cost before you spend it. Dates in this dataset are stored as integers in YYYYMMDD form (e.g. 20150101), which trips up newcomers — note the comparison style above.

BigQuery is the right answer for academic analysis, large-scale landscaping, and anything where you control a GCP project and are comfortable with SQL. Its main downsides: the SQL learning curve for the nested schema, and the fact that some legal-status events and full citation context require joining additional tables.

Path 3 (the one people expect to work): browser scraping — and why it doesn't

This is where most tutorials go wrong, so let me be precise.

Google Patents search is powered internally by an XHR endpoint (the one your browser hits as you type a query). The intuitive idea is: "I'll just fetch() that endpoint from a little web page and read the JSON." It feels like it should work. It does not, and here's the exact reason:

The query endpoint does not send a permissive CORS header. A browser running on any other origin cannot read the response — the browser blocks it before your JavaScript ever sees the data.

This isn't a bug you can header-hack around from client-side JS; CORS is enforced by the browser, not the server. So a pure in-browser scraper served from your own domain is a dead end. Combined with "no public REST API," this is why client-side patent tools can only ever build a query and show you a curated sample — a browser on another origin can't read live results, so the fetch has to happen server-side.

To actually fetch results at scale you need a server-side process (your own backend, a cloud function, or a hosted scraper) that makes the request without a browser's CORS enforcement, handles pagination, parses the response, and respects rate limits. That's real work, and it's the gap the two tools below fill.

Tying it together: a builder + a scraper

If you just need to construct a precise query and preview what the data looks like — without writing SQL or standing up a backend — the free Google Patents Search Builder lets you compose searches by assignee, inventor, and keyword and see a real sample of the structured output. Because of the CORS reality above, it's honest about what it is: a query builder with a real sample preview, not a live in-browser scraper. It's a great way to nail your query before you spend BigQuery bytes or kick off a larger run.

When you need the full export — thousands of rows, across 100+ patent offices, with the fields the built-in CSV omits — the Google Patents Intelligence actor on Apify (disclosure: I build it, and the free Search Builder above) runs the live search server-side and returns the citation graph, legal status, and claims count as CSV, JSON, Excel, or an API endpoint. It's the do-this-at-scale option for the cases where the 1,000-row cap bites and you'd rather not maintain SQL pipelines or your own scraping infrastructure.

Which path should you pick?

A quick competitor snapshot, under 1,000 results? Use the built-in CSV button. Narrow your query first.
Large-scale analysis and you know SQL? BigQuery's patents-public-data is the gold standard. Dry-run every query.
Thousands of enriched rows without SQL or servers? A hosted scraper is the pragmatic choice.

A closing note on doing this responsibly: scraping any site, Google Patents included, lives under its Terms of Service and applicable law. For bulk needs, the BigQuery dataset is the explicitly Google-supported route and the cleanest one to stand behind — prefer it when SQL is on the table, and keep request volumes reasonable when you don't. Build the right query once, and the export takes care of itself.

Read Company Hiring Signals From Public Job Board APIs (with code)

Omar Eldeeb — Sun, 31 May 2026 16:02:11 +0000

A company's open roles are the most honest document it publishes. The careers page is marketing; the job board is the budget. If you learn to read company hiring signals straight from the open requisitions, you can infer where a business is investing months before it shows up in a press release.

And the best part for developers: most of the data is sitting behind public, no-auth JSON APIs. The applicant tracking systems (ATS) that power those careers pages — Greenhouse, Lever, Ashby, SmartRecruiters — expose job boards as plain endpoints. You can fetch them, parse them, and classify the role mix yourself.

This article shows you how to do that, with a snippet that actually runs in a browser console.

Why open roles encode strategy

Headcount is the clearest expression of intent a company has. Every requisition is a funded decision someone fought for in a planning meeting. So the mix of roles, not just the count, tells a story:

A wave of Account Executives and Sales Engineers → they have a product that works and are pouring fuel on go-to-market. Likely just raised, or hitting a revenue inflection.
A spike in backend / infra / platform engineers → scaling pains. The thing is growing faster than the architecture can handle.
New "AI", "ML", or "Applied Scientist" titles where there were none → a strategic bet that didn't exist last quarter.
Roles concentrated in a new city or country → geographic expansion. A "Country Manager, Germany" is a market-entry announcement disguised as a job post.
Recruiters and People Ops hiring → they expect to hire a lot soon. Recruiting hires are often a leading indicator of broader expansion.
First Compliance / Legal / Finance leadership → maturing toward a fundraise, audit, or exit.

This is exactly the kind of intelligence that sales teams pay for under the label "hiring intent" or "buying signals." You can derive a useful slice of it yourself.

The data source: public ATS job boards

Greenhouse runs a dedicated read-only API for board content. The shape is dead simple:

GET https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs

The board_token is usually the company's slug — stripe, airbnb, etc. No API key, no OAuth, no header dance. It returns 200 OK with Content-Type: application/json and, crucially for front-end code, Access-Control-Allow-Origin: *. That wildcard CORS header means the request genuinely succeeds from a browser on any origin — you can paste the fetch below straight into DevTools and it works.

Here's the response shape (illustrative values — run it yourself for live data), so you know what you're parsing:

{
  "jobs": [
    {
      "id": 1234567,
      "title": "Account Executive, Enterprise",
      "updated_at": "2026-05-20T16:58:18-04:00",
      "location": { "name": "San Francisco, CA" },
      "absolute_url": "https://example.com/jobs/search?gh_jid=1234567"
    }
  ]
}

Each job gives you title, location.name, updated_at, and a link. That's all you need to map role mix to intent.

Fetch + classify in ~50 lines

Below is a self-contained function. It pulls a board, buckets each role into a category by keyword, and returns a sorted intent profile plus a naive "primary signal." Drop it in your browser console with any Greenhouse board token.

const SIGNALS = {
  sales:       /\b(account executive|ae|sales|business development|bdr|sdr|revenue)\b/i,
  marketing:   /\b(marketing|growth|demand gen|content|brand|seo)\b/i,
  engineering: /\b(engineer|developer|sre|devops|infrastructure|platform|backend|frontend)\b/i,
  ai_ml:       /\b(machine learning|ml engineer|applied scientist|\bai\b|research scientist|nlp)\b/i,
  product:     /\b(product manager|\bpm\b|product designer|ux|ui designer)\b/i,
  recruiting:  /\b(recruiter|talent|people ops|hr business partner)\b/i,
  finance_legal:/\b(finance|accounting|controller|legal|counsel|compliance)\b/i,
  support:     /\b(support|customer success|csm|implementation|onboarding)\b/i,
};

function classify(title) {
  for (const [label, re] of Object.entries(SIGNALS)) {
    if (re.test(title)) return label;
  }
  return "other";
}

async function hiringSignals(boardToken) {
  const url = `https://boards-api.greenhouse.io/v1/boards/${boardToken}/jobs`;
  const res = await fetch(url);
  if (!res.ok) throw new Error(`Board "${boardToken}" returned ${res.status}`);
  const { jobs } = await res.json();

  const counts = {};
  const byCity = {};
  for (const job of jobs) {
    const cat = classify(job.title);
    counts[cat] = (counts[cat] || 0) + 1;
    const city = job.location?.name || "Unknown";
    byCity[city] = (byCity[city] || 0) + 1;
  }

  const profile = Object.entries(counts).sort((a, b) => b[1] - a[1]);
  const topCities = Object.entries(byCity).sort((a, b) => b[1] - a[1]).slice(0, 5);

  return {
    board: boardToken,
    totalRoles: jobs.length,
    roleMix: profile,
    topLocations: topCities,
    primarySignal: profile[0]?.[0],
  };
}

// Try it:
hiringSignals("stripe").then(console.log);

Run it and you get something shaped like this (example numbers — boards change daily, so your run will differ):

{
  board: "stripe",
  totalRoles: 470,
  roleMix: [["engineering", 160], ["sales", 70], ["product", 40], ...],
  topLocations: [["San Francisco, CA", 64], ["New York, NY", 38], ...],
  primarySignal: "engineering"
}

Note primarySignal is just roleMix[0][0] — the highest-count category — and classify() files each title under its first matching pattern, so treat both as a rough first read, not gospel. From there, the interesting analysis isn't the snapshot — it's the delta. Save today's roleMix and diff it next week. A category that jumps from 3 to 18 roles is the signal. A new city appearing in topLocations is the signal. Absolute counts are noisy; changes are where intent lives.

Sharpen the read

A few things to layer on once the basics work:

Weight by recency. Roles with a fresh updated_at reflect current priorities more than ones reposted for months. Filter to roles updated in the last 30 days.
Watch for firsts. The first role in a category (first "Enterprise AE", first "Solutions Architect") often matters more than the tenth. Track which categories crossed from zero.
Seniority skew. A batch of "Head of" / "Director" / "VP" postings signals a layer being built out — usually ahead of an org's scaling phase.
Cross-reference with funding. Sales-and-marketing hiring spikes that line up with a recent raise are the strongest go-to-market-expansion tell.

The regexes above are deliberately simple. Real titles are messy ("Staff Software Engineer, Payments Risk Platform"), and a keyword bucket will misfile some. For anything beyond exploration, an LLM classifier handling each title against your taxonomy is far more robust than brittle patterns — but start with regex to understand your data.

Want to eyeball one company right now?

If you just want to point at a single Greenhouse board and see the role mix without writing code, there's a free browser tool that runs the same idea live: datatooly.xyz/company-hiring-signals (disclosure: I built it, and the Apify actor mentioned later). It fetches the public board client-side (thanks to that wildcard CORS header) and renders the intent breakdown. Good for a quick check on one prospect.

The other ATS platforms (and the hard one)

Greenhouse is the easiest, but it's not alone. Several major ATS platforms expose public job boards:

Endpoint shapes and headers drift over time — test each ATS before depending on it in production.

Lever — https://api.lever.co/v0/postings/{company}?mode=json returns a plain JSON array with text, categories.team, categories.location, and hostedUrl.
Ashby — a public posting API keyed by job-board name, and it also sends Access-Control-Allow-Origin: *.
SmartRecruiters — a public postings endpoint per company.

Each has a slightly different response shape, so you'd normalize them into one schema (title, location, team, updated date, url) before classifying.

Then there's Workday, which is the genuinely hard one. Workday tenants serve postings through a per-tenant CXS endpoint that you have to discover, and pagination is done via POST with an offset body rather than a clean GET — no friendly wildcard CORS, no single base URL. A meaningful share of large enterprises run on Workday, so any "company hiring signals" pipeline that ignores it has a blind spot exactly where the biggest budgets are.

Doing this at scale

Reading one board by hand is a five-minute task. Tracking 25,000+ companies, normalizing four-plus ATS schemas (including the Workday pagination dance), running an AI classifier over messy titles, and diffing week-over-week to fire alerts when a category spikes — that's a data pipeline, not a console snippet.

If you'd rather not build and maintain all of that, the ATS Hiring-Intent Scraper on Apify does the heavy lifting: it pulls across the major ATS platforms, classifies role mix into intent categories, and is built for running on a schedule so you catch the changes rather than just snapshots. Useful if hiring signals feed a sales or research workflow and you need them reliably, not as a one-off.

But for learning the concept and prototyping on a handful of targets, the fetch-and-classify snippet above is all you need — and it's a genuinely fun afternoon of code.

One honest note: these endpoints are public because companies want their jobs found, but they're meant for candidates, not bulk harvesting. Keep request rates polite, cache aggressively, respect each platform's Terms of Service and robots.txt, and don't republish personal data. Read the strategy, not the people.