Building a TechCrunch Scraper with Python: RSS First, HTML Second

TL;DR

• Use TechCrunch's public feed for recent article discovery when it satisfies the project; it is simpler and more stable than crawling listing pages.
• Use article HTML only for fields that the feed does not supply, and keep the original URL and publication metadata.
• Prefer semantic article markup and JSON-LD over presentation classes.
• Respect current terms, robots directives, copyright, and rate limits; store only the content needed for the authorized use.
• Make the pipeline idempotent with canonical URLs, publication identifiers, retrieval timestamps, and content hashes.

Why I approached it this way

For this kind of news workflow, I start with the public feed and fetch article pages only when the feed does not contain a required field. That keeps discovery simple and reduces both traffic and parser maintenance.

What is a TechCrunch scraper and why would you build one?

A TechCrunch scraper is a bounded pipeline that discovers permitted public articles and extracts selected metadata for monitoring, search, or research. For many projects, a feed reader plus selective article parsing is safer and simpler than crawling category pages.

News text is copyrighted, so store the minimum required fields and links unless the intended use and license permit more.

What do you need before you start?

Install feedparser, httpx, beautifulsoup4, and trafilatura. Define canonical_url, title, authors, published_at, section, summary, retrieved_at, and content_hash. Decide whether full text is genuinely necessary.

python -m venv .venv
source .venv/bin/activate
python -m pip install feedparser httpx beautifulsoup4 trafilatura

Check the current TechCrunch feed, terms, and robots file before execution. Use a descriptive user agent and a low, bounded request rate.

How does the TechCrunch collection workflow work?

The feed supplies recent entries and stable article links. Each accepted link can be normalized, checked against the document registry, and fetched only when new or changed. Article HTML may expose JSON-LD and semantic markup for title, author, and timestamps. Main-content extraction is a separate step from discovery.

The practical workflow

Method 1: Discover articles through RSS

Step 1: Parse the feed

import feedparser

feed = feedparser.parse("https://techcrunch.com/feed/")
if feed.bozo:
    raise RuntimeError(f"Feed parse error: {feed.bozo_exception}")

entries = [{"title": e.get("title"), "url": e.get("link"), "published": e.get("published")}
           for e in feed.entries]

Step 2: Validate each entry

Require an HTTPS TechCrunch URL and a non-empty title. Reject duplicate canonical URLs before page retrieval.

Step 3: Save a discovery checkpoint

Store the feed retrieval time and entry identifier. Do not assume feed order alone is stable enough for recovery.

Method 2: Extract metadata from article HTML

Step 1: Fetch one new article

import httpx

headers = {"User-Agent": "NewsResearch/1.0 [e-mail]"}
r = httpx.get(entries[0]["url"], headers=headers, timeout=20, follow_redirects=True)
r.raise_for_status()
html = r.text

Step 2: Parse JSON-LD

Use Beautiful Soup to locate application/ld+json and select NewsArticle or Article objects. Preserve headline, author objects, date fields, and canonical URL without inventing missing values.

Step 3: Extract only required content

Use a verified article selector or Trafilatura for main text. Record extraction method and parser version so later changes can be reproduced. The Trafilatura documentation describes current extraction controls.

Method 3: Use the WordPress REST surface when available and permitted

Step 1: Confirm the endpoint from the site

Do not assume a WordPress endpoint is public or stable. Discover it through official page metadata or documentation and verify that the intended use is allowed.

Step 2: Request narrow fields and bounded pages

Use _fields when supported, a small per_page, and explicit page limits. The WordPress REST API handbook documents general interfaces, but the site controls what it exposes.

Step 3: Treat rendered HTML as untrusted input

Sanitize content before display and retain source attribution. REST output does not grant reuse rights beyond the applicable license and terms.

Method 4: Use a managed collection layer

Step 1: Submit only approved article URLs

Keep feed discovery as the source of truth and use bounded page jobs.

Step 2: Validate body-level success

Check final URL, title, publication evidence, language, and content completeness.

Step 3: Normalize into the same registry

The managed path and HTTP path should produce identical internal metadata. Provider fields must not leak into business keys.

Why is the TechCrunch scraper missing articles or text?

Feeds can be truncated, delayed, or limited to recent items. Article templates can change, content can be embedded, and a response can be an error or consent state. Compare feed IDs, canonical URLs, JSON-LD, final URL, and accepted-field coverage. Do not solve a missing field by collecting unrelated page regions.

The Robots Exclusion Protocol is a technical policy signal, not a content license.

How do you schedule and deduplicate the pipeline?

Poll the feed at a respectful interval, upsert by canonical URL or stable entry ID, and fetch only unseen or explicitly refreshed pages. Compute a normalized content hash to distinguish metadata edits from new documents. Retry transient network failures with a cap and send parse changes to review.

What I would keep in production

I would keep this pipeline intentionally small: RSS for discovery, structured metadata for identity, selective HTML parsing for missing fields, and an idempotent store keyed by canonical URL or another stable identifier. Re-check the site’s current policies before scheduling it.

FAQ

Q: Does TechCrunch provide an RSS feed?

TechCrunch exposes a public feed at the URL used in this tutorial, but its scope and behavior should be rechecked before production use.

Q: Should a TechCrunch scraper store full article text?

Only when the project has a valid need and applicable rights. Metadata, summaries, and canonical links are often sufficient for monitoring.

Q: How do you avoid duplicate TechCrunch articles?

Use canonical URLs or stable feed identifiers, normalize conservative tracking parameters, and perform idempotent upserts.

Q: Why does the feed contain fewer articles than the site?

Feeds commonly expose a recent subset rather than a complete archive. Use only approved discovery paths and do not assume the feed is exhaustive.

Q: When is a browser required?

A browser is required only when the permitted field depends on client rendering or interaction and cannot be obtained from the feed, structured data, or initial HTML.