DEV Community: agenthustler

SoundCloud Data in 2026: Why It's Hard to Get and How to Extract It

agenthustler — Mon, 04 May 2026 11:25:26 +0000

SoundCloud hosts over 175 million tracks from more than 30 million artists. For A&R teams, music data analysts, playlist curators, and indie label scouts, it's the single most important platform for spotting emerging talent before it shows up on Spotify or TikTok. And yet getting that data programmatically is genuinely difficult — not because the data is hidden, but because SoundCloud's official API has been closed to new applicants for years, and their anti-scraping infrastructure has tightened significantly since 2024.

This post covers what data is actually available on SoundCloud, why it's hard to get at scale, who needs it and why, and how to run our actor to extract it without building or maintaining any scraping infrastructure.

Why SoundCloud data is hard to get

The official API is effectively closed. SoundCloud's public API registration form has been disabled to new developers since 2021. Existing API keys still work, but new applications return a "registrations are temporarily disabled" message — and "temporarily" is now into its fifth year. For any team that didn't get access before 2021, the API is not an option.

The anti-scraping stack has gotten serious. SoundCloud's web app is a JavaScript-heavy single-page application that loads track data through internal endpoints with rotating client IDs. A naive requests.get() returns an empty shell. Headless browsers work, but SoundCloud added behavioral detection in 2024 that flags non-human navigation patterns within a few hundred requests. High-volume extraction now requires residential proxies and careful pacing.

Rate limiting is aggressive on the unauthenticated paths. Even legitimate browser sessions hit "too many requests" responses after a few hundred page loads from the same IP. The internal API endpoints that power the web app rotate their client tokens every few hours, which breaks any scraper that hardcodes them.

Data is fragmented across surfaces. A track has metadata on its own page, but play counts and engagement numbers update through a separate stats endpoint. Artist profiles list tracks but not full play counts. Playlists embed tracks but truncate descriptions. Pulling a complete picture means hitting multiple URLs per artist or track and stitching the responses together.

The result: most teams either pay for music industry data vendors (Chartmetric, Soundcharts) at $500-2,000/month, build fragile internal scrapers that need constant maintenance as SoundCloud updates its frontend, or just skip SoundCloud entirely and rely on Spotify data — which misses the entire underground/emerging-artist layer.

Who actually needs this data

A&R and label scouting. Independent labels and major-label A&R teams use SoundCloud as their primary scouting surface for hip-hop, electronic, and alternative music. Tracking which unsigned artists are gaining play counts week-over-week is how the next generation of signings gets identified — usually 6-12 months before the artist breaks on streaming platforms.

Music data analytics products. Companies building dashboards for managers, agents, and labels need SoundCloud track and artist data as a raw input alongside Spotify, Apple Music, YouTube, and TikTok numbers. SoundCloud is often the leading indicator that other platforms lag.

Playlist curation and discovery tools. Curators building niche playlists across genres need to scan thousands of new uploads weekly, filter by play count and like-to-play ratio, and surface candidates worth a human listen. Manual discovery doesn't scale past a few dozen tracks per week.

Sync licensing and music supervision. Music supervisors searching for tracks that fit a specific mood, BPM, or genre for ads, films, and games use SoundCloud as a pool of licensable indie music. Bulk track metadata extraction lets supervisors filter at scale rather than browsing manually.

Competitor and trend analysis. Labels and managers tracking what's working for competing artists need historical play count, repost, and follower trajectory data. SoundCloud exposes this on artist pages but doesn't offer any export or trend view.

Academic and journalistic research. Music researchers studying genre evolution, regional scene dynamics, or the structure of artist-to-artist influence need bulk data. SoundCloud is one of the few platforms where genre tags and reposts make those network structures visible.

What data you actually get

Our actor extracts the following fields from public SoundCloud pages — no authenticated session required:

track_id — SoundCloud track ID
title — track title
artist_name — track uploader display name
artist_url — canonical URL of the uploading artist's profile
genre — primary genre tag
tags — list of additional tags set by the uploader
duration_ms — track duration in milliseconds
play_count — total play count
like_count — total likes
repost_count — total reposts
comment_count — total comments
release_date — date the track was uploaded
description — full track description
artwork_url — URL to the track artwork image
track_url — canonical SoundCloud track URL
playlist_id — playlist ID (when scraping playlists)
playlist_title — playlist title
playlist_track_count — number of tracks in the playlist
artist_follower_count — uploader follower count
artist_track_count — total tracks uploaded by the artist
scraped_at — timestamp of extraction

How to run the actor

Via Apify Console (no code needed):

Go to apify.com/cryptosignals/soundcloud-scraper
Click Try for free
Paste your target URLs into the urls field — accepts track URLs, artist URLs, or playlist URLs
Set max_results to cap the run if you're working through a long list
Click Start and download results as JSON or CSV

Input JSON:

{
  "urls": [
    "https://soundcloud.com/skrillex",
    "https://soundcloud.com/discover/sets/charts-top:all-music",
    "https://soundcloud.com/fred-again/turn-on-the-lights-again-feat-future"
  ],
  "max_results": 100
}

Via Apify API:

curl -X POST "https://api.apify.com/v2/acts/cryptosignals~soundcloud-scraper/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{
    "urls": ["https://soundcloud.com/skrillex"],
    "max_results": 50
  }'

Sample output record:

{
  "track_id": "1234567890",
  "title": "Turn On The Lights again..",
  "artist_name": "Fred again..",
  "artist_url": "https://soundcloud.com/fred-again",
  "genre": "Electronic",
  "tags": ["house", "uk", "fred again", "future"],
  "duration_ms": 218000,
  "play_count": 14820000,
  "like_count": 412000,
  "repost_count": 38500,
  "comment_count": 9200,
  "release_date": "2022-10-21",
  "description": "Turn On The Lights again.. (feat. Future)",
  "artwork_url": "https://i1.sndcdn.com/artworks-...",
  "track_url": "https://soundcloud.com/fred-again/turn-on-the-lights-again-feat-future",
  "artist_follower_count": 1240000,
  "artist_track_count": 87,
  "scraped_at": "2026-05-04T09:00:00+00:00"
}

Pricing

The actor uses pay-per-result pricing: $0.005 per track or profile record. The first 5 results are free so you can verify output quality before committing. For a list of 1,000 tracks, that's $5.

For high-volume A&R workflows (10,000+ tracks per run), residential proxy coverage becomes important for reliability. Oxylabs is the proxy infrastructure we've tested for this kind of music platform workload — their residential network handles SoundCloud's IP reputation checks without the rotation failures that plague datacenter proxies.

What you don't get

The actor extracts public metadata visible to any logged-out visitor. It does not download audio files — that would violate SoundCloud's terms and copyright law on every track that isn't explicitly Creative Commons. If you need audio fingerprinting or feature extraction, that's a separate workflow that operates only on tracks the artist has explicitly licensed for that use.

Private tracks, private playlists, and any data behind a SoundCloud Pro paywall are not accessible. Comment text is summarized as a count rather than a full thread dump — full comment scraping at scale runs into rate limits that aren't worth fighting. Direct messages and private interactions are obviously off-limits.

For very large historical backfills (tracking play count over time for 100,000+ tracks), the play count snapshots from a single run aren't enough — you need scheduled runs and your own time-series storage. The actor produces the snapshot; you're responsible for the longitudinal database.

The alternative

You can build this yourself. The engineering work involves: handling SoundCloud's rotating client IDs, parsing the embedded JSON state from the HTML shell, dealing with the separate stats endpoints, managing proxy rotation against the rate-limit detection, retrying partial responses, and maintaining the scraper when SoundCloud updates its frontend — which has happened three times since 2024.

That's 2-3 weeks of engineering time to build, and ongoing maintenance after that. At $0.005 per record, you'd need to extract more than 2 million records before the build-vs-buy math favors building.

For most A&R, analytics, and curation teams, the answer is clear.

Actor: apify.com/cryptosignals/soundcloud-scraper

By: Web Data Labs — data infrastructure for music industry teams.

Crunchbase Data in 2026: Why It's Hard to Get and How to Extract It

agenthustler — Mon, 04 May 2026 08:27:52 +0000

Crunchbase is the de facto database of the startup world. Over 3 million company profiles, more than 600,000 funding rounds, and an investor graph that VCs, founders, and analysts treat as ground truth. The data sits behind a login wall, a heavy JavaScript frontend, and an enterprise API that prices most teams out of the market.

This post covers what data lives on Crunchbase, why it's hard to get programmatically, who needs it, and how to run our actor to extract it without building or maintaining any scraping infrastructure.

Why Crunchbase data is hard to get

The official API is enterprise-only. Crunchbase's Enterprise API starts around $49,000/year and requires an annual contract. The cheaper Pro subscription gives you a web UI and CSV exports, but no programmatic access for automation. There is no developer tier — the company has explicitly chosen to monetize data access at the top of the funnel. For a solo founder, an analyst at a small fund, or a data team building an internal CRM enrichment, the API simply isn't an option.

Most of the interesting data is behind a login. Public visitors see a stripped-down version of company pages. Funding round details, investor lists, key employees, acquisition history, and competitor data require an authenticated session. That single architectural choice eliminates 90% of naive scraping approaches — you can't just curl a profile URL and parse the HTML.

The frontend is heavy and dynamic. Crunchbase is a single-page React-style application. Profile data loads asynchronously through internal GraphQL-style endpoints, with request signing and session-bound tokens. The HTML you get on first paint contains almost no real data — it's a shell that hydrates client-side. Headless browsers can render it, but each profile takes seconds and significant compute, and the anti-bot stack flags automated browsers quickly.

The anti-bot stack is real. Cloudflare bot management, behavioral fingerprinting, request rate analysis, and aggressive IP reputation scoring all run on Crunchbase. A datacenter IP gets challenged within a few requests. Even residential proxies need careful rotation to avoid the heuristics that look for unnatural session patterns. This is a moving target — what worked last month often breaks this month.

The result: most teams either pay for Crunchbase Pro and copy-paste manually, license bulk data from resellers, or quietly maintain a fragile in-house scraper. None of these are ideal.

Who actually needs this data

VC and angel investor research. Before a partner meeting, an associate needs to pull funding history, current valuation signals, investor syndicate, key team members, and competitor landscape for a target company. Doing this manually across a pipeline of 50 deals per week is hours of clicking. Automated extraction turns it into a single overnight run.

Competitive intelligence. Mapping a competitive landscape means pulling funding rounds, headcount trajectories, and acquisition history for 20-100 companies in a sector. The funding round data alone — who invested, at what stage, when — is the spine of any defensible competitive analysis.

Sales enrichment for B2B targeting AI/SaaS startups. A sales team selling tools to Series A-C startups needs filtered lists by funding stage, last round date, total raised, and investor list. Crunchbase is where this data is most current and most accurate. Enriching a prospect list with last-funding signals lets sales prioritize companies likely to have budget right now.

M&A and corporate development. Corp dev teams scanning the market for acquisition targets pull acquisition history, funding totals, and founder backgrounds at scale. The investor list on a target also signals which firms might block or push a deal.

Market research and analyst reports. Researchers writing sector reports — fintech, climate tech, AI infrastructure — need bulk data on hundreds of companies in a category. Funding round data over time is the raw material for "where is the money flowing" charts that anchor most industry reports.

Founder competitive due diligence. Before raising, founders benchmark themselves against competitors: how much each raised, from whom, on what trajectory. Walking into a partner meeting with this data prepared is table stakes now.

What data you actually get

Our actor extracts the following fields from public and authenticated Crunchbase company profiles:

name — official company name
crunchbase_url — canonical Crunchbase profile URL
description — short and long company descriptions
website — official company website
founded_date — founding date as listed
headquarters — city, region, country
industries — list of industry categories
operating_status — active, closed, acquired
company_type — for profit, non-profit, etc.
employee_count — headcount range
total_funding — total funding raised in USD
last_funding_round — type, date, and amount of most recent round
funding_rounds — full list of funding rounds with stage, date, amount, and lead investors
investors — list of investor entities with name, type, and lead/follow flag
founders — founder names and roles
key_people — current executives and key employees
acquisitions — companies acquired, with dates and disclosed amounts
acquired_by — acquirer details if the company was acquired
competitors — competitor list as listed on the profile
scraped_at — extraction timestamp

How to run the actor

Via Apify Console (no code needed):

Go to apify.com/cryptosignals/crunchbase-scraper
Click Try for free
Paste your company list into the companies field — accepts Crunchbase slugs (e.g., stripe) or full profile URLs
Set max_results to cap the run if you're testing
Click Start and download results as JSON or CSV

Input JSON:

{
  "companies": [
    "stripe",
    "https://www.crunchbase.com/organization/anthropic",
    "openai"
  ],
  "include_funding_rounds": true,
  "include_investors": true,
  "max_results": 50
}

Via Apify API:

curl -X POST "https://api.apify.com/v2/acts/cryptosignals~crunchbase-scraper/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{
    "companies": ["stripe", "anthropic"],
    "include_funding_rounds": true,
    "max_results": 10
  }'

Sample output record:

{
  "name": "Anthropic",
  "crunchbase_url": "https://www.crunchbase.com/organization/anthropic",
  "description": "AI safety company building reliable, interpretable AI systems.",
  "website": "https://anthropic.com",
  "founded_date": "2021-01-01",
  "headquarters": "San Francisco, California, US",
  "industries": ["Artificial Intelligence", "Machine Learning", "Software"],
  "operating_status": "Active",
  "company_type": "For Profit",
  "employee_count": "501-1000",
  "total_funding": 7600000000,
  "last_funding_round": {
    "type": "Series E",
    "date": "2025-03-01",
    "amount": 3500000000
  },
  "funding_rounds": [
    {"type": "Series A", "date": "2021-05-01", "amount": 124000000, "lead_investors": ["Jaan Tallinn"]},
    {"type": "Series B", "date": "2022-04-01", "amount": 580000000, "lead_investors": ["Sam Bankman-Fried"]},
    {"type": "Series C", "date": "2023-05-01", "amount": 450000000, "lead_investors": ["Spark Capital"]}
  ],
  "investors": [
    {"name": "Google", "type": "corporate", "lead": true},
    {"name": "Spark Capital", "type": "vc", "lead": true},
    {"name": "Salesforce Ventures", "type": "corporate", "lead": false}
  ],
  "founders": [
    {"name": "Dario Amodei", "role": "CEO"},
    {"name": "Daniela Amodei", "role": "President"}
  ],
  "scraped_at": "2026-05-04T09:00:00+00:00"
}

Pricing

The actor uses pay-per-event pricing: $0.012 per company profile with funding rounds and investors included. The first 3 results are free so you can verify output quality before committing. For a list of 1,000 companies, that's $12.

For high-volume runs, residential proxy coverage matters for reliability. Oxylabs is the proxy infrastructure we've tested for this kind of workload — their residential network handles Crunchbase's reputation scoring without the constant rotation failures that plague datacenter proxies.

What you don't get

Crunchbase profiles don't include private company financials beyond disclosed funding rounds, employee email addresses, or anything behind Crunchbase Pro's premium signals (e.g., predictive scores). For contact-level data on individuals at these companies, you need a separate enrichment step.

The actor extracts what's available on the standard profile page. Some very large or very recently created companies have partial data on Crunchbase itself — that's a Crunchbase coverage limit, not an extraction limit.

The alternative

You can build this yourself. The engineering work involves: managing an authenticated session that doesn't get flagged, handling Crunchbase's anti-bot stack, parsing the React-hydrated data without depending on internal endpoints that change, building proxy rotation and retry logic, and maintaining the whole thing as Crunchbase pushes frontend updates — which happens often.

That's 3-6 weeks of engineering time to build something reliable, plus ongoing maintenance. At $0.012 per company, you'd need to scrape over 2.5 million profiles before the build-vs-buy math favors building. Crunchbase has 3 million profiles total, so realistically you never cross that line.

For most teams the answer is clear: don't build it.

Actor: apify.com/cryptosignals/crunchbase-scraper

By: Web Data Labs — data infrastructure for B2B and investor teams.

LinkedIn Profile Data in 2026: Why It's Hard to Get and How to Extract It

agenthustler — Mon, 04 May 2026 08:27:37 +0000

LinkedIn has more than 1 billion member profiles. Every recruiter, sales team, investor, and researcher needs profile data from LinkedIn. And yet getting that data programmatically is genuinely difficult — not because the data is hidden, but because LinkedIn has built one of the most aggressive anti-scraping systems on the web, and the alternatives are either expensive seat-based subscriptions or fragile in-house scrapers.

This post covers what data is actually available on LinkedIn profiles, why it's hard to get at scale, who needs it and why, and how to run our actor to extract it without building or maintaining any scraping infrastructure.

Why LinkedIn profile data is hard to get

There is no real official API. LinkedIn's public API does not expose member profile data to general developers. The endpoints that do exist are gated behind partnership programs (Talent Solutions, Sales Navigator API) that require a partner application, an enterprise contract, and a specific approved use case. For a solo recruiter or a small sales team doing list enrichment, the official API is effectively closed.

Sales Navigator is per-seat, not per-record. The most common workaround — Sales Navigator at $99/seat/month — gives a human the ability to browse profiles, but it doesn't give you a programmatic export. Bulk extraction violates LinkedIn's ToS for that product, and accounts get flagged quickly when used with browser automation.

The anti-scraping stack is serious. LinkedIn runs browser fingerprinting, behavioral analysis, IP reputation scoring, and bot challenge pages. A naive Python script gets blocked within minutes. Even headless browsers get flagged quickly without significant evasion infrastructure. High-volume profile extraction requires residential proxies and ongoing maintenance as LinkedIn updates its detection methods — sometimes weekly.

Profile pages are auth-walled in different ways depending on viewer state. A logged-out visitor sees a public preview. A logged-in visitor sees the full profile. Different proxy strategies, session strategies, and parsing logic apply to each — and getting reliable, consistent extraction across millions of profiles is a real engineering problem.

Terms of service add legal ambiguity. The hiQ Labs v. LinkedIn ruling (affirmed by the Ninth Circuit) established that scraping publicly available data is not a Computer Fraud and Abuse Act violation. The data on public profile pages — the kind visible to any logged-out visitor — sits in the clearest legal territory. Anything behind login is a different conversation.

The result: most teams either pay for expensive enrichment vendors (ZoomInfo, Apollo, Clay), build fragile in-house scrapers that need constant maintenance, or do it manually. None of these scale to the volumes most use cases actually need.

Who actually needs this data

Recruiting and talent sourcing. Identifying candidates by current role, experience, skills, and location is the standard recruiting workflow. Sourcers spend hours per week building shortlists. Automated profile extraction across a Boolean search result turns 4 hours of copy-paste into a 5-minute job.

B2B sales prospecting. Outbound teams enrich lead lists with current title, current employer, and seniority before scoring them against the ICP. The difference between a generic blast and a real personalized opener is whether you know what someone actually does today — not what their LinkedIn URL said three years ago.

Investor due diligence. Before a pre-seed call, investors check the founders' previous companies, education, and length of relevant experience. This is profile-level data, and right now it's mostly done manually by associates flipping between tabs.

Market and labor research. Researchers studying career trajectories, skills demand, or labor market shifts need bulk profile data as raw material. Same fields that power individual sales workflows also power dashboards on what skills are growing in a sector and where talent is concentrated.

Sales intelligence products. Anyone building a tool on top of LinkedIn signals — change-of-job alerts, hiring trend tools, revenue-per-employee benchmarks — needs profile data as input. The tools that look like magic from the outside are mostly clean profile extraction on the inside.

What data you actually get

Our actor extracts the following fields from public LinkedIn profile pages — no login required:

full_name — first and last name as listed on LinkedIn
headline — the tagline under the name (current role / personal pitch)
location — city, region, country
about — full "About" section text
current_position — most recent role title and company
experience — list of past roles with title, company, dates, and description
education — schools, degrees, fields of study, dates
skills — self-reported skills list
certifications — listed certifications with issuer and date
languages — listed languages with proficiency
connection_count — connection range (e.g., "500+")
follower_count — LinkedIn follower count
profile_url — canonical LinkedIn profile URL
avatar_url — URL to the profile photo
scraped_at — timestamp of extraction

How to run the actor

Via Apify Console (no code needed):

Go to apify.com/cryptosignals/linkedin-profile-scraper
Click Try for free
Paste your profile list into the profiles field — accepts LinkedIn slugs (e.g., williamhgates) or full URLs
Set max_results if you want to cap the run
Click Start and download results as JSON or CSV

Input JSON:

{
  "profiles": [
    "williamhgates",
    "https://www.linkedin.com/in/satyanadella",
    "jeffweiner08"
  ],
  "max_results": 50
}

Via Apify API:

curl -X POST "https://api.apify.com/v2/acts/cryptosignals~linkedin-profile-scraper/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{
    "profiles": ["williamhgates", "satyanadella"],
    "max_results": 10
  }'

Sample output record:

{
  "profile_id": "satyanadella",
  "full_name": "Satya Nadella",
  "headline": "Chairman and CEO at Microsoft",
  "location": "Redmond, Washington, United States",
  "about": "As chairman and CEO of Microsoft, I define my mission...",
  "current_position": {
    "title": "Chairman and CEO",
    "company": "Microsoft"
  },
  "experience": [
    {
      "title": "Chairman and CEO",
      "company": "Microsoft",
      "start_date": "2014-02",
      "end_date": null,
      "description": "Leading Microsoft as chairman and CEO."
    }
  ],
  "education": [
    {
      "school": "The University of Chicago Booth School of Business",
      "degree": "MBA"
    }
  ],
  "skills": ["Cloud Computing", "Strategy", "Leadership"],
  "connection_count": "500+",
  "follower_count": "11200000",
  "profile_url": "https://www.linkedin.com/in/satyanadella",
  "avatar_url": "https://media.licdn.com/dms/image/...",
  "scraped_at": "2026-05-04T09:00:00+00:00"
}

Pricing

The actor uses pay-per-event pricing: $0.012 per profile. The first 5 results are free so you can verify output quality before committing. For a list of 1,000 profiles, that's $12 — roughly the cost of a single coffee meeting, returning a structured dataset that would take a sourcer two days to compile manually.

For high-volume runs (10,000+ profiles), residential proxy coverage matters for reliability. Oxylabs is the proxy infrastructure we've tested for this kind of workload — their residential network handles LinkedIn's IP reputation checks without the constant rotation failures that plague datacenter proxies.

What you don't get

Profile pages don't include private email addresses or phone numbers. LinkedIn doesn't expose those publicly, and neither does this actor. For contact-level data, you need a separate enrichment step using a waterfall provider. The actor extracts profile-level public metadata — the data visible to any unauthenticated visitor.

The actor also does not extract private posts, private connections lists, or anything behind login. If your use case needs that, it doesn't belong in a public-data scraper — and it doesn't belong in a blog post either.

The alternative

You can build this yourself. The engineering work involves: handling LinkedIn's anti-bot detection, managing residential proxy rotation, parsing the structured profile data out of pages that ship as a JavaScript app, dealing with partial responses and retry logic, normalizing the experience and education sections (which have a dozen edge cases each), and maintaining the scraper when LinkedIn changes its page structure — which happens several times per year.

That's 3-6 weeks of engineering time to build a reliable version, and ongoing maintenance after that. At $0.012 per profile, you'd need to scrape over 500,000 profiles before the build-vs-buy math favors building.

For most teams, the answer is clear.

Actor: apify.com/cryptosignals/linkedin-profile-scraper

By: Web Data Labs — data infrastructure for B2B teams.

How to Monitor Telegram Channels for Crypto and Brand Intelligence in 2026

agenthustler — Mon, 04 May 2026 08:00:22 +0000

Telegram has become the de facto communication platform for crypto projects, brand communities, and market intelligence teams. With over 950 million monthly active users and channels routinely exceeding 100K subscribers, Telegram is where breaking information surfaces first — often hours before it hits Twitter or Discord.

If you're building any kind of monitoring pipeline in 2026, Telegram channels are a data source you can't ignore. Here's how to tap into them.

Why Telegram Monitoring Matters

Three major use cases have emerged:

Crypto intelligence. Major projects like TON, Solana ecosystem funds, and dozens of DeFi protocols use Telegram as their primary announcement channel. Token listings, governance votes, partnership announcements — they hit Telegram first. Traders who monitor these channels programmatically have an information edge measured in minutes.

Brand monitoring. Consumer brands increasingly maintain Telegram communities. Monitoring sentiment shifts, complaint spikes, or competitor channel activity gives product and marketing teams real-time signal that surveys take weeks to capture.

Market research. Hedge funds and research firms track channel growth rates, message frequency, and engagement patterns as alternative data signals. A channel gaining 10K subscribers in 24 hours often correlates with upcoming announcements.

The API Challenge

If you've tried to access Telegram data programmatically, you've hit one of these walls:

Bot API — Requires your bot to be added as an admin to the target channel. Great for channels you own, useless for monitoring external channels.

User API (MTProto) — Full access, but requires a phone number, 2FA authentication, and careful session management. Telegram actively rate-limits and bans accounts that behave like scrapers. One wrong move and your phone number is blocked.

What you actually need is read-only access to public channel data — no authentication, no phone numbers, no risk of account bans.

The Web Preview Endpoint

Telegram exposes a web preview for every public channel at t.me/s/{channel_name}. This endpoint returns:

Channel metadata: subscriber count, description, channel photo, verification status
Recent messages: the last 20-50 posts with full text content
Engagement data: view counts per message, forwarding info
Timestamps: exact post dates and times

No API key required. No authentication. Just an HTTP GET request.

Python Implementation

Here's a minimal working example that extracts key data from any public Telegram channel:

import requests
from bs4 import BeautifulSoup
from datetime import datetime

def scrape_telegram_channel(channel_name: str) -> dict:
    url = f"https://t.me/s/{channel_name}"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                      "AppleWebKit/537.36 (KHTML, like Gecko) "
                      "Chrome/120.0.0.0 Safari/537.36"
    }

    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")

    # Channel metadata
    title = soup.select_one(".tgme_channel_info_header_title")
    desc = soup.select_one(".tgme_channel_info_description")
    subscribers = soup.select_one(".tgme_channel_info_counter .counter_value")
    verified = soup.select_one(".verified-icon") is not None

    # Parse recent messages
    messages = []
    for msg in soup.select(".tgme_widget_message_wrap"):
        text_el = msg.select_one(".tgme_widget_message_text")
        views_el = msg.select_one(".tgme_widget_message_views")
        date_el = msg.select_one(".tgme_widget_message_date time")

        messages.append({
            "text": text_el.get_text(strip=True) if text_el else None,
            "views": views_el.get_text(strip=True) if views_el else None,
            "date": date_el["datetime"] if date_el else None,
        })

    return {
        "channel": channel_name,
        "title": title.get_text(strip=True) if title else None,
        "description": desc.get_text(strip=True) if desc else None,
        "subscribers": subscribers.get_text(strip=True) if subscribers else None,
        "is_verified": verified,
        "messages": messages,
        "scraped_at": datetime.utcnow().isoformat(),
    }

# Example: scrape Pavel Durov's channel
data = scrape_telegram_channel("durov")
print(f"Channel: {data['title']}")
print(f"Subscribers: {data['subscribers']}")
print(f"Verified: {data['is_verified']}")
print(f"Recent messages: {len(data['messages'])}")

Running this against durov (Pavel Durov's personal channel) returns structured data you can feed into any monitoring pipeline — no tokens, no phone numbers, no rate limit anxiety.

Key Data Points You Can Extract

Data Point	Location	Use Case
`subscriber_count`	Channel header	Growth tracking, trend detection
`message_text`	Message body	Keyword alerts, sentiment analysis
`message_views`	Message footer	Engagement scoring, virality detection
`message_date`	Message timestamp	Frequency analysis, pump detection
`is_verified`	Channel badge	Legitimacy filtering
`description`	Channel info	Category classification

Comparison: Three Approaches to Telegram Monitoring

Feature	Bot API	User API (MTProto)	Web Scraping
Auth required	Bot token + admin	Phone + 2FA	None
Access to public channels	Only if admin	Yes	Yes
Message history depth	Unlimited (if admin)	Unlimited	Last 20-50
Rate limits	30 msg/sec	Aggressive	Standard HTTP
Account ban risk	None	High	Low
Setup complexity	Low	High	Low
Best for	Your own channels	Full archive access	Monitoring & alerts

For most monitoring use cases — where you need recent activity, subscriber counts, and engagement data from channels you don't own — web scraping is the pragmatic choice.

Scaling Beyond a Script

The Python example above works for ad-hoc checks, but production monitoring needs scheduling, proxy rotation, error handling, and structured output.

If you need a production-ready solution, the Telegram Channel Scraper on Apify handles all of this out of the box — batch channel processing, JSON/CSV export, and scheduled runs with webhook notifications. It's built on the same web preview approach, so no Telegram authentication is needed.

What to Build With This

Once you have the data flowing, the interesting work begins:

Alert pipelines: Trigger notifications when a crypto channel posts for the first time in 48 hours (often precedes announcements)
Growth dashboards: Track subscriber count changes across competitor channels daily
Sentiment analysis: Run message text through an LLM to classify sentiment shifts
Cross-channel correlation: Detect when multiple channels in the same niche start posting about the same topic

Telegram monitoring is one of those capabilities where the gap between "I should do this" and "I'm actually doing this" is surprisingly small. A few lines of Python and you're in.

itch.io Game Data at Scale: Who Needs It and How to Extract It

agenthustler — Mon, 04 May 2026 07:03:34 +0000

itch.io hosts over 500,000 indie games and has roughly 10 million registered users. It is the dominant platform for game jams, experimental titles, and developers who want to publish without a publisher. All of that catalog data — titles, ratings, genres, platforms, creator information — is publicly visible. None of it is available through a bulk API.

This post covers what data is actually accessible on itch.io, who needs it and why, and how to run our actor to extract it without building or maintaining any scraping infrastructure.

The data access problem

There is no public itch.io API for catalog data. itch.io provides a limited API for developers to manage their own games and purchases, but there is no endpoint that lets you query the catalog by genre, rating, platform, or creator. If you want a list of the top-rated horror games, or all puzzle games with more than 1,000 ratings, or every title a specific developer has published — you cannot get that from an API call. The data exists, it is all publicly visible on the site, but there is no programmatic way to retrieve it in bulk.

The catalog is large and fragmented. With 500,000+ games spanning dozens of genres, tags, and platforms, manually compiling even a modest dataset is impractical. A researcher trying to track rating trends across game jams, or a journalist building a "best of" list, or a developer benchmarking their own game against the market — all of them face the same problem: the data is there, but getting it out at any scale requires either hours of manual copy-paste or purpose-built extraction tooling.

Pages are server-rendered. itch.io renders its browse and search pages as standard HTML. This means the data is accessible to anyone who can read a web page — but actually collecting it at scale, handling pagination, respecting the site's rate limits, and normalizing the output into a structured format still requires non-trivial engineering work to do right.

Who actually needs this data

Indie game developers doing competitive research. Before pricing a new game, launching a jam entry, or picking a genre to target, developers want to understand the landscape. How many games in a given tag have ratings above 4.0? What is the average rating count for top-performing puzzle games? What platforms do the most-played browser games support? These are answerable questions if you have the data.

Game jam organizers and curators. Jam organizers running events on itch.io often want to survey the catalog of entries — track which games are gaining traction post-jam, compile shortlists for coverage, or analyze genre distribution across submissions. Doing this for dozens or hundreds of games without automated extraction is slow.

Game journalists and critics. Writers covering the indie space need lists. The top-rated games in a specific tag this year. The highest-rated games with under 500 ratings (hidden gems). All titles from a creator whose new release just launched. These are the kinds of queries that make for good editorial angles, and right now most journalists build them manually.

Academic researchers. Game studies researchers use itch.io as a field site. Analyzing genre trends over time, studying rating distributions, tracking which platform combinations correlate with higher engagement — all of this requires structured data from a large sample. itch.io's public catalog is one of the few places where indie game metadata is available at this scale.

Data product builders. Teams building game discovery tools, recommendation engines, or market intelligence products need raw catalog data as an input. itch.io is a natural complement to Steam data for anyone covering the full indie game market, not just the commercial storefront.

What data you actually get

Our actor extracts the following fields from public itch.io game listings — no account or API key required:

id — itch.io's internal game ID
title — game title as listed on the platform
url — direct link to the game page
creator — developer or publisher name
creator_url — link to the creator's itch.io profile
description — short game description from the listing
genre — genre classification (e.g., Action, Visual Novel, Platformer, RPG)
rating — average rating on a 0–5 scale
rating_count — total number of ratings
platforms — supported platforms: windows, mac, linux, android, browser
thumbnail — URL of the game's cover image

How to run the actor

The actor supports four modes: browsing top-rated games, browsing featured/popular games, searching by keyword, and filtering by tag. All modes return the same structured output.

Via Apify Console (no code needed):

Go to apify.com/cryptosignals/itchio-scraper
Click Try for free
Choose your action and set any query parameters
Click Start and download results as JSON or CSV

Input: top-rated games

{
  "action": "top",
  "maxItems": 100
}

Input: games by tag

{
  "action": "tag",
  "query": "horror",
  "maxItems": 50
}

Input: keyword search

{
  "action": "search",
  "query": "bullet hell",
  "maxItems": 50
}

Input: featured games

{
  "action": "featured",
  "maxItems": 36
}

Via Apify API:

curl -X POST "https://api.apify.com/v2/acts/cryptosignals~itchio-scraper/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{
    "action": "tag",
    "query": "puzzle",
    "maxItems": 100
  }'

Sample output record:

{
  "id": 123456,
  "title": "HoloCure - Save the Fans!",
  "url": "https://kay-yu.itch.io/holocure",
  "creator": "Kay Yu",
  "creator_url": "https://kay-yu.itch.io",
  "description": "Fan-made inspired by a certain VTuber group.",
  "genre": "Action",
  "rating": 4.94,
  "rating_count": 18432,
  "platforms": ["windows"],
  "thumbnail": "https://img.itch.zone/..."
}

Results are returned as a JSON array and can be exported as CSV directly from the Apify console.

Pricing

The actor uses pay-per-event pricing — you pay per game extracted, and the first results are free so you can verify output quality before committing to a larger run. For typical research datasets of a few hundred games, the cost is low enough that the build-vs-buy decision is clear. Check the actor page for current pricing.

What you don't get

Price is not available in itch.io's listing pages. itch.io does not expose game prices in browse or search results — pricing is only visible on individual game pages. The actor extracts listing-level data (what you see when browsing by top, featured, search, or tag), not the full detail page for each game.

Individual game page data — full description text, all screenshots, download counts, comment threads, or pricing — would require a separate per-game fetch for each title. The current actor focuses on catalog-level metadata.

The alternative

You can build this yourself. The engineering work involves: handling itch.io's pagination across potentially hundreds of pages, normalizing inconsistent genre and platform markup, implementing polite rate limiting to avoid overloading the server, parsing structured data out of HTML, and maintaining the scraper when the page structure changes. That is a day or two of engineering time to build, plus ongoing maintenance.

For most research workflows, teams do not need a custom scraper — they need the data. The actor delivers structured JSON ready for analysis without any infrastructure overhead.

Actor: apify.com/cryptosignals/itchio-scraper

By: Web Data Labs — data infrastructure for developers and researchers.

Scraping Shopify Stores in 2026: Product Catalog, Pricing & Inventory Data

agenthustler — Mon, 04 May 2026 05:28:20 +0000

If you've ever tried to monitor competitor pricing on a Shopify store, build a dropshipping research pipeline, or feed a market-intel dashboard with live e-commerce data, you've probably learned the hard way that "just scrape it" is a sentence that hides a lot of pain.

Shopify powers somewhere north of 4.6 million live storefronts in 2026. Each one is a goldmine of structured data — product catalogs, variant matrices, real-time inventory, pricing changes — but extracting that data reliably across thousands of stores is an engineering problem that gets messy fast.

This post walks through why scraping Shopify is harder than it looks, what kinds of business problems good Shopify data solves, and how to plug a managed scraper into your stack without writing (or maintaining) your own.

Why businesses need Shopify store data

Shopify's open architecture means a lot of useful data is technically reachable — and that creates demand from teams that aren't going to build a scraper from scratch:

Price monitoring. DTC brands and retailers want to know when competitors discount, restock, or change MSRP. A daily snapshot across 200 competitor stores beats a quarterly manual audit.
Competitive intelligence. Which SKUs is a competitor pushing on their homepage? Which collections did they reshuffle this week? Which products are quietly hidden but still sold? This is the data that ends up in board decks.
Dropshipping research. Finding winning products before they saturate is the entire dropshipping playbook. Tracking new listings, sudden inventory drops, and review velocity across hundreds of niche stores is how serious dropshippers find signal.
Inventory tracking. Suppliers and resellers need to know when a hot product is back in stock — sometimes within minutes. Polling key SKUs is a real-money problem.
Market analysis. Aggregating product, price, and category data across a vertical (say, sustainable fashion or pet supplements) tells you category-level trends no single store can.

The pattern across all of these: structured product data, refreshed often, normalized across many stores. That's deceptively hard.

Why scraping Shopify is non-trivial

Shopify looks easy. The platform is consistent: every store has predictable URL patterns, products have stable IDs, and a lot of data is exposed in JSON. New scrapers usually start optimistic and get humbled within a week. Here's what bites them:

1. Rate limits and adaptive throttling

Shopify's edge applies aggressive rate limiting that adapts to traffic patterns. A naive scraper hammering one store will get throttled within minutes. The signs are subtle — slower responses, partial pages, soft 429s wrapped as 200s with truncated bodies. By the time you notice, your dataset is already corrupt.

Doing this at scale (hundreds of stores, hourly) needs distributed request scheduling, exponential backoff, and a rotating residential proxy pool. Providers like Oxylabs and ScraperAPI exist precisely because this layer is non-trivial — they handle the proxy rotation, geo-targeting, and CAPTCHA solving that you'd otherwise be reinventing.

2. Pagination quirks

Shopify exposes product listings through several different endpoints, each with its own pagination quirks, page-size caps, and silent truncation behaviors. Some endpoints will happily return the first 250 products and then stop. Others paginate cleanly until they suddenly return duplicates. Building a scraper that actually gets every product on a 50,000-SKU store, every time, is a long debugging exercise.

3. Variant explosion

A single product can have dozens of variants — size × color × material × bundle. A store with 1,000 visible products can expand to 30,000+ variant rows once you flatten it. Storage, deduplication, and "is this the same product?" matching all become real concerns.

4. JSON vs HTML endpoints disagree

The JSON endpoints, the rendered HTML, and the search results sometimes disagree about what's available. A product can be hidden from collection pages but still purchasable via direct URL. Inventory counts shown in HTML may not match the underlying JSON. A robust scraper has to reconcile these views — and decide which is the source of truth.

5. Anti-bot defenses are getting smarter

Cloudflare, custom JS challenges, fingerprinting, behavioral detection — Shopify stores are increasingly protected. Tools like ScrapeOps help with monitoring and bypass orchestration, but the cat-and-mouse game eats engineering time you'd rather spend on your actual product.

The honest summary: writing a one-off scraper for one Shopify store on a Tuesday afternoon is fine. Running a reliable, normalized, multi-store scraping pipeline in production is a months-long engineering project most teams shouldn't take on.

How to use our actor

The Shopify Store Scraper on Apify is a managed solution. You give it a list of stores and parameters; you get back a clean, normalized dataset.

Input

The actor takes a simple JSON input:

{
  "startUrls": [
    { "url": "https://allbirds.com" },
    { "url": "https://gymshark.com" },
    { "url": "https://kith.com" }
  ],
  "maxProducts": 5000,
  "includeVariants": true,
  "includeInventory": true,
  "currency": "USD"
}

That's the whole interface. No proxy config, no rate-limit tuning, no pagination strategy — those are the actor's job.

Output

Each product comes back as a normalized record:

{
  "store": "allbirds.com",
  "productId": "7891234567890",
  "handle": "wool-runner-mizzles",
  "title": "Wool Runner Mizzles",
  "vendor": "Allbirds",
  "productType": "Shoes",
  "tags": ["mens", "weather-ready", "wool"],
  "url": "https://allbirds.com/products/wool-runner-mizzles",
  "images": [
    "https://cdn.shopify.com/.../mizzle-1.jpg",
    "https://cdn.shopify.com/.../mizzle-2.jpg"
  ],
  "price": {
    "min": 115.00,
    "max": 135.00,
    "currency": "USD"
  },
  "variants": [
    {
      "variantId": "44123456789",
      "title": "M9 / Natural Black",
      "sku": "WRM-M9-NB",
      "price": 115.00,
      "compareAtPrice": 135.00,
      "available": true,
      "inventoryQuantity": 23,
      "options": { "size": "M9", "color": "Natural Black" }
    }
  ],
  "createdAt": "2025-09-12T10:14:00Z",
  "updatedAt": "2026-04-30T08:21:00Z",
  "scrapedAt": "2026-05-04T14:00:00Z"
}

This is the shape your downstream code wants — flat, predictable, normalized currency, ISO timestamps, a stable productId you can dedupe on. Drop it into BigQuery, Postgres, a vector DB, or a spreadsheet and it just works.

Calling it from code

You don't need to learn the actor's internals. From any language with HTTP, you start a run with the input above against the Apify API and pull results from the dataset when it's done. There's a Python client, a JS client, and webhook delivery if you want push instead of pull.

Use cases

A few concrete patterns we've seen users build:

Dropshippers

Run the actor nightly across a curated list of 100–300 niche stores. Diff against yesterday's snapshot to surface:

New product launches (likely test SKUs)
Sudden inventory drops (hot product signal)
Price increases (validated demand)
Variants that keep selling out (winners)

The whole pipeline is one cron job, one Postgres table, and a Slack notifier on the diff.

E-commerce teams

Marketing and merchandising teams use it to monitor 20–50 competitors. The output feeds a dashboard that flags pricing changes the moment they happen — which means the brand can respond same-day instead of finding out at the next quarterly review.

Market analysts

Researchers building reports on a vertical (sustainable beauty, technical apparel, indie coffee) point the actor at 500+ stores in the category, then aggregate average prices, top tags, common product types, and category mix. What used to be a six-week consultant engagement becomes a weekend of analysis.

Re-sellers and inventory bots

Polling specific SKUs across supplier stores to catch restocks. The actor's variant-level inventory output makes this clean — you watch one variant ID and trigger when available flips to true.

Try it

If any of those sound like your problem, the Shopify Store Scraper on Apify is the fastest way to skip the months of pipeline engineering and get straight to the data. There's a free tier — point it at one store and see the output for yourself before committing.

If you do decide to roll your own (some teams need to), at least save yourself the proxy-and-bypass headache — Oxylabs and ScraperAPI handle the infrastructure layer that's the most painful to maintain, and ScrapeOps gives you visibility into what's actually happening when things break.

Either way: stop scraping the slow way. The data's there — go get it.

[DRAFT - IGNORE]

agenthustler — Mon, 04 May 2026 03:59:37 +0000

Internal test draft.

LinkedIn Company Data in 2026: Why It's Hard to Get and How to Extract It

agenthustler — Mon, 04 May 2026 03:59:04 +0000

LinkedIn has over 67 million company pages. Every B2B sales team, investor, and recruiter needs company data from LinkedIn. And yet getting that data programmatically is genuinely difficult — not because the data is hidden, but because LinkedIn has built one of the most aggressive anti-scraping systems on the web, and their official API is priced for enterprise budgets only.

This post covers what data is actually available on LinkedIn company pages, why it's hard to get at scale, who needs it and why, and how to run our actor to extract it without building or maintaining any scraping infrastructure.

Why LinkedIn company data is hard to get

The official API is not a real option for most teams. LinkedIn's Marketing Developer Platform costs $15,000+/year and requires a partner application process. The data endpoints available through official channels are primarily designed for ad targeting and HR software integrations — not bulk company research. For a solo founder or a small data team doing ICP analysis or competitive research, the API is effectively unavailable.

The anti-scraping stack is serious. LinkedIn runs browser fingerprinting, behavioral analysis, IP reputation scoring, and bot challenge pages. A naive Python requests script gets blocked within minutes. Even headless browsers get flagged quickly without significant investment in evasion infrastructure. High-volume extraction requires residential proxies — which add meaningful cost — and constant maintenance as LinkedIn updates its detection methods.

Terms of service add legal ambiguity. LinkedIn's ToS restricts automated data collection. The hiQ Labs v. LinkedIn ruling (affirmed by the Ninth Circuit) established that scraping publicly available data is not a Computer Fraud and Abuse Act violation, but companies still need to assess their own risk tolerance. The data on public company pages — the kind visible to any logged-out visitor — sits in the clearest legal territory.

The result: most teams either pay for expensive data vendors (ZoomInfo, Clearbit), build fragile in-house scrapers that need constant maintenance, or just do it manually. None of these scale.

Who actually needs this data

B2B sales and ICP research. Building an ideal customer profile requires enriching company lists with industry, headcount, HQ location, and founding year. Teams doing outbound at scale need to filter thousands of companies down to the 200 that actually fit their ICP. LinkedIn company pages are the canonical source for this data — more accurate and more current than most third-party databases.

Investor due diligence. Before a call, investors verify headcount growth signals (employee count vs. last quarter), check the company description for pivot signals, and confirm website and contact details. LinkedIn is the ground truth that other sources pull from. Automating this enrichment across a deal pipeline saves hours per week.

Competitive landscape analysis. Mapping a competitive landscape means collecting industry, size, HQ, founding year, and specialties for 20-100 companies. Doing this manually in a spreadsheet is an afternoon of copy-paste. Automated extraction turns it into a 5-minute job.

Recruitment targeting. Identifying companies in a specific industry, headcount band, and city before sourcing candidates from those companies is a standard recruiting workflow. LinkedIn company data is the filter layer.

Market research and data products. Research teams building industry reports, data enrichment services, or market intelligence products need bulk company data as a raw material. The same fields that power sales enrichment also power competitive benchmarking tools and market maps.

What data you actually get

Our actor extracts the following fields from public LinkedIn company pages — no login required:

name — official company name as listed on LinkedIn
industry — LinkedIn industry classification (e.g., "Software Development", "Financial Services")
employee_count — headcount range (e.g., "501-1000", "10001+")
follower_count — LinkedIn follower count
headquarters — city, state/region, country
founded_year — year the company was founded
website — official company website URL
company_url — canonical LinkedIn company page URL
description — full company description text
tagline — short company tagline
specialties — list of self-reported specialty areas
logo_url — URL to the company logo image
scraped_at — timestamp of extraction

How to run the actor

Via Apify Console (no code needed):

Go to apify.com/cryptosignals/linkedin-company-scraper
Click Try for free
Paste your company list into the companies field — accepts LinkedIn slugs (e.g., stripe) or full URLs
Set max_results if you want to cap the run
Click Start and download results as JSON or CSV

Input JSON:

{
  "companies": [
    "stripe",
    "https://www.linkedin.com/company/shopify",
    "notion"
  ],
  "max_results": 50
}

Via Apify API:

curl -X POST "https://api.apify.com/v2/acts/cryptosignals~linkedin-company-scraper/runs" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -d '{
    "companies": ["stripe", "shopify"],
    "max_results": 10
  }'

Sample output record:

{
  "company_id": "stripe",
  "name": "Stripe",
  "tagline": "Financial infrastructure for the internet",
  "description": "Stripe is a financial infrastructure platform for businesses...",
  "industry": "Software Development",
  "employee_count": "5001-10000",
  "follower_count": "1240000",
  "headquarters": "San Francisco, California, US",
  "founded_year": 2010,
  "website": "https://stripe.com",
  "specialties": ["Payments", "Financial Infrastructure", "Developer Tools"],
  "logo_url": "https://media.licdn.com/dms/image/...",
  "company_url": "https://www.linkedin.com/company/stripe",
  "scraped_at": "2026-05-04T09:00:00+00:00"
}

Pricing

The actor uses pay-per-event pricing: $0.008 per company starting May 17, 2026. The first 5 results are free so you can verify output quality before committing. For a list of 1,000 companies, that's $8.

For high-volume runs (10,000+ companies), residential proxy coverage becomes important for reliability. Oxylabs is the proxy infrastructure we've tested and trust for this kind of workload — their residential network handles LinkedIn's IP reputation checks without constant rotation failures that plague datacenter proxies.

What you don't get

Company pages don't include employee email addresses, phone numbers, or individual employee profiles. For contact-level data, you need a separate enrichment step. The actor extracts company-level public metadata — the data visible to any unauthenticated visitor on a public company page.

LinkedIn also rate-limits aggressively on certain pages. The actor handles this, but very large runs (5,000+ companies) benefit from Bright Data's residential network as a proxy layer to maintain throughput.

The alternative

You can build this yourself. The engineering work involves: handling LinkedIn's anti-bot detection, managing proxy rotation, parsing the structured data out of the page (LinkedIn embeds JSON-LD in company pages), dealing with partial responses and retry logic, and maintaining the scraper when LinkedIn changes its page structure — which happens several times per year.

That's 2-4 weeks of engineering time to build, and ongoing maintenance after that. At $0.008 per company, you'd need to scrape over 1 million companies before the build-vs-buy math favors building.

For most teams, the answer is clear.

Actor: apify.com/cryptosignals/linkedin-company-scraper

By: Web Data Labs — data infrastructure for B2B teams.

Crunchbase API in 2026: Free Tier Gone — What Startup Data Hunters Do Now

agenthustler — Sat, 02 May 2026 08:00:16 +0000

Crunchbase API: The Free Tier Is Dead

If you’re a developer who used Crunchbase’s free API tier for startup research, funding data, or market analysis — it’s gone. As of 2025, Crunchbase eliminated free API access entirely. The cheapest plan now starts at $49/month (Basic), with the full-featured API requiring the Pro plan at $99/month.

For indie developers, researchers, and early-stage startups who need startup ecosystem data, this pricing change fundamentally changes the equation.

Crunchbase API Pricing in 2026

Plan	Price	API Access	Daily Limit	Data Available
Free	$0 Removed	No	—	—
Basic	$49/mo	Limited	200 calls/min	Basic company data
Pro	$99/mo	Full	200 calls/min	Full dataset + exports
Enterprise	Custom	Full	Custom	Everything + support

What $49/Month Gets You

import requests

CB_API_KEY = "your_api_key_here"

def search_companies(query, limit=25):
    url = "https://api.crunchbase.com/api/v4/searches/organizations"
    headers = {"X-cb-user-key": CB_API_KEY}
    payload = {
        "field_ids": ["identifier", "short_description", "funding_total",
                      "num_funding_rounds", "founded_on"],
        "query": [{"type": "predicate",
                   "field_id": "identifier",
                   "operator_id": "contains",
                   "values": [query]}],
        "limit": limit
    }

    resp = requests.post(url, json=payload, headers=headers)
    return resp.json()

results = search_companies("ai agent")
for entity in results.get("entities", []):
    props = entity["properties"]
    funding = props.get("funding_total", {}).get("value_usd", 0)
    print(f"{props['identifier']['value']} — ${funding:,.0f} raised")

The data quality is excellent. But $588-$1,188/year is a hard sell for individual developers or side projects.

What You Lose Without Crunchbase API

Crunchbase’s dataset is uniquely valuable:

Funding rounds — who invested, how much, what stage
Company profiles — founding date, team size, location, categories
Acquisition data — who bought whom and for how much
People data — founders, executives, board members
Market maps — companies by category and geography

No other single source combines all of this. But paying $49+/month for a data project that may or may not produce value? That’s a tough startup cost.

The Web Scraping Alternative

Crunchbase’s company profiles are publicly accessible on the web. The data displayed on their website is the same data behind the API:

import requests
from bs4 import BeautifulSoup

def get_crunchbase_company(company_slug):
    url = f"https://www.crunchbase.com/organization/{company_slug}"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }

    resp = requests.get(url, headers=headers)
    soup = BeautifulSoup(resp.text, "html.parser")

    # Note: Crunchbase heavily uses JavaScript rendering
    # Basic requests won’t get much — you need browser automation
    title = soup.select_one("h1")

    return {
        "name": title.text.strip() if title else None,
        "url": url
    }

The challenge: Crunchbase is a React single-page application. Most data loads dynamically via JavaScript, which means simple HTTP requests won’t work. You need:

Headless browser — Playwright or Puppeteer to render JavaScript
Proxy rotation — Crunchbase blocks datacenter IPs quickly
Anti-detection — fingerprint management, human-like behavior
Rate management — respectful pacing to avoid blocks

API vs Scraping: Side-by-Side

Feature	Crunchbase API ($49+/mo)	Web Scraping
Cost	$49-99/month	Infrastructure only
Access barrier	Credit card required	None
Data format	Clean JSON	Requires parsing
Company profiles	Full	Public data
Funding data	Detailed	As displayed
People/team data	Yes	Public profiles
Historical data	Full history	Limited to current
Bulk export	Pro plan only ($99/mo)	Unlimited
Rate limits	200 calls/min	Self-managed
Setup complexity	Low (API keys)	High (browser automation)

Scaling Crunchbase Data Collection

For production use, building Crunchbase scraping infrastructure from scratch is complex. The SPA rendering, anti-bot measures, and data structure changes require ongoing maintenance.

Managed scraping tools like this Crunchbase scraper on Apify handle the browser automation and proxy management:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("cryptosignals/crunchbase-scraper").call(
    run_input={
        "searchQuery": "artificial intelligence",
        "location": "San Francisco",
        "fundingStage": "Series A",
        "maxResults": 200
    }
)

for company in client.dataset(run["defaultDatasetId"]).iterate_items():
    funding = company.get("totalFunding", 0)
    print(f"{company['name']} — ${funding:,.0f} — {company.get('category')}")

Alternative Data Sources for Startup Research

If neither the API nor scraping fits your needs, consider these alternatives:

Source	Cost	Strengths	Weaknesses
PitchBook	Enterprise ($$$)	Most comprehensive	Expensive
Dealroom	Free tier available	EU/startup focus	Limited US data
OpenVC	Free	VC-focused	Smaller dataset
Tracxn	Free tier available	Good coverage	Limited free access
LinkedIn	Free (limited)	People data strong	No funding data
AngelList/Wellfound	Free	Startup jobs + data	Limited API

The Cost Comparison

Let’s do the math for a typical startup research project:

Crunchbase API (1 year):
  Basic: $49 x 12 = $588/year
  Pro:   $99 x 12 = $1,188/year

Web scraping (Apify, typical usage):
  Pay-per-result: ~$5-20/month for moderate use
  Annual: $60-240/year

Savings: 60-90% depending on usage

The API wins on convenience and data structure. Scraping wins on cost and flexibility.

When to Use What

Use the Crunchbase API if:

You have budget and need clean, reliable data
You’re building a product where Crunchbase data is core
You need historical funding data going back years
You want zero maintenance overhead

Use web scraping if:

You’re exploring and don’t want to commit $49+/month
You need bulk data beyond API rate limits
You’re combining data from multiple sources
You need flexibility the API doesn’t offer

The Bottom Line

Crunchbase’s decision to remove the free tier makes business sense — their data is genuinely valuable and they’re entitled to charge for it. But it also means the barrier to entry for startup data access has gone from $0 to $588/year overnight.

For developers and researchers who need startup ecosystem data without the subscription commitment, web scraping provides a cost-effective alternative. The key is choosing the right approach for your specific use case and budget.

How do you source startup and funding data? Found a good Crunchbase alternative? Let me know in the comments.

Amazon Product API (PA-API) in 2026: Restrictions, Alternatives, and Web Scraping

agenthustler — Fri, 01 May 2026 08:00:18 +0000

Amazon’s Product Advertising API: The Access Problem

Amazon’s Product Advertising API (PA-API 5.0) is powerful — when you can use it. The catch? You need an active Amazon Associates account with at least 3 qualifying sales in the past 30 days just to maintain access.

For new developers, researchers, and startups building price comparison tools or product databases, this creates a chicken-and-egg problem: you need the API to build your product, but you need sales (from a product you haven’t built yet) to keep API access.

Let’s examine the current state of PA-API, its restrictions, and why web scraping has become the go-to alternative.

PA-API 5.0 Requirements in 2026

Requirement	Details
Associates account	Must be approved for your country
Sales requirement	3 qualifying sales in 30 days
Rate limit	1 req/sec (scales with revenue)
Initial quota	8,640 requests/day
Data available	Product info, pricing, reviews (limited)
Geographic restriction	Separate API per marketplace (US, UK, DE, etc.)
Revenue share	1-10% depending on category

The Access Cliff

Here’s the brutal part: if your Associates account goes 30 days without 3 sales, Amazon revokes your API keys. You have to reapply.

import requests

response = requests.post(
    "https://webservices.amazon.com/paapi5/getitems",
    headers={"Authorization": "..."},
    json={"ItemIds": ["B09V3KXJPB"]}
)

# After 30 days without 3 sales:
# {
#   "Errors": [{
#     "Code": "TooManyRequests",
#     "Message": "Your access has been revoked."
#   }]
# }

Who Gets Blocked by PA-API Requirements?

New developers building their first e-commerce tool
Researchers analyzing product trends or pricing history
Startups building comparison shopping engines
International developers in countries without Associates programs
Data analysts who need bulk product data for market research

If you’re in any of these categories, the API isn’t a realistic starting point.

What PA-API Actually Provides (When You Have Access)

from paapi5_python_sdk.api.default_api import DefaultApi
from paapi5_python_sdk.models.get_items_request import GetItemsRequest
from paapi5_python_sdk.models.partner_type import PartnerType

api = DefaultApi(
    access_key="YOUR_ACCESS_KEY",
    secret_key="YOUR_SECRET_KEY",
    host="webservices.amazon.com",
    region="us-east-1"
)

request = GetItemsRequest(
    partner_tag="your-tag-20",
    partner_type=PartnerType.ASSOCIATES,
    item_ids=["B09V3KXJPB"],
    resources=[
        "ItemInfo.Title",
        "Offers.Listings.Price",
        "Images.Primary.Large",
        "CustomerReviews.StarRating"
    ]
)

response = api.get_items(request)
for item in response.items_result.items:
    print(f"{item.item_info.title.display_value}")
    print(f"Price: {item.offers.listings[0].price.display_amount}")

The data is clean and structured. But at 1 request per second with sales-gated access, it’s not built for data collection — it’s built for affiliate links.

Web Scraping: The Practical Alternative

Amazon’s product pages are publicly accessible. Scraping extracts the same data without the affiliate sales requirement:

import requests
from bs4 import BeautifulSoup

def scrape_amazon_product(asin):
    url = f"https://www.amazon.com/dp/{asin}"
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
        "Accept-Language": "en-US,en;q=0.9"
    }

    resp = requests.get(url, headers=headers)
    soup = BeautifulSoup(resp.text, "html.parser")

    title = soup.select_one("#productTitle")
    price = soup.select_one(".a-price .a-offscreen")
    rating = soup.select_one("#acrPopover")

    return {
        "asin": asin,
        "title": title.text.strip() if title else None,
        "price": price.text if price else None,
        "rating": rating.get("title") if rating else None
    }

product = scrape_amazon_product("B09V3KXJPB")
print(product)

Warning: Amazon has some of the most aggressive anti-bot systems on the web. Basic scraping like the above will get blocked within 10-20 requests. For any real use case, you need:

Residential proxy rotation
Browser fingerprint management
CAPTCHA solving
Request throttling
Geographic proxy matching (US proxies for amazon.com)

API vs Scraping: Head-to-Head

Feature	PA-API 5.0	Web Scraping
Access barrier	3 sales/30 days	None
Cost	Free (if qualified)	Proxy/infrastructure costs
Rate limit	1 req/sec	Self-managed
Product data	Structured JSON	Requires parsing
Price history	Current only	Can build your own
Review text	Not available	Available
Bulk collection	Very limited	Scalable
Setup time	Days (approval wait)	Hours
Reliability	High (when accessible)	Requires maintenance

Scaling Amazon Data Collection

Building and maintaining Amazon scraping infrastructure is a significant engineering challenge. Anti-bot systems update frequently, and what works today may break tomorrow.

Managed scraping tools abstract this away. This Amazon scraper on Apify handles the proxy rotation, CAPTCHA solving, and anti-detection for you:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("cryptosignals/amazon-scraper").call(
    run_input={
        "searchTerms": ["wireless headphones"],
        "marketplace": "amazon.com",
        "maxProducts": 200,
        "includeReviews": True
    }
)

for product in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{product['title'][:60]} — ${product['price']} — {product['rating']} stars")

Alternative Data Sources

Beyond scraping Amazon directly, consider:

Keepa API — Amazon price history tracking ($20/mo)
Rainforest API — structured Amazon data (pay-per-request)
CamelCamelCamel — price history (no API, scrapeable)
Google Shopping API — aggregated product data

Each has tradeoffs in cost, coverage, and freshness.

The Bottom Line

Amazon’s PA-API is designed for affiliate marketers who are already making sales, not for developers who need product data. The 3-sales-in-30-days requirement isn’t a bug — it’s Amazon’s way of ensuring the API serves their affiliate program’s interests.

For developers who need Amazon product data without the affiliate prerequisite, web scraping is the practical path. The tooling has matured — managed scraping platforms handle the infrastructure complexity that used to require dedicated engineering teams.

Start with the API if you qualify. Fall back to scraping if you don’t. Either way, build your application — don’t let access gates stop you.

Struggled with PA-API access requirements? Found a better alternative? Share your experience below.

GitHub API Rate Limits in 2026: When Web Scraping Is the Better Choice

agenthustler — Thu, 30 Apr 2026 08:00:22 +0000

GitHub API Rate Limits: The Numbers That Block Your Project

GitHub’s REST API is one of the most generous public APIs out there — until it isn’t. At 5,000 requests per hour (authenticated) or a mere 60 requests per hour (unauthenticated), developers routinely hit walls when building anything beyond basic integrations.

If you’re doing repository analysis, tracking open-source trends, monitoring competitor activity, or aggregating data across thousands of repos — you’ll burn through that quota in minutes.

Let’s look at when the API is sufficient, when it’s not, and when web scraping becomes the pragmatic alternative.

GitHub API Rate Limits Explained (2026)

Tier	Rate Limit	Auth Required	Best For
Unauthenticated	60 req/hr	No	Quick lookups
Personal Access Token	5,000 req/hr	Yes	Standard dev work
GitHub App	5,000 req/hr + 50/repo	Yes	Org integrations
Enterprise	15,000 req/hr	Yes	Large-scale use

Sounds generous until you do the math:

# How fast can you exhaust 5,000 requests?

# Scenario: Analyze top 1,000 Python repos
requests_per_repo = 5  # repo info + contributors + languages + commits + issues
total_requests = 1000 * 5  # = 5,000
# Result: One scan = entire hourly quota

# Scenario: Monitor 200 repos for new releases
checks_per_hour = 200 * 1  # = 200 per cycle
cycles_per_hour = 5000 / 200  # = 25 cycles/hr (one every 2.4 min)
# Seems OK, but add commit history and you’re cooked

What the API Gives You (and What It Doesn’t)

GitHub’s API is excellent for structured data:

Repository metadata, stars, forks
Issues and pull requests
Commit history (paginated)
User profiles and contributions
Release and tag information

But several things are not available or practical through the API:

Trending repositories — no API endpoint for GitHub Trending
Search ranking factors — can’t see why repos rank where they do
Contribution graphs at scale — rate-limited per-user fetch
Topic/tag aggregations — limited search API (30 req/min)
Bulk profile data — fetching 10K developer profiles = 2+ hours

Real-World Rate Limit Pain Points

import requests
import time

token = "ghp_your_token_here"
headers = {"Authorization": f"token {token}"}

def check_rate_limit():
    r = requests.get("https://api.github.com/rate_limit", headers=headers)
    data = r.json()
    remaining = data["resources"]["core"]["remaining"]
    reset_time = data["resources"]["core"]["reset"]
    return remaining, reset_time

remaining, reset = check_rate_limit()
print(f"Remaining: {remaining}/5000")
print(f"Reset in: {reset - time.time():.0f} seconds")

# The dreaded 403
# {
#   "message": "API rate limit exceeded for user ID 12345.",
#   "documentation_url": "https://docs.github.com/rest/overview/rate-limits-for-the-rest-api"
# }

When you hit that 403, your options are:

Wait — up to 60 minutes for reset
Use GraphQL — separate 5,000-point budget, but complex queries cost more points
Multiple tokens — technically against ToS
Web scraping — for data the API limits or doesn’t expose

When Web Scraping Makes More Sense

Web scraping GitHub works best for:

1. Trending Repositories

GitHub’s trending page has no API. Period.

from bs4 import BeautifulSoup
import requests

def get_trending(language="python", since="daily"):
    url = f"https://github.com/trending/{language}?since={since}"
    resp = requests.get(url)
    soup = BeautifulSoup(resp.text, "html.parser")

    repos = []
    for article in soup.select("article.Box-row"):
        name = article.select_one("h2 a").text.strip()
        description = article.select_one("p")
        stars = article.select_one(".Link--muted.d-inline-block.mr-3")
        repos.append({
            "name": name,
            "description": description.text.strip() if description else "",
            "stars_today": stars.text.strip() if stars else "0"
        })
    return repos

trending = get_trending("python", "weekly")
for repo in trending[:5]:
    print(f"{repo['name']} — {repo['stars_today']}")

2. Bulk Data Collection Without Rate Limits

Scraping doesn’t have a 5,000/hour cap — you’re limited only by request pacing and proxy infrastructure.

3. Data the API Doesn’t Expose

Repository traffic insights (normally owner-only)
Dependency graphs in full
Community health metrics across many repos

Scaling GitHub Scraping

For anything beyond basic scraping, you need to handle:

GitHub’s bot detection
JavaScript-rendered content (some pages use React)
Session management
Respectful rate limiting (don’t hammer their servers)

Managed scraping tools handle this. This GitHub scraper on Apify manages proxy rotation and rendering for bulk data extraction:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("cryptosignals/github-scraper").call(
    run_input={
        "searchQuery": "machine learning",
        "language": "python",
        "maxRepos": 500,
        "includeReadme": True
    }
)

for repo in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{repo['fullName']} | {repo['stars']} stars")

API vs Scraping: Decision Matrix

Use Case	Best Approach	Why
Single repo data	API	Fast, structured, within limits
CI/CD integration	API	Real-time webhooks available
Trending repos	Scraping	No API endpoint exists
1000+ repo analysis	Scraping	API quota exhausted in minutes
User profile aggregation	Scraping	Bulk fetching is rate-limited
Commit monitoring (few repos)	API	Efficient with conditional requests
Cross-platform comparison	Scraping	Need to combine multiple sources

Hybrid Approach: Best of Both

The smartest strategy combines both:

def get_repo_data(owner, repo, token):
    # Use API for structured data within limits
    api_data = fetch_from_api(owner, repo, token)

    # Use scraping for data API doesn’t provide
    if api_data.get("rate_limited"):
        return fetch_from_scraper(owner, repo)

    # Enrich with scraped data
    api_data["trending_rank"] = get_trending_rank(owner, repo)
    return api_data

The Bottom Line

GitHub’s API is excellent for standard integrations and moderate-scale use. But for data analysis, market research, trend tracking, and bulk operations, the rate limits become a genuine blocker.

Web scraping isn’t a replacement for the API — it’s a complement for the cases where 5,000 requests per hour simply isn’t enough, or where the data you need doesn’t have an API endpoint at all.

For production-grade GitHub data collection at scale, managed scraping solutions save weeks of infrastructure work.

Hit GitHub rate limits on a project? What workaround did you use? Share in the comments.

Glassdoor API in 2026: Why Developers Are Switching to Web Scraping

agenthustler — Wed, 29 Apr 2026 08:00:14 +0000

Glassdoor API in 2026: The Landscape Has Changed

If you’ve tried accessing Glassdoor’s API recently, you already know: the public API is gone. Glassdoor shut down open developer access and now only offers data through enterprise partnerships with undisclosed pricing.

This leaves thousands of developers, recruiters, and data analysts in the dark. Whether you’re building a salary comparison tool, analyzing company reviews for investment research, or aggregating job market data — the official path is effectively closed.

Let’s break down what happened, what your options actually are in 2026, and why web scraping has become the practical alternative.

What Happened to the Glassdoor API?

Glassdoor originally offered a public API that let developers access:

Company reviews and ratings
Salary estimates by role and location
Job listings
Interview questions and experiences

In 2024, Glassdoor (now owned by Recruit Holdings alongside Indeed) restricted API access to enterprise partners only. There’s no public documentation, no free tier, no developer signup page.

The reasoning? Data monetization. Glassdoor’s salary and review data is their core asset, and they’ve decided to gate it behind B2B contracts.

Official API vs Web Scraping: Direct Comparison

Feature	Glassdoor API (Enterprise)	Web Scraping
Access	Enterprise partnership only	Open to anyone
Cost	Custom pricing ($$$$)	Infrastructure costs only
Data available	Structured JSON	Requires parsing
Rate limits	Contract-dependent	Self-managed
Salary data	Yes	Yes
Company reviews	Yes	Yes
Real-time data	Near real-time	On-demand
Setup time	Weeks (sales process)	Hours
Legal clarity	Licensed	Gray area (public data)

When the API Made Sense (and When It Doesn’t)

The enterprise API still makes sense if you’re a large HR tech company with budget and a direct relationship with Glassdoor. For everyone else — individual developers, startups, researchers — the API path is a dead end.

Here’s the reality check:

No public signup exists
No pricing page exists
No documentation is available
Response times for partnership inquiries: weeks to months

The Web Scraping Alternative

Web scraping lets you extract the same data Glassdoor displays publicly on their website. Here’s a basic Python example of what the data extraction looks like:

import requests
from bs4 import BeautifulSoup
import json

def get_glassdoor_company(company_url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }

    response = requests.get(company_url, headers=headers)
    soup = BeautifulSoup(response.text, "html.parser")

    # Extract rating
    rating_elem = soup.select_one("[data-test='rating']")
    rating = rating_elem.text if rating_elem else "N/A"

    # Extract review count
    review_elem = soup.select_one(".reviews-count")
    reviews = review_elem.text if review_elem else "N/A"

    return {
        "rating": rating,
        "total_reviews": reviews,
    }

data = get_glassdoor_company("https://glassdoor.com/Overview/company-overview.htm")
print(json.dumps(data, indent=2))

This works for small-scale use, but Glassdoor has aggressive anti-bot measures:

CAPTCHAs after a few requests
IP blocking
JavaScript-rendered content
Session validation

Scaling Glassdoor Data Collection

For production use cases, you need infrastructure that handles these challenges. Key requirements:

Rotating proxies — residential proxies work best for Glassdoor
Browser automation — much of Glassdoor’s content loads via JavaScript
CAPTCHA handling — automated solving or avoidance strategies
Rate management — respectful request pacing to avoid blocks

Rather than building all of this yourself, managed scraping tools handle the infrastructure. For example, this Glassdoor scraper on Apify handles proxy rotation, browser rendering, and anti-bot bypasses out of the box.

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("cryptosignals/glassdoor-scraper").call(
    run_input={
        "searchTerms": ["software engineer"],
        "location": "San Francisco, CA",
        "maxResults": 100
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{item['company']} - Rating: {item['rating']} - Salary: {item.get('salary')}")

What Data Can You Actually Get?

Through web scraping, you can extract everything Glassdoor shows publicly:

Company ratings (overall, culture, work-life balance, compensation)
Individual reviews with pros, cons, and advice to management
Salary reports by role, location, and experience level
Interview experiences including difficulty ratings and questions asked
Job listings with salary estimates
CEO approval ratings

Legal Considerations

Scraping publicly available data has been largely upheld in US courts (see hiQ Labs v. LinkedIn, 2022). However:

Respect robots.txt directives
Don’t bypass authentication walls
Don’t overload servers with aggressive request rates
Check Glassdoor’s Terms of Service for your jurisdiction
Use data responsibly — don’t republish raw review content

The Bottom Line

Glassdoor’s decision to gate their API behind enterprise contracts makes business sense for them, but it’s left the developer community without a practical option. Web scraping fills that gap for salary research, market analysis, and recruitment data needs.

If you’re building something that needs Glassdoor data in 2026, your realistic options are:

Enterprise partnership — if you have the budget and patience
Web scraping — for everyone else
Alternative data sources — LinkedIn, Levels.fyi, Blind (each with their own limitations)

The tooling for option 2 has matured significantly. What used to take weeks of proxy management and CAPTCHA solving can now be handled by managed scraping platforms in a few lines of code.

What’s your experience with Glassdoor data access? Have you found alternative approaches? Drop a comment below.