DEV Community: Emmanuel Uchenna

How to get Twitter (X) data for sentiment analysis

Emmanuel Uchenna — Fri, 03 Apr 2026 14:40:48 +0000

When a product fails publicly, Twitter (X) is usually the first place it shows up. Not the press. Not review sites. Twitter. Within hours, a single product flaw can spawn thousands of tweets, drive a hashtag into the top ten trends in three countries, and permanently shift how a brand is perceived.

That's why researchers, brand managers, journalists, and quantitative analysts all want the same thing: reliable Twitter data, fast, and at scale.

The problem is that getting that data has never been harder or more expensive. Twitter's own Application Programming Interface (API) now costs anywhere from $100 to $5,000 per month, and even then, it doesn't give you clean, structured trend data. Do-it-yourself scrapers require proxies, constant maintenance, and a tolerance for early-morning debugging sessions when Twitter changes its UI without warning.

This article is about a smarter path. You'll learn what kinds of Twitter data are most useful for sentiment analysis, how the main collection methods compare, and how to get structured, real-time Twitter trend data using the Twitter (X) Trends Scraper (PPE) on Apify - in under five minutes, with no setup.

What is Twitter sentiment analysis?

Sentiment analysis is the process of using natural language processing (NLP) or large language models (LLMs) to automatically classify the emotional tone behind a piece of text - typically as positive, negative, or neutral. When applied to Twitter, it turns raw social data into measurable public opinion signals.

Twitter (X) is uniquely valuable for this because people say what they actually think there. Unlike LinkedIn, where professional image management dominates, or product review sites, where feedback is structured and often incentivized, Twitter is where unfiltered reactions occur in real-time. A product launch, a policy announcement, a celebrity controversy - they all land on Twitter first.

Two distinct types of Twitter data matter for sentiment analysis:

Tweet content - the actual text of individual tweets, replies, quote-tweets, and threads posted about a topic, keyword, or brand. This is what most sentiment analysis tutorials focus on. You scrape tweets, run them through a classifier, and obtain a sentiment label for each tweet.
Trending topic data - the ranked list of hashtags and topics that are dominating conversation in a specific location at a specific point in time. This includes the trend rank, name, tweet count, and hourly history. I wrote an article on how to get Twitter (X) Trends Without Breaking the Bank.

Why trend data is an underrated input for sentiment analysis

Here's a question most sentiment analysis tutorials skip: what if you don't know which topic to analyze yet?

Tweet-level sentiment analysis works well when you already have a clear target - a brand name, a product, a specific hashtag. But trend data answers a different and often more valuable question: What are people talking about right now, and how intense is that conversation? That intensity is itself a sentiment signal.

Consider this: #ProductX is trending in São Paulo with 45k tweets and rising. You don't need to read a single tweet to know something significant is happening. The trend data alone indicate that a significant number of people are discussing this topic simultaneously, which almost always reflects a strong emotional reaction - positive or negative.

Here's what trend data actually reveals:

Tweet count as a proxy for emotional intensity. High tweet counts on a trending topic consistently correlate with strong emotional reactions. A hashtag with 80k tweets is generating far more sentiment "energy" than one with 2k. That count becomes your first filter for what's worth analyzing in depth.

Hourly trend history reveals sentiment momentum. The Twitter (X) Trends Scraper (PPE) captures trends across hourly snapshots, not just the current moment. A trend that appears at rank 40 with 5k tweets and climbs to rank 3 with 50k tweets in two hours signals something going viral. A trend that hits rank 1 and then rapidly falls often signals a controversy that burned hot and fast. These patterns are distinct and actionable.

Location-based trends reveal geographic sentiment clusters. A consumer product campaign trending in New York but not in London, or a political hashtag dominating Lagos but absent in Nairobi, gives you geographic precision that tweet-level analysis rarely provides without significant pre-processing.

Practical use cases where trend data drives sentiment decisions:

Brand monitoring: Catch a PR crisis developing before it peaks, while you still have time to respond.
Product launch tracking: See whether your campaign is gaining traction geographically, not just globally.
Competitor intelligence: Know the moment your competitor's brand starts trending - and with what intensity.
Academic research: Track regional public opinion on political events, elections, or policy announcements.
Content strategy: Identify topics with proven high engagement before committing creative resources to them.

So what does an effective data collection strategy actually look like?

Methods for getting Twitter (X) data

There are three realistic options for collecting Twitter data at scale. Here's an honest comparison.

The official X API

The X API is Twitter's own data access layer. It gives you structured, compliant access to tweets, user profiles, and some trend data via REST and streaming endpoints.

The problem is the pricing model. The free tier is limited to 1,500 tweets per month for write-only access - virtually useless for sentiment analysis. The Basic plan costs $100.00 per month and gives you limited read access. The Pro plan, which provides meaningful volume, costs $5,000.00 per month. Enterprise pricing isn't even listed publicly.

For trend data specifically, the API provides access to the trends/place endpoint, but it's been progressively restricted alongside the broader API changes since 2023.

Best for: Enterprise teams with compliance requirements who need guaranteed terms-of-service alignment and have the budget to justify it.

DIY web scraping

You could write your own scraper using tools like Selenium or Playwright to automate a browser, log in to Twitter, and extract trend data from the web interface.

The appeal is obvious: potentially free and fully flexible. The reality is less appealing. Twitter's front-end changes frequently, and every UI update risks breaking your scraper. To avoid IP blocks, you'll need rotating residential proxies, which can cost hundreds of dollars per month on their own. And all of that assumes you have the time and technical skill to build and maintain the scraper indefinitely.

Best for: Developers who treat scraper maintenance as a learning exercise rather than a cost.

Ready-made Apify Actors

Apify Actors are pre-built, cloud-hosted scrapers maintained by their developers and hosted on the Apify platform. They handle proxy management, authentication avoidance, output formatting, and cloud execution - so you don't have to.

The Twitter (X) Trends Scraper (PPE) is exactly this kind of tool. It was built specifically to extract real-time trending topic data from Twitter, supports granular location targeting across hundreds of countries and cities, and requires zero configuration beyond selecting your location.

Operating on a pay-per-event model (starting from $1.00 per 1,000 results), it is a fraction of the cost of the Pro API plan - and delivers trend data the API doesn't even expose cleanly. You only pay for the successful scrapes you actually perform.

Best for: Researchers, marketers, and developers who need reliable data without infrastructure overhead or expensive monthly subscriptions.

At a glance: method comparison

	Official X API	DIY scraping	Apify Actor (PPE)
Cost	$100–$5,000+/month	"Free" + proxy costs	From $1.00 per 1,000 results
Trend data quality	Limited, restricted since 2023	Variable (breaks on UI changes)	Structured, real-time, hourly history
Setup complexity	High (API keys, OAuth, rate limits)	Very high (proxies, parsing, maintenance)	None (click to run)
Maintenance burden	Low (API is stable until deprecated)	High (breaks on any UI update)	None (maintained by developer)
Location targeting	Country-level only	Depends on implementation	300+ countries and cities
Structured output	Yes (JSON)	Manual (requires parsing)	Yes (JSON, CSV, Excel, Sheets)
ToS compliance	✅ Fully compliant	⚠️ Gray area	✅ Public data only
Best for	Enterprise compliance-first teams	Developers who enjoy scraper maintenance	Researchers, marketers, and developers

How to use the Twitter (X) Trends Scraper (PPE) to get data for sentiment analysis

The Twitter (X) Trends Scraper (PPE) extracts real-time trending topics from Twitter (X) for any country or city in its location list. It captures trend rank, hashtag names, tweet counts, direct search links, and a rolling hourly history - making it directly useful as a sentiment intensity tracker rather than just a static snapshot tool.

Here's how to get your first dataset.

Step 1: Access the Actor

Go to Apify Store and search for "Twitter (X) Trends Scraper (PPE)", or navigate directly to apify.com/eunit/x-twitter-trends-ppe.

If you don't already have an Apify account, sign up for free. No credit card is required to start the trial. Once on the Actor page, click Try for free.

Step 2: Configure your input

The input form has a single field: Country / City. This simplicity is intentional - the Actor is designed to return complete trend data for your selected location in one run.

The dropdown supports a wide range of locations, from top-level country selections like Worldwide or united-kingdom, to city-level targeting such as united-states/new-york, nigeria/lagos, or india/mumbai.

For sentiment analysis purposes, location selection strategy matters:

If you're monitoring a brand or product, start with Worldwide to establish whether the topic is trending globally.
Then re-run with specific market locations (e.g., united-states, united-kingdom, australia) to understand where the conversation is concentrated.

The equivalent JSON input looks like this:

{
  "country": "united-states/new-york"
}

You can also pass this input directly via the API or Apify client libraries, which is covered in the automation section below.

Step 3: Run and inspect the output

Click Run. The Actor typically completes in seconds and returns a structured JSON dataset. Here's what each key field contains:

scraped_at - the UTC timestamp of the run. Critical for tracking data freshness and building time-series datasets.
country_input - the location string you submitted, useful for multi-location runs.
timeline - an array of hourly snapshots, each containing up to 50 ranked trends. Each trend includes its rank, name (the hashtag or topic), tweet_count, and a direct search link. This is the richest output for sentiment analysis.
tag_cloud - a broader set of trending terms for that location, useful for topic discovery.
table_data - the current top 50 ranked trends in a flat table format, ideal for quick dashboards.

A sample of the timeline output:

{
  "scraped_at": "2026-04-03T10:30:00.000Z",
  "country_input": "united-states/new-york",
  "timeline": [
    {
      "datetime": "2 minutes ago",
      "timestamp": "1743672600",
      "trends": [
        {
          "rank": 1,
          "name": "#TechLayoffs",
          "link": "https://twitter.com/search?q=%23TechLayoffs",
          "tweet_count": "82K"
        },
        {
          "rank": 2,
          "name": "New York Giants",
          "link": "https://twitter.com/search?q=%22New+York+Giants%22",
          "tweet_count": "14K"
        }
      ]
    }
  ]
}

Step 4: Export and use the data

Once the run completes, Apify Console gives you several export options directly from the dataset view: JSON, CSV, Excel, XML, and a direct Google Sheets integration.

For sentiment analysis pipelines, JSON or CSV are the most practical formats. JSON preserves the nested timeline structure (essential if you're tracking momentum across hourly snapshots), while CSV flattens the data for easy import into tools like Excel, Google Sheets, or pandas.

Tip: For a fully automated pipeline - where you scrape trends, then instantly analyze the sentiment of trending topics without writing any code - connect this Actor with LLM Dataset Processor directly in Apify Console. Point the LLM processor at your trend dataset, define a prompt like "Classify the likely sentiment associated with this trending hashtag as Positive, Negative, or Neutral based on context", and export the enriched results to Google Sheets automatically.

For developers: automating the pipeline

If you want to integrate live Twitter trend data into your own application or run it on a schedule, Apify provides client libraries for Python and JavaScript that make this straightforward.

First, install the Python client:

pip install apify-client

Then run the Actor programmatically:

from apify_client import ApifyClient

# Initialize the client with your Apify API token
client = ApifyClient("<YOUR_API_TOKEN>")

# Define your input
run_input = {
    "country": "united-states/new-york",
}

# Run the Actor and wait for completion
run = client.actor("eunit/x-twitter-trends-ppe").call(run_input=run_input)

# Fetch and process the results
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Trends for: {item['country_input']}")
    for trend in item["timeline"][0]["trends"][:10]:
        print(f"  #{trend['rank']} {trend['name']} - {trend['tweet_count']} tweets")

For JavaScript/Node.js projects, the pattern is identical using the apify-client npm package:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });

const run = await client.actor("eunit/x-twitter-trends-ppe").call({
    country: "united-states/new-york",
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach(item => {
    item.timeline[0].trends.slice(0, 10).forEach(trend => {
        console.log(`#${trend.rank} ${trend.name} - ${trend.tweet_count}`);
    });
});

Scheduling for longitudinal datasets. The real power for sentiment analysis is in running the Actor on a regular cadence - say, every hour - and storing the results in a database or data warehouse. This builds a time-series dataset of trend rank and tweet count, which lets you track sentiment momentum over hours, days, or weeks. Apify's built-in scheduler handles this without external cron jobs or infrastructure.

AI agent integration. If you're building an AI agent that needs to be aware of current social trends, the Actor supports Model Context Protocol (MCP) server integration. Your agent can query live trend data and use it as context for downstream reasoning tasks.

Is it legal to scrape Twitter (X) trends?

This is a fair question, and it deserves a direct answer.

The Twitter (X) Trends Scraper (PPE) collects publicly available, factual data - specifically the list of trending hashtags and topics that Twitter displays to any unauthenticated visitor. It does not scrape private profiles, direct messages, content behind a login wall, or any personally identifiable information.

Scraping publicly available factual data is generally considered legal in most jurisdictions. The landmark hiQ Labs v. LinkedIn case in the United States established that scraping publicly available data does not constitute a violation of the Computer Fraud and Abuse Act (CFAA). Trend data - hashtag names, rankings, and aggregate tweet counts - is about as factual and public as data gets.

That said, you should always review the Terms of Service of any platform you interact with, and ensure your use case complies with applicable regulations. The General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) govern personal data, not aggregated metadata like trend rankings and hashtag names. As long as you're collecting and using trend data responsibly, you're on solid ground.

Wrapping Up

The hardest part of any Twitter sentiment analysis project isn't the analysis. It's getting clean, reliable, structured data to analyze. The official API is prohibitively expensive for most teams. DIY scrapers are fragile and require time-consuming maintenance. And most sentiment analysis tutorials skip past the data collection problem entirely.

The Twitter (X) Trends Scraper (PPE) fills that gap directly. It gives you real-time Twitter trend data - ranked hashtags, tweet counts, hourly history, and location-level granularity - in a structured format that plugs directly into your sentiment pipeline, whether that's a Python NLP script, an LLM-powered classifier, or a no-code Apify workflow.

The free trial requires no credit card. A single run takes about ten seconds. And the data you get back can be the difference between catching a PR crisis before it peaks and reading about it in the Monday morning report.

Try the Twitter (X) Trends Scraper (PPE) on Apify and start turning Twitter's real-time noise into a clear, actionable signal.

Frequently asked questions

How do you get data from Twitter (X) for sentiment analysis?

The three main options are the official X API (expensive, limited trend data), custom web scrapers (fragile, requires proxy infrastructure), and ready-made tools like Apify Actors. The Twitter (X) Trends Scraper (PPE) is the most practical option for sentiment-focused use cases, as it returns structured trend data, including tweet counts and hourly history with zero configuration.

What is the best tool for scraping Twitter trends in 2026?

The Twitter (X) Trends Scraper (PPE) on Apify is the most practical option for most use cases. It supports granular location data for hundreds of countries and cities, requires no proxy setup, and uses a fair pay-per-event pricing model (from $1.00 / 1,000 results) - a fraction of the cost of the official API.

Is it legal to scrape Twitter (X) for sentiment analysis data?

Scraping publicly available trend data is generally legal. Trend rankings and hashtag names are factual, public-facing data, not personal information subject to GDPR or CCPA. However, you should always review the platform's Terms of Service and ensure your use case complies with local regulations.

How much does it cost to get Twitter data for sentiment analysis?

The official X API starts at $100 per month for basic access and $5,000 per month for professional volume. The Twitter (X) Trends Scraper (PPE) on Apify operates on a pay-per-event basis, meaning you only pay for the data you successfully scrape. If you need tweet-level data in addition to trends, additional Actors on the Apify Store are available, each with their own pricing.

Can I get historical Twitter trend data?

Twitter does not provide historical trend data through its official API. The Twitter (X) Trends Scraper (PPE) captures trends at the time of each run, including the hourly timeline snapshot that Twitter maintains for the current period. To build a historical dataset, schedule the Actor to run at regular intervals (e.g., hourly) and store each run's output in your own database. Apify Console also retains your run history and datasets for later access.

How to scrape TikTok search results: A complete guide for 2026

Emmanuel Uchenna — Sat, 21 Mar 2026 14:58:52 +0000

TikTok has transformed from a simple video-sharing app into a global cultural engine. With over 1 billion active users, the platform dictates what we listen to, what we buy, and how we communicate. For businesses, marketers, and researchers, TikTok is not just entertainment; it is a repository of real-time consumer sentiment and trend data.

However, getting that data is a top challenge as TikTok employs some of the most sophisticated anti-scraping technologies in the world. In this article, we will explore why TikTok data is essential, the technical barriers to getting it, and how you can use the TikTok Search Scraper to automate your data collection at scale.

The importance of TikTok data in modern marketing

In the past, brands relied on slow-moving market research surveys. Today, they rely on TikTok in 2026 as a primary growth engine, utilizing authentic, low-polish content, behind-the-scenes storytelling, and trending sounds to drive viral engagement. With over 5 billion daily users, it is crucial for building brand loyalty and reaching audiences who spend a significant amount of time on social media. The speed at which a trend moves from a single video to a global phenomenon is unprecedented.

Trend tracking and cultural shifts

By scraping TikTok search results, you can identify emerging trends before they hit the mainstream. For example, a search for "sustainable fashion" might reveal specific local hashtags or challenges that are gaining momentum. Tracking these shifts allows you to adjust your content strategy in real-time, ensuring your brand remains relevant.

Audience behavior and sentiment analysis

Comments, view counts, and video descriptions provide insight into how users feel about specific topics. Analyzing these data points helps you understand the language your target audience uses, their pain points, and what kind of content they find engaging.

Competitor and influencer benchmarking

Who are the top creators in your niche? What music are they using? How often do they post? You can answer all these questions by scraping search results for relevant keywords. This data helps you benchmark your performance against competitors and identify potential influencers for future collaborations.

Key aspects of brand reliance on TikTok

Authenticity over production: Brands, particularly in fashion and beauty, succeed by using smartphone cameras to show raw "before and after" transformations rather than polished ads.
Driving growth: Small businesses in sectors like fashion, fintech, and beauty use TikTok as their primary marketing channel to reach new audiences.
Immersion & trends: Successful brands like Sandro Paris and Yahoo create immersive, trend-driven content to stay relevant in pop culture.
Massive reach & user-generated content (UGC): Major brands like ESPN hold millions of followers, while companies increasingly rely on UGC to build trust.
Strategic outsourcing: To keep up with the pace of trends, many companies outsource their TikTok creative to specialists

Why traditional scraping methods fail

If you have tried to build a simple Python script to scrape TikTok, you have likely encountered immediate blocks or endless captchas. TikTok's security infrastructure is designed to distinguish between automated bots and real human users.

Dynamic content and JavaScript heavy lifting

TikTok is a single-page application (SPA) that heavily relies on JavaScript to load content. A basic HTTP request will return an empty shell of a page. To get actual video data, you need a browser-based scraper that can render the DOM and handle infinite scrolling.

Advanced bot detection

TikTok checks for several technical signals to identify automation, among which include:

Navigator.webdriver: Many automated browsers leave this flag set to true, making them easy to spot.
Fingerprinting: Systems analyze your browser's font list, canvas rendering, and hardware identifiers to see if they match a standard device.
Client Hints: Modern anti-bot systems check sec-ch-ua headers. If your User-Agent indicates that you are on Chrome 123 but your headers say something else, you will be blocked.

The solution: TikTok Search Scraper

The TikTok Search Scraper is a professional-grade Apify Actor designed to bypass these barriers. It does not just "read" the page; it interacts with it as a human would.

Key features for maximum resilience

Human-like behavior: The scraper mimics real user interaction. It does not just jump to the bottom of the page; it performs smooth, variable-speed scrolling. It even "pauses" to simulate a user watching a video, which significantly reduces the risk of triggering behavioral blocks.
Browser profile rotation: Every run can use a different browser profile. These profiles include curated User-Agents, realistic viewports, and synchronized Client Hints. This makes each request appear to be coming from a unique, legitimate device.
Stealth technology: By leveraging the stealth plugin, the Actor masks its automated nature, overriding the navigator.webdriver property and other common detection vectors.
Rehydration script fallback: TikTok stores its initial data in a JSON blob called a rehydration script. If the DOM structure changes or is temporarily hidden, the Actor can pull data directly from this script, ensuring high reliability even during platform updates.

What data can you extract?

The TikTok Search Scraper provides a comprehensive set of data points for every video result:

Field	Description
Video ID	The unique identifier for the video (`id`).
Video URL	The direct link to the TikTok video (`url`).
Description	The full caption, including hashtags and mentions (`description`).
View Count	The number of times the video has been played (`views`).
Likes	The number of likes the video has received (`likes`).
Comments	The number of comments on the video (`comments`).
Shares	The number of times the video has been shared (`shares`).
Author Info	Username (`uniqueId`), nickname, avatar URL, verification status (`verified`), and `followers` count (`author`).
Music Title	The song or sound used in the video (`music`).
Music Author	The creator of the music or sound used in the video (`musicAuthor`).
Hashtags	A list of hashtags included in the video (`hashtags`).
Duration	The length or duration of the video (`duration`).
Timestamp	The date and time the video was published (`timestamp`).
Scraped At	The exact date and time the data was extracted (`scrapedAt`).
Source	The method or source used to extract the data (`source`).

Here is a sample of the extracted JSON output:

 {
    "id": "7580436439..........",
    "url": "https://www.tiktok.com/@techuser112/video/7580436439..........",
    "description": "Comment \"tech\" if you want a road map on how to get into any of these careers",
    "views": "11K",
    "likes": "",
    "comments": "",
    "shares": "",
    "author": {
      "uniqueId": "techuser112",
      "nickname": "tech with Lesedi’s profile",
      "avatar": "https://p16-sign-sg.tiktokcdn.com/tos-alisg-avt-0068/9f813384bac2a.......",
      "verified": false,
      "followers": ""
 },
    "music": "Lesedi with Adrian’s 7AM - Slowed + Reverb",
    "musicAuthor": "",
    "hashtags": ["technology", "techjobs", "techreview", "techroadmap"],
    "duration": "2025-12-5",
    "timestamp": "",
    "scrapedAt": "2026-03-21T10:47:10.773Z",
    "source": "dom"
 }

How to use the scraper on the Apify platform

Running the TikTok Search Scraper on the Apify platform is a simple four-step process. You do not need any coding knowledge to get started.

Step 1: Set your queries

Once you are on the TikTok Search Scraper page, navigate to the Input tab. In the Search Queries field, enter an array of search terms you want to analyze. For example: ["tech review", "laptop unboxing", "coding tips"]. This screenshot below shows what you need to get started:

Step 2: Define your limits

Use the maxItems field to specify how many videos you want to collect per query. If you are conducting a deep dive, consider setting this to 500. For a quick pulse check, 20 results might be enough.

Step 3: Configure proxy settings

TikTok has strict regional and IP-based limits. To avoid being blocked, always use Apify Proxy. We highly recommend using Residential proxies, as they use IP addresses assigned to real households, making your scraping activity indistinguishable from regular traffic. Learn more about ApifyProxy from the Apify documentation.

Step 4: Run and export

Click the Start button at the bottom of the screen. You can monitor the progress in the Log tab. Once the run is finished, go to the Storage tab to export your data. You can choose from several formats, e.g., JSON, CSV, Excel, or XML.

Optimization and cost efficiency with Pay-Per-Event

Many scrapers charge based on the computing power they consume, which can be unpredictable and vary significantly. The TikTok Search Scraper uses a more transparent model: Pay-Per-Event (PPE).

With PPE, you are charged a fixed price of $0.29 for every 1,000 video results. This means:

You only pay for the data you actually get.
Your costs are perfectly predictable.
You do not pay for the platform's overhead or idle time.

Use cases: Putting the data to work

Influencer marketing

Agencies use the scraper to find rising stars. By searching for niche keywords, they can identify creators who have high engagement but have not yet been "discovered" by major brands.

Market research

Pharma and retail companies scrape TikTok to see how people are using their products. Are they finding new use cases? Are they complaining about a specific feature? This qualitative data is often more honest than what you get in a focus group.

Trend capitalization

Media companies track trending hashtags to decide what topics to cover next. By seeing which videos are gaining the most views in a short period, they can jump on trends while they are still hot.

For Developer: Automate with JavaScript

If you are a developer looking to integrate the TikTok Search Scraper directly into your projects, you can use the official Apify API JavaScript/TypeScript client.

Installation

First, install the apify-client package in your project:

npm install apify-client

Usage Example

To get started, initialize the client with your Apify API token and pass your search queries. Here is an example script showing how to scrape data for the "AI coding" query:

import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
 token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "AI coding"
 ],
    "maxItems": 5,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
 ]
 }
};

// Run the Actor and wait for it to finish
const run = await client.actor("eunit/tiktok-search-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
 console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Legal and ethical considerations

When scraping TikTok, it is important to act responsibly.

Respect privacy: Only scrape publicly available data. Never attempt to access private accounts or unauthorized sections of the platform.
Avoid overloading: Use the scraper's built-in delays and human-like interactions to avoid putting unnecessary strain on TikTok's servers.
Comply with regulations: Ensure your use of the data complies with local regulations, such as GDPR in the EU or CCPA in California.

Wrapping Up

TikTok is a powerful source of real-time intelligence, but accessing it requires a professional approach. The TikTok Search Scraper on the Apify platform provides the stealth, speed, and reliability you need to turn raw social media activity into actionable business insights.

Start your first run today and stop guessing about the latest trends.

Happy automating!

How to actually start your fitness journey and stick to it (with the FitJourney.app platform)

Emmanuel Uchenna — Sat, 07 Mar 2026 12:46:36 +0000

You have been thinking about it for months. Maybe even years. That expensive gym membership is gathering dust, your workout clothes still have the tags on them, and you have a growing, endless list of reasons why “tomorrow” will be the perfect day to begin. Here is the honest, unfiltered truth about health and wellness: starting a fitness journey is absolutely the hardest part, but it is also the most transformative decision you will ever make for your body and your mind.

The first workout is always harder mentally than it is physically. We often build up incredible, insurmountable barriers in our minds. When you finally take that first step, you are not just beginning a workout routine - you are rewiring decades of self-doubt and breaking down established negative habits. However, taking that step can feel incredibly daunting because the modern fitness landscape is undeniably confusing. You have fitness influencers screaming conflicting advice, terrifying big box gyms full of confusing machines, and a dozen different apps you supposedly need to track your life.

This article is designed to cut through the noise. We are going to deeply explore why starting your health and fitness journey feels so overwhelming, the science-backed benefits of committing to a routine, and how to find the lasting fitness motivation you need. Most importantly, we will introduce the FitJourney platform, an all-in-one ecosystem that drastically simplifies progress tracking, fitness challenges, and educational content so you can finally stop planning and start doing.

Why starting your fitness journey feels so overwhelming

Obstacle 1: Analysis paralysis

You have probably researched every workout program, compared various gym memberships in your local area, and read countless articles about the ”best” way to get started. You spend hours watching videos on the optimal grip for a barbell deadlift, the exact macronutrient ratio for fat loss, or the perfect supplements to buy before you have even picked up a pair of dumbbells. Meanwhile, weeks turn into months of intense planning rather than taking any real action.

The truth about how to start a fitness journey is simple: the best program is the one you will actually follow consistently. Analysis paralysis keeps you stuck on the starting line, waiting for perfect conditions that will never arrive. You don't need to know everything to begin. You need to take the first step and learn as you go.

Obstacle 2: Fragmentation

Relying on scattered, disconnected tools completely kills your momentum. Right now, you might be using a basic notes app for tracking your sets and reps, a separate application for logging your meals and counting calories, another website for learning how to perform exercises safely, and a group chat for accountability. This fragmentation makes the simple act of working out feel like a stressful, overwhelming part-time job.

When your fitness tools are scattered across the wind, it is incredibly easy to drop the ball on one of them, which often leads to dropping the whole routine out of pure frustration. A unified, dedicated fitness tracking platform is essential for keeping everything organized and keeping you focused on the actual physical work.

Obstacle 3: Fear of judgment and unrealistic expectations

Walking into a new gym feels intimidating, especially when you are convinced that everyone else there has it perfectly figured out. Big box gyms can dramatically amplify this fear of judgment. You might worry that people are watching you struggle with a machine, or you might feel deeply embarrassed because you do not look like the fitness models prominently featured on your social media feed.

It is vital to remember that social media only shows the highlight reel, not the messy, sweaty middle of the journey. Furthermore, unrealistic expectations set you up for immediate failure. You might expect to drop 10 pounds in your first week and get discouraged when the scale barely moves. Sustainable fitness is about showing up consistently, embracing the learning curve, and rejecting the pressure to be perfect from day one.

It is vital to remember that social media only shows the highlight reel, not the messy, sweaty middle of the journey

The core benefits of committing to your fitness

Drawing inspiration from real science and established medical research, it becomes clear that committing to regular exercise is the ultimate game-changer for your entire life, far beyond just looking good in the mirror.

Physical health

The most obvious benefits of a health and fitness journey are physical. Exercise helps control weight by helping you burn calories and build active, lean muscle mass. But beyond the aesthetics, regular physical activity is a powerhouse for your internal, functional health. It vigorously combats chronic health conditions and diseases. Whether you are worried about hereditary heart disease or hoping to naturally prevent high blood pressure, being active boosts your high-density lipoprotein (HDL) cholesterol—often known as the “good” cholesterol—and decreases unhealthy triglycerides.

This one-two punch keeps your blood flowing smoothly, which significantly lowers your risk of cardiovascular diseases, type 2 diabetes, metabolic syndrome, and even certain types of cancer. Additionally, as you build muscle strength and greatly boost your endurance, you will find you have significantly more physical energy to tackle your daily chores, play with your children, and live life without getting easily winded.

Mental well-being

If you currently need a significant emotional lift or a reliable way to lower stress after a difficult, demanding day at work, a solid workout session can help tremendously. Physical activity stimulates various powerful brain chemicals, including endorphins and dopamine, that leave you feeling happier, significantly more relaxed, and noticeably less anxious. Exercise is a potent, natural antidepressant that helps you regulate cortisol levels in your body.

Longevity

Fitness is a lifelong marathon, not a frantic 30-day sprint. It is the single best investment you can make in your future self and your long-term independence. By maintaining your muscle mass and increasing your bone density through dedicated resistance training, you actively prevent the risk of debilitating falls and arthritis as you age. Getting fit is not just about extending the total years of your lifespan; it is about drastically improving your health span. This ensures that your later years are vibrant, active, and fulfilling, free from the extreme physical limitations that plague so many older adults.

How to define your "why" and set realistic goals

Before nervously lacing up your running shoes or paying for an expensive gym membership, you absolutely need to understand the psychology of goal setting and intrinsic motivation.

Finding your motivation

To continually sustain your health and fitness journey through the tough days, you must clearly define your underlying "why". Why do you genuinely want to get fit? Your motivation must be unique to you. If your initial goal is "to look good in a swimsuit," your motivation will likely plummet to zero the moment you put on a cozy sweater in the winter.

You need to look beyond surface-level aesthetics to understand your deepest, most personal reasons for getting fit. Do you deeply want to have the boundless energy to play with your children without losing your breath? Do you desperately want to reverse a frightening pre-diabetes diagnosis so you can live a long, full life? Do you want to build the physical confidence to hike a steep mountain on your next vacation? Write down your deeply rooted reasons on paper, and revisit them whenever your motivation wavers. Knowing your true "why" gives your fitness journey unshakeable, relentless purpose.

To continually sustain your health and fitness journey through the tough days, you must clearly define your underlying "why". Why do you genuinely want to get fit?

Goal setting

Establishing clear, intelligent, and achievable goals is crucial for maintaining constant motivation. However, there is a massive conceptual difference between outcome goals and behavioral goals.

An outcome goal might be something like "I want to lose 20 pounds by summer." While this is a perfectly fine desire, many complex biological factors influence the actual, literal outcome, which is outside of your direct control, such as your shifting metabolism, hormonal water weight fluctuations, and daily stress levels.

A behavioral goal is an actionable step that is 100% within your complete control, such as "I will work out three times a week for exactly 45 minutes" or "I will eat a serving of green vegetables with every single dinner." Focus your mental energy entirely on mastering behavioral goals. Celebrate each small behavioral victory, because these repeatable actions serve as tangible evidence of your commitment and will inevitably lead to the desired long-term outcome.

Consistency over perfection

Consistency beats perfection every single time. It is the golden, unbreakable rule of fitness. You don't need to execute perfectly to start a fitness journey, and you absolutely do not need to have a flawless diet to see amazing results. Life happens to all of us. Work gets crazy, the kids get sick, and natural motivation fluctuates from day to day.

If you miss a scheduled workout, it does not mean you have failed. If you eat a piece of cake at a birthday party, your progress is not magically ruined forever. Remind yourself constantly that taking a brisk 15-minute walk on a bad, stressful day is infinitely better than doing nothing at all. The people who ultimately succeed in fitness are not those who never make mistakes; they are the ones who refuse ever to quit.

Enter the FitJourney platform: your all-in-one fitness ecosystem

Now that you have a comprehensive understanding of the mental challenges, the profound medical benefits, and the resilient mindset explicitly required, you desperately need the right tools to execute your plan flawlessly. This is where the FitJourney platform comes in to revolutionize your approach.

Introducing the FitJourney.app platform

We built the FitJourney platform to permanently eliminate the confusing guesswork from your health and wellness goals. Remember the detrimental obstacle of fragmentation we discussed earlier? The FitJourney platform solves this critical issue by providing a comprehensive, all-in-one ecosystem. You no longer need five different, annoying applications to manage your health. Our integrated fitness tracking platform brings your scheduled workouts, your ongoing education, your nutrition guidelines, and your vital progress data into a single, beautiful, and highly intuitive interface.

Structured challenges

One of the biggest hurdles for nervous beginners is bravely walking into a gym and thinking, "What in the world do I do now?" The FitJourney platform removes this paralyzing anxiety entirely through our structured, expertly designed programs. By navigating to the Challenges section of the platform, you can quickly enroll in fitness challenges purposefully tailored to your specific experience level.

Whether you are tentatively stepping into a gym for the very first time in your life or you are an intermediate lifter looking to break through a plateau aggressively, our comprehensive challenges provide step-by-step guidance. You will know exactly which exercises to do, how many sets and reps to complete properly, and how to progress safely week after week.

Education and practicals

We firmly believe that education is the ultimate form of empowerment. We do not just want you to follow a generic workout routine; we desperately want you to understand exactly how your body works fully. The FitJourney platform features built-in, highly detailed educational articles and interactive practical exercises that teach you exactly how to train effectively and eat perfectly right for your dynamic goals.

A practical is a hands-on learning module that rigorously tests your knowledge in a real-world setting. You will thoroughly learn about the scientific principles of progressive overload, how to properly fuel your precious body before a grueling workout, and how to proactively prevent common injuries. This deep, expanding well of knowledge strongly ensures that you are building sustainable, intelligent habits that will truly last a lifetime, rather than just seeking an impossible 30-day quick fix.

All-in-one tracking

Seeing clear, tangible data is unequivocally the most reliable source of endless fitness motivation. When you can objectively see on a chart that you are getting stronger, running faster, and becoming healthier, everything falls into place wonderfully.

The FitJourney platform makes tracking completely effortless and highly rewarding. Your personalized Dashboard provides a clear, bird's-eye view of your current active streaks, your upcoming workouts, and your recent achievements. Meanwhile, Your Progress Tracker allows you to intimately log the specific, granular details of your journey, securely storing your daily workout data, body measurements, and athletic performance milestones over time. Seeing your personal line graph steadily trending in the right direction provides the incredibly powerful psychological reinforcement you need to keep showing up every single week.

3 simple steps to kickstart your journey today

Your fitness journey is quietly waiting for you. Not next Monday. Not after the busy holidays. Not when life miraculously gets less busy. Right today. The objectively hardest part is truly just making the bold decision that you are worth the time and physical investment. To help you aggressively overcome analysis paralysis, here are three simple, actionable steps you can confidently take right now to begin successfully.

Step 1: Get set up

Your very first action is to swiftly create your account on the FitJourney platform. The entire sign-up process is remarkably quick, intentionally. Once you are successfully logged in, take a few dedicated minutes to explore your newly personalized Dashboard. Familiarize yourself with the clean interface, set your initial user preferences, and complete your basic profile. Setting up your digital environment is a crucially important psychological cue to your brain that you are officially beginning a completely new chapter of your life.

Step 2: Join a challenge

Once your account is fully set up, purposefully navigate to the Challenges tab located in the main menu. Enthusiastically browse through the available, structured programs and select a beginner-friendly challenge that perfectly aligns with your current fitness level and your personal goals. Do not overthink it; the primary goal right now is to build the consistent, daily habit of working out.

Step 3: Track and learn

When it is finally time for your critical first workout, open up the platform, diligently follow the detailed instructions safely provided in your chosen challenge, and give it your absolute best effort. Immediately after finishing, triumphantly jump into Your Progress Tracker to carefully log your completed workout. Openly celebrate this massive, foundational win! Then, take 10 minutes to read your first educational article properly or complete an engaging practical on the platform.

Wrapping up

Actively starting your health and fitness journey is undoubtedly the hardest part of the entire process. It demands that you aggressively overcome deep-seated self-doubt, push decisively past paralyzing analysis paralysis, and bravely step into the vast unknown. But the staggering rewards—boundless daily energy, profound mental clarity, and exceptional long-term health—are ultimately worth infinitely more than the brief initial discomfort.

To predictably succeed, you must define your deeply personal "why", prioritize relentless consistency over impossible perfection, and set behavioral goals that you can easily and directly control. Furthermore, strategically leveraging the right tools is paramount. By actively using the FitJourney platform, you permanently eliminate the distracting confusion of fragmented apps and systematically replace it with a focused, all-in-one fitness ecosystem that tracks your progress and educates your mind.

Ready to definitively stop planning and start doing? Create your free account on the FitJourney platform and join your very first challenge today. Your future self will profoundly thank you for the incredible courage you showed right now.

How to scrape Telegram channels: The complete guide to extracting posts, media, and metadata

Emmanuel Uchenna — Sat, 07 Feb 2026 12:28:27 +0000

Telegram has evolved into a global powerhouse for real-time information, hosting over 1 billion active users in 2026. From niche specialized groups to massive news broadcasts with millions of subscribers, the platform is a goldmine for market researchers, AI developers, and competitive intelligence analysts. However, the platform is notoriously difficult to scrape. Its data is locked behind a mobile-first interface and strictly protected by anti-bot measures.

Manual data collection doesn't scale in today's era of automation. If you are trying to monitor trends across a hundred channels or build a dataset for a large language model, you need an automated solution. This is where Telegram Channels Scraper comes in.

In this walkthrough, we will show you how to use this advanced Apify Actor to extract everything from message text and timestamps to direct media links and engagement metrics.

Why scrape Telegram data in 2026?

Telegram is more than just a messaging app. It is a news agency, a marketplace, and a community hub. Telegram has been referred to as a "Global Digital Hub". As of 2026, Telegram has evolved far beyond a simple instant messaging tool into a massive, multifaceted digital ecosystem. It serves as a "super-app" combining social networking, information, and commerce, particularly popular for its algorithm-free broadcast capabilities.

Organizations and individual developers are increasingly looking for ways to "unlock" this siloed data for several key reasons:

1. Market research and competitive intelligence

In many industries, specifically crypto, finance, and tech, Telegram is where news breaks first. By scraping competitor channels, you can monitor their announcements, track which content resonates with their audience based on reaction counts, and analyze their growth through subscriber metrics.

2. Dataset building for AI and NLP

Generative AI models are only as good as the data on which they are trained. Telegram provides high-quality, real-time human conversation in dozens of languages. It is an ideal source for training sentiment analysis models or building niche-specific knowledge bases for AI agents.

3. Content monitoring and automation

Many businesses use Telegram scrapers to feed external dashboards or notification systems. For example, a financial analyst might scrape various "trading signals" channels to consolidate all data into a single, searchable database or a Slack notification bot.

4. Audience behavior and sentiment analysis

Understanding which emojis (reactions) users use in response to specific news can provide deep qualitative insights into brand sentiment. Tracking views per post relative to the total subscriber count helps calculate actual engagement rates.

What is the Telegram Channels Scraper?

Telegram Channels Scraper is a powerful, production-ready Apify Actor designed to extract structured content from public Telegram channels. Unlike many other tools that require you to manage your own proxies or deal with complex API authorizations, this Actor handles the technical heavy lifting for you.

Key features of the tool

QR code self-authentication: No developer API keys needed. Users authenticate with their own Telegram account by scanning a QR code directly in the console — just like linking a desktop device.
Session string persistence: After scanning the QR code once, the Actor outputs a reusable session string. Paste it into the input on future runs to skip the login step entirely.
Full engagement metrics: Extract views, reaction counts, and post links.
Media extraction: Get direct download links for images, videos, and documents.
Developer-friendly: Includes a built-in FastAPI server for real-time requests and support for Model Context Protocol (MCP) to integrate with AI agents like ChatGPT or Claude.
Pay-per-result pricing: The Actor uses Apify's Pay-Per-Event model — you only pay for what you actually get, making it highly cost-effective for both small and large-scale projects.

What data can you extract?

The Actor produces a clean, structured JSON output (also exportable to CSV, Excel, or XML). Here is the data you can expect for every post:

Field	Description
Channel Name	The display name of the Telegram channel.
Username	The @handle of the channel.
Bio & Subs	The channel's description and current subscriber count.
Post ID & Link	The unique identifier and direct URL to the message.
Date	ISO-formatted timestamp of the post.
Text	The full text content of the message.
Views	Number of views at the time of scraping.
Reactions	An object containing counts for every emoji used (e.g., 👍, ❤️, 🔥).
Media Type	Whether the post is text-only, a photo, a video, or a document.
Media URLs	Direct links to the media files.

Step-by-step walkthrough: Extracting your first dataset

Getting started with Apify Actors is easy. You don't need to write a single line of code unless you want to integrate the results into your own application.

Step 1: Find the Actor on Apify Store

Navigate to the Telegram Channels Scraper page. If you don't have an Apify account yet, you can sign up for free in just a few seconds.

Step 2: Configure the input

Once you are on the Actor page, click on the Input tab. You will see these fields:

Channels: Enter a comma-separated list of the channel usernames or links you want to target. For example: @svtvnews, https://t.me/cryptninjas.
Limit: Decide how many posts you want to extract from each channel. If you need the most recent news, 10 or 20 is usually enough. For historical research, you can set this higher.
Days back: Specify how far back in time you want to go. This ensures you don't waste resources on outdated content if you only care about the last 3 days.
Saved Session String (optional): Paste a previously generated session token here to skip the QR login on repeat runs.
2FA Password (optional): If your Telegram account has Two-Step Verification enabled, provide it here so the Actor can complete the login automatically.

Step 3: Authenticate with QR code (first run only)

On your very first run — or when no session string is provided — the Actor will display a QR code directly in the Log tab of the Apify Console. This is how you authenticate:

Open Telegram on your mobile device.
Go to Settings → Devices → Link Desktop Device.
Scan the QR code shown in the log.

Once scanned, the Actor logs your session string prominently in the console and also saves it to the Key-Value Store under the key SESSION_STRING. Copy it and paste it into the Saved Session String input field — all subsequent runs will skip the QR prompt entirely.

Note: The QR code automatically refreshes every 30 seconds if not scanned. The Actor will regenerate it up to 10 times, so you have plenty of time.

Step 4: Run the Actor

Click the Start button at the bottom of the page. You will be redirected to the Runs tab, where you can see the log in real-time. The Actor will connect to Telegram, navigate to the specified channels, and push the data into your default dataset.

Step 5: Download your results

When the run finishes, navigate to the Storage tab. You can preview the data in a table format or download it in your preferred format. Most users find CSV or Excel to be the best formats for analysis, while developers typically prefer JSON.

Advanced features for power users

This Actor is built for scale. If you are a developer or a business with high-volume needs, you can leverage its advanced capabilities.

Real-time scraping via API (Standby mode)

The Actor supports Standby mode. This means you can keep the Actor running in the background and send it HTTP requests whenever you need data. Instead of waiting for a container to boot up, you get an immediate response.

You can trigger a scrape by sending a POST request to the Actor's standard URL with the following JSON body:

{
  "channels": "@example_channel",
  "limit": 5,
  "daysBack": 1,
  "session": "YOUR_SESSION_STRING_HERE"
}

AI integration with MCP

In the era of AI agents, data extraction needs to be "AI-readable". This Actor supports the Model Context Protocol (MCP). This allows AI tools like Claude Desktop or custom GPTs to "call" the scraper as a tool. You can ask your AI, "What are the top 5 news posts from the last 24 hours on @svtvnews?" and it will use the Actor to fetch the data and summarize it for you.

Pay-Per-Event pricing: you only pay for what you get

The Actor uses Apify's Pay-Per-Event (PPE) monetization model, meaning the cost is tied directly to your results:

Actor start: A small flat fee is charged once per run to cover Telegram connection and authentication overhead.
Per result: A charge is applied for each post successfully scraped and pushed to your dataset.

Critically, the Actor is built to respect your budget limit. The moment you reach the maximum charge you have configured, the Actor stops scraping immediately — it will never do work beyond what you have agreed to pay for. This makes it far more cost-predictable than time-based or flat-rate pricing models.

Why use Apify Actors instead of building your own scraper?

You might be tempted to use a library like Telethon or Pyrogram to build your own script. While this works for hobby projects, it quickly breaks down in production:

Anti-blocking: Telegram is very aggressive with "Too Many Requests" (FloodWait) errors. Apify Actors include logic to handle these delays gracefully.
No infrastructure to manage: You don't need to worry about Docker containers, server costs, or task scheduling. Apify Cloud manages everything.
Easy scaling: Whether you want to scrape one channel or ten thousand, the platform scales the resources automatically.
Standardized data: No need to spend hours cleaning messy HTML or inconsistent JSON responses.

Use cases in action

Scenario A: The crypto trader

A crypto trader wants to monitor sentiment on the "CryptoNinjas" channel. By scraping the reaction counts for every post, they can determine if a specific announcement is being received positively (lots of 🔥 and 👍) or negatively (lots of 🤡 or 🤬). They utilize the Apify API to fetch this data hourly and feed it into a sentiment score dashboard.

Scenario B: The news aggregator

A media startup wants to create a consolidated feed of information from various independent news channels. They use the Media URLs provided by the Actor to automatically download photos and videos, which they then republish on their own platform (respecting copyright and platform terms, of course).

Scenario C: The academic researcher

A sociology student is studying the spread of specific political narratives. They use the scraper to download 5,000 posts across various community channels. Because the data is delivered in a clean CSV format, it can be immediately uploaded to a data analysis tool, like NVivo or R, for qualitative coding.

Best practices for responsible scraping

When using any web scraping tool, it is important to be a good "web citizen":

Respect rate limits: Don't try to scrape millions of posts in a single second. Use the Actor's built-in limits to stay under the radar.
Public data only: Only scrape data from public channels. This Actor is not designed to violate privacy or access private messages.
Adhere to GDPR and CCPA: If you are collecting data that contains usernames, ensure you are handling this information in compliance with local data protection laws.
Keep your session string private: Your session string grants full access to your Telegram account. Never share it publicly or commit it to source control.

Wrapping Up

The days of manually scrolling through Telegram channels to find the information you need are now behind us. With Telegram Channels Scraper, you can turn the world's most popular messaging app into your own structured database — all authenticated with nothing more than a QR code scan.

Whether you are building the next big AI model, monitoring your competitors, or just trying to stay organized, our tool provides the reliability and depth of data you need to succeed.

Try Telegram Channels Scraper on Apify today and get ahead of your competitors!

FAQ

Do I need a Telegram account to use this?
Yes, a Telegram account is required for authentication. However, the process is frictionless — you simply scan a QR code on your first run, just like linking Telegram Web. No API keys, no developer credentials.

Do I need to create a Telegram Developer App?
No. The Actor uses built-in Telegram credentials by default, so you don't need to register at my.telegram.org or generate an api_id / api_hash. If you prefer to use your own credentials for extra reliability, you can optionally set TELEGRAM_API_ID and TELEGRAM_API_HASH as environment variables (as a developer).

What is the session string and how do I save it?
After your first QR code login, the Actor prints a session string in the log. This string lets you re-authenticate on future runs without scanning the QR code again. Copy it from the log (or retrieve it from the SESSION_STRING key in your run's Key-Value Store) and paste it into the Saved Session String input field.

Is it legal to scrape Telegram?
Extracting publicly available factual data is generally considered legal. However, you should always check Telegram's Terms of Service and consult with legal counsel if you are unsure about your specific use case.

Can I export the data to my CRM?
Yes. Since Apify provides a robust API and integrations with platforms like Zapier and Make, you can automatically send your scraped Telegram data to Salesforce, HubSpot, or even a simple Google Sheet.

What formats do you support?
The Actor supports JSON, CSV, Excel (XLSX), XML, and HTML table formats.

How to Build a Global Internationalization (I18n) App with Next.js and AI

Emmanuel Uchenna — Sat, 10 Jan 2026 13:16:17 +0000

Before the proliferation of AI, localization was often a nightmare. You had to hire translators, manage extensive JSON files for every language, and pray that your context didn't get lost in translation. If you wanted to translate an entire website on the fly? Forget about it. That was a job for enterprise giants with bottomless budgets.

In this article, we will build Lingo.app from scratch. It is a stunning, production-ready web application that can scrape any website and instantly localize its content into 83+ languages. We will do this using the power of Next.js, Tailwind CSS, and the AI Website Content Localizer & Scraper Actor on the Apify platform.

We wrote this article for developers who want to ship global apps without the headache. Intermediate developers can likely speed through the setup, but we will explain every step clearly.

By the end of this article, you will have a fully functional app that manages scraping of any website, context-aware AI translation, and UI state—all in a scalable Next.js architecture.

Note: This article was originally published in Eunit.me.

If you are excited, then let’s dive in.

TL;DR

Checkout the complete repository on GitHub: https://github.com/Eunit99/lingo.app 📖
Checkout the deployed version of the Lingo.app live: https://lingo-app-azure.vercel.app/ 🚀

The Architecture

We are keeping it modern and simple:

Frontend: Next.js 15 (App Router) for the framework.
Styling: Tailwind CSS v4 for that premium, dark-mode aesthetic.
Backend: A Next.js API Route acting as a secure proxy.
Intelligence: The AI Website Content Localizer & Scraper Actor on Apify, which handles the headless browser scraping (Playwright) and AI translation (Lingo.dev).

File tree

Below is how the file structure of this app will be:

|   .gitignore
|   next-env.d.ts
|   eslint.config.mjs
|   next.config.ts
|   postcss.config.mjs
|   tsconfig.json
|   package.json
|   package-lock.json
|   env.example
|   .env.local
|   tsconfig.tsbuildinfo
|   README.md
|   
+---app
|   |   page.tsx
|   |   globals.css
|   |   icon.png
|   |   layout.tsx
|   |   
|   \---api
|       \---localize
|               route.ts
|               
+---public
|       grid.svg
|       
+---components
|       Features.tsx
|       Hero.tsx
|       LocalizationForm.tsx
|       
+---lib
|       apify.ts

Prerequisites

To follow along, you will need:

Node.js installed (v18 or higher).
An Apify account. You can sign up for free.
A text editor like VS Code.

Step 1: Initialize the project

First, we create a new Next.js application. We use the --no-src-dir flag to keep our project root clean and flat.

npx create-next-app@latest lingo.app --typescript --eslint --tailwind --no-src-dir --app

Navigate into your new directory and install the necessary dependencies, specifically the explicit React types to avoid TS errors and the Apify client.

cd lingo.app
npm install apify-client
npm install --save-dev @types/react @types/react-dom

Setting up the Global Design System

We want a premium feel right from the start. Open app/globals.css and replace the default styles with this modern Tailwind v4 setup. We are defining a deep slate color palette for a developer-focused dark mode.

@import "tailwindcss";

:root {
  --background: #0f172a; /* Slate 900 */
  --foreground: #f8fafc; /* Slate 50 */
}

body {
  color: var(--foreground);
  background: var(--background);
  font-family: Arial, Helvetica, sans-serif;
  overflow-x: hidden;
}

@layer components {
  .btn-primary {
    @apply px-6 py-3 bg-indigo-600 hover:bg-indigo-500 text-white rounded-lg font-medium transition-all duration-200 shadow-lg hover:shadow-indigo-500/30 active:scale-95 disabled:opacity-50 disabled:cursor-not-allowed;
  }
  .input-field {
    @apply w-full bg-slate-800 border border-slate-700 rounded-lg px-4 py-3 text-white placeholder-slate-400 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:border-transparent transition-all;
  }
  .card {
    @apply bg-slate-800/50 backdrop-blur-md border border-slate-700/50 rounded-xl p-6 shadow-xl;
  }
}

Next, let's configure the root layout to apply this background globally. Open app/layout.tsx. We are adding a subtle grid background pattern for texture.

import type { Metadata } from 'next';
import { Inter } from 'next/font/google';
import './globals.css';

const inter = Inter({ subsets: ['latin'] });

export const metadata: Metadata = {
  title: 'Lingo.app - Globalize Your Web',
  description: 'Instantly localize any website or text into 83+ languages using AI.',
};

export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <body className={`${inter.className} min-h-screen bg-gradient-to-br from-slate-900 via-slate-900 to-indigo-950 selection:bg-indigo-500/30`}>
        {/* Subtle Grid Background */}
        <div className="fixed inset-0 bg-[url('/grid.svg')] bg-center [mask-image:linear-gradient(180deg,white,rgba(255,255,255,0))]" />

        <main className="relative z-10 w-full max-w-7xl mx-auto px-4 sm:px-6 lg:px-8 py-8">
             {/* Simple Header */}
            <header className="flex justify-between items-center py-6 mb-12">
                <div className="flex items-center space-x-2">
                    <span className="text-2xl font-bold bg-clip-text text-transparent bg-gradient-to-r from-white to-slate-400">lingo.app</span>
                </div>
            </header>
            {children}
        </main>
      </body>
    </html>
  );
}

(Note: You will need a simple public/grid.svg file for the background. You can find free SVG patterns online or generate one.) See the one use for the lingo.app project on GitHub.

Step 2: Integrate the Apify Client

Now for the engine room. We need a way to communicate with the Apify platform securely.

The Client Wrapper (`lib/apify.ts`)

This client will enable communication with the AI Website Content Localizer & Scraper Actor on Apify.

Create a folder lib and a file apify.ts. This acts as our localized SDK. It initializes the client and defines the types for our input.

import { ApifyClient } from 'apify-client';

export interface LocalizerInput {
  token: string;
  lingoApiKey: string; // The secret key for Lingo.dev
  mode: 'WEB' | 'TEXT';
  targetLanguages: string[];
  startUrls?: { url: string }[];
  text?: string;
}

export const runLocalizerActor = async (params: LocalizerInput) => {
  const { token, lingoApiKey, mode, targetLanguages, startUrls, text } = params;

  const client = new ApifyClient({
    token: token,
  });

  const input = {
    mode,
    lingoApiKey,
    targetLanguages,
    startUrls: mode === 'WEB' ? startUrls : undefined,
    text: mode === 'TEXT' ? text : undefined,
  };

  try {
    // Run the Actor and wait for it to finish
    const run = await client.actor("eunit/ai-website-content-localizer-scraper").call(input);

    if (!run) {
        throw new Error("Actor run failed to start or return.");
    }

    // Fetch results from the default dataset
    const { items } = await client.dataset(run.defaultDatasetId).listItems();
    return items;
  } catch (error) {
    console.error("Apify Actor Run Error:", error);
    throw error;
  }
};

The API Proxy (`app/api/localize/route.ts`)

We cannot leak our Apify API Token to the client-side browser code. To solve this, we create an API Route that acts as a proxy. The client sends the URL/Text, and the server attaches the token and calls Apify.

Create app/api/localize/route.ts:

import { NextResponse } from 'next/server';
import { runLocalizerActor } from '@/lib/apify';

export async function POST(req: Request) {
  try {
    const body = await req.json();
    const { url, text, mode, targetLanguages, lingoApiKey } = body;

    const apifyToken = process.env.APIFY_API_TOKEN;

    if (!apifyToken) {
      return NextResponse.json({ error: "Server misconfigured: APIFY_API_TOKEN missing. Please set it in .env.local" }, { status: 500 });
    }

    const finalLingoKey = lingoApiKey || process.env.LINGO_API_KEY;
    if (!finalLingoKey) {
      return NextResponse.json({ error: "Lingo API Key is required. Please provide it in the form or set LINGO_API_KEY env." }, { status: 400 });
    }

    if (mode === 'TEXT' && text && text.length > 500) {
      return NextResponse.json({ error: "Text content exceeds the 500 character limit." }, { status: 400 });
    }

    const startUrls = url ? [{ url }] : undefined;

    // Default languages if not provided
    const languages = targetLanguages && targetLanguages.length > 0 ? targetLanguages : ['es', 'fr', 'de'];

    const items = await runLocalizerActor({
      token: apifyToken,
      lingoApiKey: finalLingoKey,
      mode: mode as 'WEB' | 'TEXT',
      targetLanguages: languages,
      startUrls,
      text
    });

    return NextResponse.json({ success: true, data: items || [] });
  } catch (error: any) {
    console.error("API Route Error:", error);
    return NextResponse.json({ success: false, error: error.message || "Unknown error" }, { status: 500 });
  }
}

To prevent usage abuse and manage API costs, we implemented a strict 500-character limit across both the frontend and backend.

In the user interface in components/LocalizationForm.tsx, we verified that the localizer input remains within bounds by adding a maxLength attribute and a real-time character counter (e.g., "0/500") that visually alerts the user as they type.

Crucially, we backed this up with server-side validation in app/api/localize/route.ts. Before sending any data to Apify, the API route checks the text length and immediately rejects requests exceeding 500 characters with a 400 Bad Request. This ensures the application remains secure and cost-efficient, even if the client-side restrictions are bypassed.

Step 3: Building the UI Components

Now let's build the interactive frontend.

The Hero Component

In components/Hero.tsx, we create a bold introduction.

import React from 'react';

export default function Hero() {
  return (
    <div className="text-center max-w-3xl mx-auto space-y-6">
      <div className="inline-flex items-center space-x-2 px-3 py-1 rounded-full bg-indigo-500/10 border border-indigo-500/20 text-indigo-400 text-sm font-medium mb-4">
        <span>Powered by Lingo.dev & Apify</span>
      </div>
      <h1 className="text-5xl md:text-7xl font-extrabold tracking-tight text-white leading-tight">
        Globalize your <span className="text-indigo-500">content</span> instantly.
      </h1>
      <p className="text-lg md:text-xl text-slate-400 max-w-2xl mx-auto">
        Scrape any website and translate it into 83+ languages with context-aware AI.
        The ultimate tool for shipping global apps fast.
      </p>
    </div>
  );
}

The Main Logic: LocalizationForm

This is where the user interacts with the app. We need to handle:

Toggling between WEB (URL) and TEXT mode.
Selecting target languages.
Calling our API route.
Displaying the results in a tabbed interface.

Create components/LocalizationForm.tsx:

'use client';

import React, { useState } from 'react';

export default function LocalizationForm() {
  const [mode, setMode] = useState<'WEB' | 'TEXT'>('WEB');
  const [url, setUrl] = useState('');
  const [textInput, setTextInput] = useState('');
  const [selectedLangs, setSelectedLangs] = useState<string[]>(['es', 'fr', 'de']);
  const [loading, setLoading] = useState(false);
  const [results, setResults] = useState<any[]>([]);

  // ... (Toggle logic helper functions)

  const popularLangs = [
    { code: 'es', label: 'Spanish' },
    { code: 'fr', label: 'French' },
    { code: 'de', label: 'German' },
    { code: 'ja', label: 'Japanese' },
    { code: 'zh', label: 'Chinese' },
    { code: 'pt', label: 'Portuguese' },
    { code: 'it', label: 'Italian' },
    { code: 'ko', label: 'Korean' },
  ];

  const toggleLang = (code: string) => {
    if (selectedLangs.includes(code)) {
      setSelectedLangs(selectedLangs.filter(c => c !== code));
    } else {
      setSelectedLangs([...selectedLangs, code]);
    }
  };

  const handleSubmit = async (e: React.FormEvent) => {
    e.preventDefault();
    setLoading(true);
    setResults([]);

    try {
      const res = await fetch('/api/localize', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          mode,
          url: mode === 'WEB' ? url : undefined,
          text: mode === 'TEXT' ? textInput : undefined,
          targetLanguages: selectedLangs,
        }),
      });

      const data = await res.json();
      if (data.success) {
          setResults(data.data);
      }
    } catch (err) {
      console.error(err);
    } finally {
      setLoading(false);
    }
  };

  return (
    <div className="w-full max-w-4xl mx-auto space-y-8">
      <div className="card">
        {/* Mode Toggles */}
        <div className="flex space-x-4 mb-6 border-b border-slate-700/50 pb-4">
          <button onClick={() => setMode('WEB')} className={`... ${mode === 'WEB' ? 'text-indigo-400' : 'text-slate-400'}`}>
            Web Scraper
          </button>
          <button onClick={() => setMode('TEXT')} className={`... ${mode === 'TEXT' ? 'text-indigo-400' : 'text-slate-400'}`}>
            Text Localizer
          </button>
        </div>

        <form onSubmit={handleSubmit} className="space-y-6">
            {/* Input Fields specific to mode */}
            {mode === 'WEB' ? (
                <input type="url" value={url} onChange={(e) => setUrl(e.target.value)} className="input-field" placeholder="https://example.com" />
            ) : (
                <div>
                  <div className="flex justify-between items-center mb-2">
                    <label className="block text-sm font-medium text-slate-300">Text Content</label>
                    <span className={`text-xs ${textInput.length >= 500 ? 'text-red-400' : 'text-slate-500'}`}>
                      {textInput.length}/500
                    </span>
                  </div>
                  <textarea
                    placeholder="Paste your text here to localize..."
                    className="input-field min-h-[150px]"
                    value={textInput}
                    onChange={(e) => setTextInput(e.target.value.slice(0, 500))}
                    maxLength={500}
                    required={mode === 'TEXT'}
                  />
                </div>
            )}

            {/* Language Selector Logic Here */}
            {/* ... */}

            <div className="flex justify-end">
                <button type="submit" disabled={loading} className="btn-primary">
                    {loading ? 'Processing...' : 'Localize Content'}
                </button>
            </div>
        </form>
      </div>

      {/* Results Rendering */}
      {results.map((item, idx) => (
          <ResultCard key={idx} item={item} />
      ))}
    </div>
  );
}

function ResultCard({ item }: { item: any }) {
    // Shows original vs localized tabs
    // ...
}

Step 4: Page Assembly

Finally, we string it all together in app/page.tsx.

import React from 'react';
import Hero from '@/components/Hero';
import LocalizationForm from '@/components/LocalizationForm';
import Features from '@/components/Features';

export default function Home() {
  return (
    <div className="flex flex-col space-y-24">
      <div className="flex flex-col items-center justify-start space-y-12">
        <Hero />
        <LocalizationForm />
      </div>
      <Features />
    </div>
  );
}

Why the Apify Actor Integration Wins

You might be wondering: "Why can't I just use fetch to get the HTML and send it to ChatGPT?"

You could try, but you would likely fail for three reasons. Integrating the AI Website Content Localizer & Scraper Actor via Apify solves these specific engineering headaches:

1. The "Context" Problem

Standard LLMs see text as a flat string. They don't know that "Home" is a navigation button or that "Get Started" is a primary CTA. If you strip the HTML, you lose the context. If you keep the HTML, you confuse the model with thousands of lines of markup.

This AI Website Content Localizer & Scraper Actor is smart. It extracts the semantics of the page, identifies the translatable nodes, and sends them to Lingo.dev, which is specifically trained to understand UI and content context. The result? Buttons stay short, and articles sound natural.

2. The Scraping Battlefield

Modern websites are hostile. They use hydration (client-side rendering), anti-bot protections (Cloudflare, CAPTCHAs), and complex DOM structures.

Without Apify: You have to manage a fleet of headless browsers, rotate proxies, handle retries, and parse dynamic JavaScript.
With Apify: You make one API call. The Actor handles the headless browser infrastructure, proxy rotation, and Playwright execution for you.

3. Serverless Scalability

Translating a massive documentation site or a large e-commerce catalog requires significant computing power. If you run this logic in your own Next.js API route, you risk timeouts and server crashes.

By offloading this to Apify, you get a serverless queue system. You can submit up to 100 URLs at once, and the Apify platform scales up the necessary Actors to process them in parallel. Your Next.js app remains lightweight and responsive.

Step 5: Setup & Deploy

We are ready to launch. Create a .env.local file in your root folder:

APIFY_API_TOKEN=your_apify_token
LINGO_API_KEY=your_lingo_key

Get your APIFY_API_TOKEN by signing up at Apify. Apify gives you a free 5$ credit every month. This is more than enough to build your first app.

To get your LINGO_API_KEY, sign up at Lingo.dev. Lingo.dev gives you 10, 000 free words per month to try out new things.

Once set, it is time to run your app.

Running your App

Run your development server:

npm run dev

Open http://localhost:3000, and you will see your global localization app, ready to scrape and translate the web.

Wrapping Up

We didn't just build a demo. We built a tool that addresses a real business problem: accessing global information. By using the AI Website Content Localizer & Scraper on Apify, we offloaded the most challenging aspects of scraping and AI context management, allowing us to focus on building a premium user experience.

If you are building global applications, you need tools that scale with you. Check out the AI Website Content Localizer & Scraper on Apify.

Happy building 🚀

Building a Modern Digital Garden with Google AI: My New Year, New You Portfolio

Emmanuel Uchenna — Sat, 03 Jan 2026 22:55:23 +0000

This is a submission for the New Year, New You Portfolio Challenge Presented by Google AI

About me

Hi, I’m Emmanuel Uchenna - a software engineer, technical writer, and digital health advocate who is passionate about building technology that empowers people.

My journey in tech began over five years ago, fueled by a curiosity about how lines of code could translate into meaningful human experiences. Today, I specialize in crafting clean, scalable user interfaces using React, Next.js, and the broader modern web ecosystem. But for me, code is just one part of the equation. I also love translating complex technical ideas into clear, engaging content through articles, documentation, and whitepapers.

Beyond the terminal, I am deeply invested in the intersection of technology and healthcare. As a digital health advocate, I explore how software can be used to enhance patient outcomes and increase the accessibility of health information.

My goal with this portfolio was simple yet ambitious: to create a platform that is fast, accessible, and truly reflects my current skillset and style. I wanted a space that didn't just list my projects but demonstrated my philosophy of "premium simplicity" - a belief that the most effective digital experiences are those that get out of the user's way while providing delight through subtle interactions and solid performance.

Portfolio

Emmanuel Uchenna | Software developer and technical writer

Emmanuel Uchenna is a software engineer, technical writer, and digital health advocate passionate about building technology that empowers people. With over five years of experience, he specializes in crafting clean, scalable web and mobile applications with React, Next.js, and modern web tooling, while also translating complex technical ideas into clear, engaging content through articles, documentation, and whitepapers.

eunit.me

(Note: The above embed links to my portfolio deployed on Google Cloud Run. You can also visit the live site at eunit.me)

How I built it

Building a portfolio in 2025 is an interesting challenge. The tools available to us have undergone significant evolution. For this project, I decided to lean heavily into the Google AI ecosystem to see how much it could accelerate my workflow without compromising on quality. The results were genuinely surprising, transforming what could have been a week-long slog into an inspired sprint.

The tech stack

I chose a stack that emphasizes performance, scalability, and developer experience:

Core Framework: Next.js (App Router) for a robust, server-side rendered foundation. The file-based routing and React Server Components allowed me to keep the application fast and SEO-friendly.
Styling: Tailwind CSS. I adopted a utility-first approach to implement a strict design system. This made handling responsive layouts and adopting dark mode incredibly seamless.
Database: PostgreSQL. I needed a reliable relational database to manage dynamic content, such as tracking blog post views and managing newsletter subscriptions.
Deployment: Google Cloud Run. I wanted a serverless, containerized deployment that could scale to zero when not in use but handle traffic spikes if a post went viral.

The Deployment Pipeline: From Local to Global

One of the requirements for this challenge was deploying to Google Cloud Run, and honestly, it was one of the smoothest parts of the process. I didn't want to "throw code over the wall"; I wanted a reproducible, containerized pipeline.

I created a Dockerfile that leverages a multi-stage build directly for Next.js. This keeps the final image lightweight, only around 80MB, by stripping out development dependencies and unused files.

# ... standard Next.js multi-stage build ...
CMD ["node", "server.js"]

Deploying was as simple as running a single command with the Gemini CLI helper to push the container to Google Artifact Registry and creating a simpler Cloud Run service. Now, whenever I push to my main branch, a seamless CI/CD pipeline builds the container and updates the revision. This "git-push-to-deploy" workflow gives me the confidence to ship small, iterative updates frequently.

Leveraging Google AI tools

This wasn't just a standard build; it was an AI-assisted development process. Here is a deep dive into how I used Google's tools to bring this vision to life:

1. Google's Antigravity (The AI-first IDE)

Antigravity was the centerpiece of my development environment. It acted as my pair programmer throughout the entire process, fundamentally changing how I wrote code. Instead of constantly context-switching between my editor and a browser to look up documentation or debug errors, I stayed in the flow.

Refactoring with confidence
One of the most complex tasks was refactoring my custom UI components to be more reusable. For instance, I had multiple ad-hoc modal implementations scattered across the codebase—ReviewModal, DriverProfileModal, and others. I used Antigravity to analyze these disparate files. It suggested a unified Modal architecture using framer-motion for smooth entry and exit animations. It didn't just give me a snippet; it walked me through the implementation plan, ensuring I correctly handled prop drilling and accessibility (ARIA labels).

Intelligent debugging
During the build, I encountered a tricky TypeScript error with a Chakra UI Text component regarding polymorphic props (as prop conflicts). Usually, this would send me down a generic Stack Overflow rabbit hole. Instead, I asked Antigravity. It analyzed my specific component usage, explained why the type inference was failing, and offered a precise fix that satisfied the TypeScript compiler without using any.

Proactive optimization
Antigravity was like having a senior engineer looking over my shoulder. It proactively suggested checks for image optimization, reminding me to use the sizes prop on my next/image components to prevent layout shifts (CLS) and improve LCP scores.

2. AI Studio & Gemini Models

While Antigravity handled the code, AI Studio dealt with the creative direction. I used it as my "Editor-in-Chief" to brainstorm the content strategy and structure of the site.

Content generation and refinement
I fed the model my previous bio and project descriptions and asked it to help me refine the tone to be more professional yet approachable. The "New Year, New You" theme resonated with me, so I utilized Gemini to help draft sections of the site that needed fresh text, particularly for my new "Health" section. It helped me articulate my dual passion for engineering and digital health in a way that felt cohesive rather than disjointed.

Idea generation
When I was stuck on how to present my "Research" papers, I used AI Studio to generate layout ideas. It suggested a card-based layout with abstract summaries, which I then implemented in code. It essentially bridged the gap between a vague idea and a concrete design specification.

3. Gemini CLI

For repetitive tasks and automation, the Gemini CLI was a lifesaver. I used it to quickly scaffold standard boilerplate code for new pages and components. It sped up the "boring" parts of development so I could focus on the creative aspects, like the micro-interactions and layout adjustments.

Design decisions

The "Premium Simplicity" aesthetic
I opted for a strict black-and-white (B&W) aesthetic. I believe that constraints breed creativity. By removing color as a primary tool for hierarchy, I was forced to rely on spacing, typography, and contrast to guide the user's eye. This resulted in a design that feels premium, modern, and uncluttered. It helps the actual content, my articles and projects, stand out without visual noise.

Component-driven architecture
I prioritized a component-driven architecture to ensure maintainability and flexibility. Every piece of the UI, from the ProjectList to the BlogCard, is a self-contained component. This modularity means that if I want to update the design of my cards next year, I only have to change it in one place.

Typography as interface
In a black-and-white design, typography does the heavy lifting. I chose a variable font that allows for subtle weight transitions on hover, creating a tactile feel without the need for color shifts. It’s accessible, performant, and sharp on high-DPI displays. I also ensured that all interactive elements have sufficient contrast ratios (meeting WCAG AAA standards), proving that "stylish" doesn't have to mean "unreadable."

What I'm most proud of

There are a few aspects of this project that I am particularly proud of, as they represent significant personal and technical growth.

1. The "Health" section
Integrating my work in digital health into my developer portfolio was a big step. It’s a dedicated space for articles covering health and well-being topics, bridging the gap between my technical skills and my advocacy work. For a long time, I kept these two worlds separate; my GitHub profile showed one person, while my community work showed another.

Merging them here makes the portfolio feel uniquely mine and tells a more complete story of who I am.

2. Performance & accessibility
I didn't want just a pretty site; I wanted a fast one. With the help of Antigravity's suggestions, I optimized my images, refined my script loading strategies (for things like analytics), and minimized layout shifts. Achieving high Lighthouse scores was a key milestone. It proves that you don't have to sacrifice performance for a modern aesthetic.

3. The seamless AI workflow
Honesty, I am most proud of how effectively I integrated AI into the loop. Using Antigravity and Gemini felt less like using a tool and more like working with a collaborator who knows my codebase inside out. It allowed me to be more ambitious with my features, such as adding dynamic view counters and complex animations, and more confident in my code quality. It shifted my role from "writer of code" to "architect of solutions."

Wrapping Up

This challenge was the perfect excuse to finally ship the v3 of my portfolio. It represents not just a visual refresh, but a "backend" refresh of my own skills and workflows. By embracing AI-first development with Google's tools, I've built something that sets the tone for my work in 2026.

How to scrape YouTube trends and popular channels

Emmanuel Uchenna — Sat, 03 Jan 2026 17:10:49 +0000

In today's world, where data is the new gold, it is important to have access to the data you need to make informed decisions. YouTube, as the world's second-largest search engine, is a repository of consumer insights, viral trends, and competitor strategies. But accessing this data at scale can be challenging.

If you have ever tried to build a dashboard for viral content, conduct market research, or keep a pulse on what is hot around the globe, you have likely hit a wall. The YouTube Data API comes with strict usage quotas that stifle your growth, while building your own scrapers often leads to a maintenance headache of broken selectors and IP bans.

It doesn't have to be this way; there must surely be a better way to obtain this YouTube data. In this article, we will explore how you can use production-grade Apify Actors to scrape real-time YouTube trending videos and popular channel data without managing infrastructure, worrying about API quotas, or writing a single line of complex code.

Why scrape YouTube trends?

Before we dive into the "how," let's explore the "why." Scraping YouTube trends allows you to make data-driven decisions that can significantly impact your content strategy and marketing ROI. Here are the key reasons why top brands and creators are extracting this data:

1. Trend analysis and content strategy

Knowing what is popular right now allows you to ride the wave of viral content. By identifying trending topics, video formats, and relevant keywords, you can create timely, relevant, and engaging content that aligns with current audience interests. Instead of guessing what might work, you can base your creative decisions on hard data from millions of views.

2. Competitive intelligence

Your competitors are likely already experimenting with new formats and strategies. By monitoring their most successful videos and analyzing their metadata (titles, tags, descriptions), you can reverse-engineer their success. This will enable you to find gaps in the market that they are missing, or refine your own approach to outperform them in the algorithm.

3. Deep market research

YouTube comments and engagement metrics offer an unfiltered view of consumer sentiment. Scraping this data enables you to analyze user behavior, understand pain points, and discover emerging needs before they become mainstream. This is crucial for product development and tailoring your messaging to resonate with your target audience.

4. Global lead generation

It is a no-brainer that trends vary drastically by region; as such, detailed scraping allows you to discover new opportunities and potential customers by analyzing what different audiences are searching for and engaging with across over 25 countries. Even if you are expanding into Brazil, India, or Japan, local trend data is invaluable.

The challenge with the official API and DIY scraping

You might be asking, "Doesn't YouTube have an API for this?" Yes, but it comes with significant limitations for serious data analysis:

Strict quotas: The API has a daily credit limit that is easily exhausted by even modest data collection efforts. Scaling requires expensive enterprise agreements or complex quota management.
Limited availability: Some granular data points, like specific trending categories in certain regions, can be difficult to access or require convoluted workarounds.
Maintenance: If you choose to build your own scraper using tools like Selenium or Playwright, you face the constant battle of "cat and mouse." This is because YouTube frequently updates its HTML structure, which can cause custom scripts to break. Additionally, you need to manage rotating proxies and handle CAPTCHA to avoid IP bans.

The solution: Serverless YouTube scrapers

This is where Apify Actors come in. Actors are cloud-based programs that can perform any action in a web browser, from scraping data to automating workflows. They handle the heavy lifting, such as infrastructure, proxies, scaling, and anti-scraping protections, so that you can focus on the data.

We will focus on two specific Actors designed to solve the YouTube data puzzle:

These tools operate on a Pay-Per-Event model, meaning you only pay for the successful results you get, not for the time the server runs.

Tool 1: YouTube Trending Videos by Categories Scraper

The YouTube Trending Videos by Categories Scraper is your go-to tool for tracking viral content with precision. While standard tools may provide a generic "trending" list, this scraper enables granular filtering.

Key features:

Category-specific scraping: Don't just see what is trending overall. Drill down into specific niches like Music, Gaming, News & Politics, Sports, Film & Animation, and more.
Global coverage: access real-time trends from over 25 countries, including the US, UK, Brazil, India, Japan, France, and Germany.
Rich data extraction: Get vastly more than just a video title. This Actor extracts the rank, full video metadata, channel name, view counts, like counts, comment counts, and thumbnail URLs.

This level of detail is perfect for social media managers who need to spot viral gaming clips in the UK or music trends in South Korea instantly.

Tool 2: YouTube Popular Channels Scraper

The YouTube Popular Channels Scraper is the specialized Actor you need if your goal is influencer marketing or channel discovery. It focuses on the creators behind the content.

Key features:

Influencer discovery: Find out which channels are currently surging in popularity within a specific country or category.
Recent hits: Instead of just channel stats, get a snapshot of their most recent trending videos. This helps you verify if a channel's popularity is current or fading.
Keyword intelligence: Extract the "Popular Keywords" of the day. This is a goldmine for SEO specialists looking to optimize video titles and descriptions for search traffic.

Zero-to-data walkthrough: How to scrape YouTube trends in 3 minutes

You don't need to be a Python wizard to get this data. Here is a step-by-step guide to downloading your first dataset using these scrapers on the Apify platform.

Step 1: Access the YouTube Trending Videos by Categories Actor

Navigate to the YouTube Trending Videos by Categories page on the Apify Store. Click the Try for free button. If you don't have an account, you can sign up using your email or GitHub credentials.

Step 2: Configure the input

Once you are in the Apify Console, you will see a user-friendly input form. This is where you define exactly what you want to scrape.

Country: Select your target region from the dropdown menu (e.g., United Kingdom or United States). Leaving this blank or selecting World will retrieve global trends.
Category: choose a specific vertical. For example, if you are analyzing the gaming market, select Gaming. For a broad overview, you can select All.

Step 3: Run the YouTube Trending Videos by Categories Actor

Click the green Start button at the bottom of the page.

The Actor will now spin up a cloud server, navigate to YouTube, extract the data using high-quality residential proxies to avoid detection, and process the results. This usually takes just a few seconds to a minute, depending on the amount of data.

Step 4: Export the data

When the run finishes, you will see a status of "Succeeded." Click on the Export button to view your data. You can download it in various formats depending on your needs:

Excel / CSV: Perfect for opening in spreadsheets to perform quick pivot tables or charts.
JSON: Ideal if you are feeding this data into another application or database.
HTML / XML: Available for legacy integrations.

You now have a clean, structured dataset of the top trending videos, ready for analysis.

Zero-to-data walkthrough: How to scrape Popular Channels

Step 1: Access the YouTube Popular Channels Scraper Actor

Navigate to the YouTube Popular Channels Scraper page on the Apify Store. Click the Try for free button. If you don't have an account, you can sign up using your email or GitHub credentials.

Step 2: Configure the channels scraper input

Once you are in the Apify Console, the input form allows you to specify your target audience.

Country: Select your target region from the dropdown menu (e.g., United Kingdom or India). Leaving this blank or selecting World will fetch global popular channels.
Category: Choose a specific vertical. For example, if you are looking for gaming influencers, select Gaming. For a broad overview, you can select All.

Step 3: Run the YouTube Popular Channels Scraper

Click the green Start button at the bottom of the page.

The Actor will now spin up a cloud server, navigate to YouTube, extract the popular channel data using high-quality residential proxies to avoid detection, and process the results. This usually takes just a few seconds to a minute, depending on the amount of data being processed.

Step 4: Export the channels data

When the run finishes, you will see a status of "Succeeded." Click on the Export button to view your data. You can download it in various formats depending on your needs:

Excel / CSV: Perfect for opening in spreadsheets to perform quick pivot tables or charts.
JSON: Ideal if you are feeding this data into another application or database.
HTML / XML: Available for legacy integrations.

You now have a clean, structured dataset of the top popular channels and keywords, ready for analysis.

For developers: Automate with Python

If you are building a custom dashboard or feeding data into a machine learning model, you will want to automate this process. The apify-client for Python makes this incredibly simple.

First, install the client:

pip install apify-client

Now, you can write a simple script to fetch trending gaming videos in the UK or any country of your choosing. You will find your API token in the Settings > API & Integrations section of the Apify Console.

pip install apify-client

Now, you can write a simple script to fetch trending gaming videos in the UK or any country of your choosing. You will need your API token, which you can find in the Settings > API & Integrations section of the Apify Console.

from apify_client import ApifyClient

# 1. Initialize the ApifyClient with your API token
client = ApifyClient("YOUR_API_TOKEN")

# 2. Prepare the Actor input
# We want trending Gaming videos in the UK
run_input = {
    "country": "united-kingdom",
    "category": "gaming",
}

print("🎥 Fetching YouTube gaming trends for UK...")

# 3. Run the Actor
# We use the actor ID: eunit/youtube-trending-videos-by-categories
run = client.actor("eunit/youtube-trending-videos-by-categories").call(run_input=run_input)

# 4. Fetch and print Actor results from the run's dataset
print(f"Stats for run {run['id']}:")

# Iterate through the items in the dataset
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    # item is a dictionary containing all the scraped data
    rank = item.get('rank', 'N/A')
    title = item.get('video_title', 'Unknown')
    views = item.get('views', '0')
    channel = item.get('channel_name', 'Unknown')

    print(f"#{rank} | {title} by {channel} ({views} views)")

This script can be set to run on a schedule (using Cron or Apify Schedules), ensuring your database is always up-to-date with the latest trends without manual intervention.

Pricing: The pay-per-event advantage

One of the biggest advantages of using these specific Apify Actors is the Pay-Per-Event pricing model.

Traditional SaaS tools often charge hefty monthly subscriptions regardless of how much you use them. With Pay-Per-Event, you are charged a small, fixed amount for each result you successfully scrape.

Cost-effective: If you only need to scrape trends once a week, you pay pennies. You don't waste money on idle server time.
Predictable: You can easily calculate costs. If scraping 1,000 videos costs $X, then scraping 10,000 videos costs $10X. There are no hidden tiers or surprise overage fees.
Scale effortlessly: Need to scrape data for 20 different countries simultaneously? You can trigger 20 concurrent runs via the API. The cost per result remains the same, and Apify's infrastructure handles the load.

Legal and ethical considerations

When scraping public data, it is important to operate ethically. These Actors are designed to extract publicly available information—data that any user could see by visiting the website without logging in. They do not access private videos, user passwords, or personal emails.

Wrapping Up

YouTube data is too valuable to be locked behind restrictive APIs or complex coding barriers. Whether you are a solo content creator looking to optimize your next thumbnail or a marketing agency analyzing global trends for a Fortune 500 client, having reliable, scalable access to this data is a game-changer.

By using the YouTube Trending Videos by Categories and YouTube Popular Channels Scraper Actors, you can bypass the technical hurdles and go straight to the insights.

Happy scraping!

How to Scrape Twitter (X) Trends Without Breaking the Bank

Emmanuel Uchenna — Fri, 02 Jan 2026 09:26:00 +0000

If you want to know what the world is thinking right now, you look at Twitter (X). With over 500 million tweets sent every single day, Twitter (X) is the undisputed pulse of the internet. It is described as the place where news breaks, memes go viral, and public opinion shifts in real-time. In fact, nearly half of all US social media users rely on X specifically to get their latest news. For marketers, researchers, and developers, this real-time data is a goldmine, but lately, it’s been buried behind an incredibly expensive paywall.

Since the platform's overhaul, the days of easy, free API access are gone. Official enterprise tiers now command prices that start at a staggering $42,000 a month. Practically overnight, individual developers and small teams were priced out of the conversation. And if you’ve tried building your own "DIY" scraper lately, you know it’s a constant battle against IP bans and fragile selectors that break every time the site updates its layout.

If you’re done fighting with broken scripts or staring at impossible API invoices, this article will guide you towards scraping Twitter (X) trends without breaking the bank. We’re going to explore how you can scrape Twitter Trend data reliably and affordably in 2026 and beyond using the Twitter (X) Trends Scraper on Apify.

The "Now" Economy: Why You Can't Afford to Ignore Twitter Trends

Before we dive into the how, let's talk about the why. Why are brands, hedge funds, and news agencies so obsessed with "Trending Topics"?

Because social media moves so fast that if you blink, you're already behind. We’re currently living in a world where the rules are being rewritten every few weeks. AI influencers with millions of followers are the new norm, raw audio-first spaces are replacing video curation, and one-click social shopping has fundamentally changed how we spend money.

The old ways of measuring success are dying, too. We're seeing a pivot away from "likes" toward meaningful shares and saves, while the rise of "finstas" (fake Instagram accounts) shows just how much people are craving raw, unfiltered connection over curated perfection. In this type of landscape marked by ephemeral content and overtly personalized niche communities, you either have real-time data or you’re invisible.

This is why X (Twitter) is the global town square. While other platforms are for highlights and professional posturing, X is where news breaks, memes are born, and brand reputations are either made or broken in real-time. If you aren't tracking the pulse of X, you're trying to navigate the digital landscape with a map from ten years ago, ultimately making you obsolete.

Why Scrape Twitter?

Monitoring trends isn't just about watching hashtags. Monitoring trends is also about decoding the world’s most active conversation in real-time. Here is how businesses, researchers, and creators are turning raw X data into a competitive advantage:

Master the Art of "Newsjacking"

Newsjacking is the skill of injecting your brand into a breaking story at the perfect moment. Think back to Oreo’s "You can still dunk in the dark" tweet during a Super Bowl blackout. That tweet wasn't luck; it was real-time awareness. To pull this off, you need to know what is trending the moment it happens. Speed is the only metric that matters here.

Hyper-Local Market Intelligence

Global trends are interesting, but local trends are profitable. If you’re a retailer, knowing that #WinterCoats is trending in New York while #BeachVibes is dominant in Sydney allows for precision targeting. No matter where you are, whether in Lagos, London, or Los Angeles, having access to granular location data ensures your message hits the right people at the right time.

Unmasking Customer Behavior & Sentiment

What are people actually feeling? By scraping interactions and performing sentiment analysis, you can move past raw numbers to understand customer pain points and expectations. This emotional intelligence helps you tailor your products and services to what your audience actually wants, rather than what you think they want.

Competitive Intelligence on Autopilot

Monitoring your competitors’ activities reveals their marketing playbook in real-time. From spotting their latest product launches to seeing how their customers are reacting to a new policy, this data provides you with the tactical insights you need to stay one step ahead.

A Goldmine for Social & Academic Research

For researchers studying everything from political movements to public health trends, X is the ultimate dataset. Automated scraping provides the scale and historical context needed to uncover patterns, track the spread of information, and gauge public opinion on a global scale—all backed by hard data.

Content Inspiration on Tap

Writer's block is a productivity killer. By examining trending "tag clouds" and emerging discussions, content creators can instantly see what topics are currently resonating. This ensures your next post or video rides an existing wave of interest rather than struggling for attention in a vacuum.

The Problem: The Great Data Wall of X

The business case for this data is ironclad, but the reality is that actually accessing it has become a significant challenge.

The Death of the Free API

For a decade, the Twitter API was the playground of the internet. You could write a Python script in 10 minutes and stream tweets. Today, those gates are locked tight.

X has replaced the once-open ecosystem with a tiered pricing structure that makes data access either trivial or prohibitively expensive:

Free ($0/mo): Strictly for testing and bots that post content. You get a measly 100 posts/month in reads. That’s barely enough to refresh a single feed once.
Basic ($200/mo): Aimed at hobbyists and prototypes. It bumps you up to 15,000 reads per month, which disappears in a flash if you’re trying to track shifting trends across different regions.
Pro ($5,000/mo): This is where you get 1 million reads and full-archive search. It’s for scaling businesses, but the price tag is a massive barrier for most developers and small teams.
Enterprise: For anything larger, you’re looking at custom solutions. With enterprise pricing rumored to start at $42,000 a month, it’s clear that a "Pay-to-Play" reality has replaced the "Golden Age" of data access.

If you’re an indie developer or a small marketing team, these invoices aren't just an expense; they're definitely a death sentence for your project.

The "DIY Scraper" Trap

"Fine," you think, "I'll just whip up a Python script with Selenium or Playwright and do it myself."

Before you start writing that first line of code, you need to understand the technical gauntlet X has thrown down. Modern Twitter isn't just a website; it’s a system designed to keep automated traffic out. Here is why your "quick weekend project" will likely turn into a full-time maintenance nightmare:

Obfuscated and Dynamic DOM: X doesn't use semantic classes like trend-item. Instead, they use auto-generated, randomized strings such as css-175oi2r r-18u37iz. These can change during a deployment or even based on your session. Your selectors will break frequently, requiring you to rewrite your parsing logic every week.
Aggressive Fingerprinting: It’s not just about your IP address anymore. X analyzes your browser fingerprint, Canvas rendering, WebGL info, and even how you handle headers. If your headless browser looks like a bot, you’ll be met with a "Something went wrong" screen or a permanent block before you even fetch the trends.
The Resource Drain of SPAs: X is a heavy Single Page Application (SPA). To scrape it, you must run a full headless browser to execute JavaScript and handle infinite scrolling. This consumes massive amounts of CPU and RAM, especially when scaling. What starts "free" on your laptop quickly becomes a triple-digit server bill.
Evolving Bot Detection: Between Cloudflare protection and internal behavioral analysis, X is constantly looking for patterns. If you don't vary your mouse movements, scroll speeds, and request intervals perfectly, your scraper will be flagged and throttled into oblivion.

Suddenly, that "free" DIY project is costing you 20+ hours a week in troubleshooting to keep the data flowing.

The Solution: The Twitter (X) Trends Scraper on Apify

Enter the Twitter (X) Trends Scraper. This isn't just another script; it's a production-grade Actor hosted on the Apify platform. It serves as a bridge between the chaos of the open web and the structured data needs of your business. It bypasses the complexity of the official API and the fragility of DIY scrapers.

Here is why it’s the superior choice for 2026 and beyond.

1. Granular Location Targeting (The Secret Sauce)

Most generic scrapers only provide options for "Worldwide" or "USA". This Actor goes deeper. Much deeper. It leverages Twitter (X) data to allow you to select specific Cities and Countries.

Want to know what's buzzing in Kano, Nigeria? You can.
Need to compare trends in Birmingham, UK vs Birmingham, Alabama? Done. This level of detail is critical for localized marketing campaigns and sociological research.

2. No Proxies? No Problem

If you were running a scraper locally, you'd need to buy a pool of Rotating Residential Proxies to avoid getting blocked. These can cost hundreds of dollars a month. The Twitter Trend Scraper handles all of this infrastructure for you. When you run the Actor on Apify, it utilizes their vast pool of IP addresses. You don't need to configure anything; you click "Run".

3. Pay-Per-Event Pricing (The Fair Model)

Subscription models are annoying. Why pay $99/month if you only need to scrape data once a week? The Twitter Trend Scraper operates on a Pay-Per-Event model. You are charged a tiny fee only when you successfully scrape a scraping run. If you run it 10 times, you pay for 10 runs. If you don't use it for a month, you pay $0. It scales perfectly with your needs, from a hobbyist project to an enterprise dashboard.

4. Rich, Structured Data Output

It doesn't just give you a list of hashtags. You get a comprehensive JSON dataset including:

Rank: Is it #1 or #49?
Name: The hashtag or keyword.
Tweet Volume: "10K Tweets" vs "2M Tweets".
Context: The direct link to the search page.
History: Hourly timeline data.

Example Output from the Actor

[
  {
    "scraped_at": "2025-12-31T19:00:09.948042",
    "country_input": "Worldwide",
    "timeline": [
      {
        "datetime": "Wed Dec 31 2025 18:08:19 GMT+0000 (Coordinated Universal Time)",
        "timestamp": "1767204499.747",
        "trends": [
          {
            "rank": 1,
            "name": "Happy New Year",
            "link": "https://twitter.com/search?q=Happy%20New%20Year",
            "tweet_count": "2141151"
          },
          {
            "rank": 2,
            "name": "#CDTVライブライブ",
            "link": "https://twitter.com/search?q=%23CDTV%E3%83%A9%E3%82%A4%E3%83%96%E3%83%A9%E3%82%A4%E3%83%96",
            "tweet_count": "269605"
          },
          // ...
        ]
      },
      {
        "datetime": "Wed Dec 31 2025 17:16:15 GMT+0000 (Coordinated Universal Time)",
        "timestamp": "1767201375.892",
        "trends": [
          {
            "rank": 1,
            "name": "Happy New Year",
            "link": "https://twitter.com/search?q=Happy%20New%20Year",
            "tweet_count": "1708901"
          },
          {
            "rank": 2,
            "name": "新年早々",
            "link": "https://twitter.com/search?q=%E6%96%B0%E5%B9%B4%E6%97%A9%E3%80%85",
            "tweet_count": "80971"
          },
          // ...
        ]
      },
      {
        "datetime": "Wed Dec 31 2025 16:24:31 GMT+0000 (Coordinated Universal Time)",
        "timestamp": "1767198271.186",
        "trends": [
          {
            "rank": 1,
            "name": "Happy New Year",
            "link": "https://twitter.com/search?q=Happy%20New%20Year",
            "tweet_count": "1270217"
          },
          {
            "rank": 2,
            "name": "#STARTOtoMOVE生配信",
            "link": "https://twitter.com/search?q=%23STARTOtoMOVE%E7%94%9F%E9%85%8D%E4%BF%A1",
            "tweet_count": "282088"
          },
          // ...
        ]
      },
      {
        "datetime": "Wed Dec 31 2025 16:24:18 GMT+0000 (Coordinated Universal Time)",
        "timestamp": "1767198258.898",
        "trends": [
          {
            "rank": 1,
            "name": "Happy New Year",
            "link": "https://twitter.com/search?q=Happy%20New%20Year",
            "tweet_count": "1270217"
          },
          {
            "rank": 2,
            "name": "#STARTOtoMOVE生配信",
            "link": "https://twitter.com/search?q=%23STARTOtoMOVE%E7%94%9F%E9%85%8D%E4%BF%A1",
            "tweet_count": "282088"
          },
          // ...
        ]
      },
      // ...
    ],
    "tag_cloud": [],
    "table_data": []
  }
]

How to Get Started (In Under 5 Minutes)

You don't need to be a coding wizard to use this Actor. Here is your Zero-to-Data walkthrough.

Step 1: Access the Actor

Head over to the Twitter (X) Trends Scraper page on Apify. Click the "Try for free" button to create your account.

Step 2: Configure Your Input

You will see a user-friendly interface.

Look for the Country dropdown.
Select your target. Let's say... United Kingdom - London.
(Optional) You can leave everything else as the default.

Step 3: Run & Export

Hit the green Save & Start button at the bottom.
Boom. The Actor will spin up, navigate the web, extract the data, and shut down.
In a few seconds, you will see your results in the Output tab. You can view them as a table or download them in JSON, CSV, Excel, or XML formats.

For the Developers: Automating the Pipeline

If you are a developer, you probably want to integrate this Actor into your own app. Maybe you're building a dashboard that alerts you when crypto coins start trending.

You can control this Actor programmatically using the Apify Client for Python (or Node.js).

from apify_client import ApifyClient

# 1. Initialize the client
client = ApifyClient("YOUR_API_TOKEN")

# 2. Define your Input
run_input = {
    "country": "united-states/new-york",
}

# 3. Call the Actor
print("Fetching trends...")
run = client.actor("eunit/x-twitter-trends-scraper").call(run_input=run_input)

# 4. Fetch the results
dataset_items = client.dataset(run["defaultDatasetId"]).iterate_items()

for item in dataset_items:
    print(f"Stats for {item['country_input']}:")
    for trend in item['timeline'][0]['trends'][:5]:
        print(f"#{trend['rank']} {trend['name']} - {trend['tweet_count']}")

This snippet does in 10 lines what would take 500 lines of Selenium code to attempt (and fail) to do.

Is Scraping X Legal?

Ethical scraping is important. This actor scrapes publicly available factual data (trends). It does not scrape private profiles, log into accounts, or scrape personal data behind a login wall.
Scraping public information is considered legal, but it is essential to review the Terms of Service of any site you interact with and ensure your usage complies with relevant regulations such as the GDPR.

Wrapping Up

Let’s be real: trying to keep up with Twitter (X) without automated data is like trying to catch a waterfall with a spoon. You might see a few drops, but the wave you need to ride has usually passed you by.

The $42,000-per-month "pay-to-play" era was intended to render social listening for everyone but the biggest corporations. But as we've seen, you don't need a massive enterprise budget to get enterprise-grade insights.

Stop wasting your time fighting with broken Selenium scripts or staring at invoices you can't justify. Focus on what you actually do best: analyzing the trends and making informed business decisions.

Try the Twitter (X) Trends Scraper today and start turning the chaos of social media into a clear signal for your success.

How To Automatically Submit Sitemap To Google Programmatically

Emmanuel Uchenna — Mon, 29 Dec 2025 12:32:21 +0000

Generating a sitemap is only the first half of the SEO battle. If you’ve read our previous article on how to create and submit an XML sitemap, you already know how to use the Fast Sitemap Generator to crawl your site and create a clean, search-engine-ready file. However, here is the frustrating reality: even after you’ve uploaded that sitemap to Google Search Console, you might still be waiting days, weeks, or even months for Google to crawl and index those new pages.

If you are running a news website, an e-commerce store with changing inventory, or a platform that publishes time-sensitive content, you don't have time to wait for Google’s "natural selection" process. You need your content indexed now.

In this article, we’ll walk you through how to take control of your indexing by automatically submitting your sitemap URLs to Google programmatically. We will cover the manual way (the "hard" way) using Python and the Google Indexing API, and then we’ll show you the professional way to automate the entire pipeline using the Google Indexer & Instant SEO Submitter Apify Actor.

Note: This post was originally published in Eunit.me

Why Just Submitting A Sitemap Isn't Enough

When you submit a sitemap in Google Search Console, you are essentially leaving a note on Google’s doorstep saying, "Hey, I have some new stuff in here whenever you're ready." Google will address it when it wants to. For most sites, this is fine. But if you have:

Frequently updated content: Price changes, stock availability, or breaking news.
A massive site: Large sites often struggle with "crawl budget." You want to tell Google exactly which pages are the most important at the moment.
A brand new domain: New sites have zero authority and might sit in the "discovered - currently not indexed" purgatory for a long time.

The Google Indexing API addresses this issue. While originally intended for jobs and live broadcast content, SEO professionals have found it to be incredibly effective for almost any type of content looking for an "instant" crawl nudge.

Part 1: Setting Up The Google Cloud Infrastructure

Before you can write a single line of code or use any automation tools, you need to navigate the Google Cloud Console. This is where most people tend to get stuck, so let's walk through it step by step.

1. Create a Google Cloud Project

Go to the Google Cloud Console. If you don't have a project yet, click on the project dropdown in the top-left corner and select New Project. Give it a descriptive name, such as "SEO Indexing Automator."

2. Enable the Indexing API

Indexing is not enabled by default. You need to tell Google you intend to use it.

Navigate to APIs & Services > Library.
Search for "Web Search Indexing API" (sometimes just called "Indexing API").
Click Enable.

3. Create a Service Account

This is the "person" who will be acting on your behalf.

Now, go back to IAM & Admin > Service Accounts.
Click Create Service Account.
Give it a name like "indexing-bot" and a description.
For the role, you can choose Owner for simplicity during setup, or Project > Editor for a more restricted approach.
Once created, click on the email address of the service account. It should look something like indexing-bot@your-project.iam.gserviceaccount.com. Copy this email; you need it for GSC later.

4. Create and Download the JSON Key File

Inside your service account details, go to the Keys tab.
Click Add Key > Create new key.
Select JSON and click Create. Your browser will download a file. Keep this safe! It contains the "password" to your indexing bot.

Part 2: Connect the bot to Google Search Console

This is the step everyone misses. Even if you have the API enabled, Google won't let your service account submit URLs unless it has verified ownership of the site.

Open Google Search Console.
Select the property (website) you want to automate.
Go to Settings > Users and permissions.
Click Add user.
Paste the service account email you copied earlier.
Set the permission to Owner. (Google requires owner-level permissions to use the Indexing API).

Now the pipes are connected. Let's look at how to push the data.

Part 3: The Developer Way - Programmatically Submitting Urls

If you want to build this yourself, you can write a script that parses your sitemap and sends each URL to the API. Here is a high-level overview of how you'd do it in Python.

The "DIY" Python script

You’ll need a few libraries:

pip install requests google-auth google-auth-httplib2 lxml

The script works by reading your sitemap XML and iterating through the <loc> tags. For each URL, it makes a POST request to Google.

import requests
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession
from lxml import etree

# Constants
SERVICE_ACCOUNT_FILE = "./your-key.json"
SITEMAP_URL = "https://example.com/sitemap.xml"
API_URL = "https://indexing.googleapis.com/v3/urlNotifications:publish"

def submit_to_google(url, authed_session):
 data = {"url": url, "type": "URL_UPDATED"}
 response = authed_session.post(API_URL, json=data)
    return response.status_code

# 1. Auth
credentials = service_account.Credentials.from_service_account_file(
    SERVICE_ACCOUNT_FILE, 
    scopes=["https://www.googleapis.com/auth/indexing"]
)
session = AuthorizedSession(credentials)

# 2. Parse Sitemap
resp = requests.get(SITEMAP_URL)
root = etree.fromstring(resp.content)
urls = [loc.text for loc in root.findall(".//{http://www.sitemaps.org/schemas/sitemap/0.9}loc")]

# 3. Submit
for url in urls:
 status = submit_to_google(url, session)
    print(f"Submitted {url}: {status}")

The Limitations Of DIY Automation

While the script works, it lacks several production features:

Rate limiting: Google has strict quotas (usually 200 URLs per day). You need to handle 429 errors gracefully.
Incremental updates: You don't want to submit 10,000 URLs every day if only 5 are new.
Reporting: You need to know which URLs failed and why.
Scheduling: You have to host this script somewhere and set up a cron job.

Part 4: The Professional Way - Using The Apify Platform

If you don't want to manage servers, handle complex authentication boilerplate, or write your own rate-limiting logic, you should use the Google Indexer & Instant SEO Submitter on the Apify platform.

This Actor is designed to handle all the heavy lifting for you. It can read your sitemap directly, handle authentication via your JSON key, and even integrate with other Actors to form a complete SEO pipeline.

Why Use The Google Indexing Actor?

No-Code configuration: Paste your JSON key and your sitemap URL.
Dataset integration: It can take a dataset_id from a previous crawl (like from the Fast Sitemap Generator) and index those results immediately.
Error handling: It automatically detects rate limits and waits or stops as needed to prevent account flag.
Pay-Per-Event pricing: You only pay for successful submissions. It is incredibly cost-effective, starting at just $0.01 per 1,000 results.
Test mode: You can run a dry-run to see which URLs would be submitted without actually calling the API or incurring costs.

Part 5: Building a Complete SEO Automation Pipeline

The real power of Apify comes when you connect these tools. Instead of manually generating a sitemap and then submitting it manually, you can automate the entire chain.

The Workflow

Step 1: Use the Fast Sitemap Generator to crawl your site. This Actor will find all active pages and generate a sitemap. Check out our previous article detailing how to generate your free sitemap using the Fast Sitemap Generator.
Step 2: The Google Indexer & Instant SEO Submitter takes the output of the first Actor.
Step 3: It iterates through the discovered URLs and notifies Google.

How to set this up on Apify

Run the Fast Sitemap Generator with your Start URLs.
Copy the Dataset ID from the run results.
Open the Google Indexer & Instant SEO Submitter.
In the Dataset ID field, paste the ID you just copied.
Add your Google Service Account JSON.
Click Start.

The Actor will now intelligently parse the XML generated by the first step and submit every unique URL it finds. Because it tracks sources, it ensures you aren't double-charged or submitting duplicates.

Best Practices For Programmatic Indexing

To get the most out of these tools, keep these tips in mind:

Don't spam Google: Use URL_UPDATED only for truly new or significantly updated content. If you are testing, use the Test Mode toggle in the Actor settings.
Match your canonicals: Ensure the URLs in your sitemap are the exact canonical versions. If your site uses https://www., don't submit https://.
Monitor the quotas: Standard projects get 200 requests per day. If you have a massive site, you can request a quota increase from Google, but the Google Indexing Actor will help you manage this limit automatically by stopping when the limit is reached.
Check Search Console: After running the automation, keep an eye on the "Indexing" report in GSC. You should see "Crawled - currently not indexed" change to "Indexed" much faster than usual.

Wrapping Up

The days of waiting for Google to notice your hard work are over. By combining the power of the Fast Sitemap Generator and the Google Indexer & Instant SEO Submitter, you can build a professional-grade SEO engine that ensures your content is live in search results within hours, not weeks.

How to Generate and Submit an XML Sitemap: The Ultimate Guide

Emmanuel Uchenna — Sun, 28 Dec 2025 23:59:42 +0000

Note: This blog was originally published in Eunit.me

Nowadays, creating a website is only half the worry for getting a business online. The other half of the concern, and arguably the more critical one, is ensuring that people can actually find your website. This is where Search Engine Optimization (SEO) comes into play. While keyword research, backlink building, and content strategy are often the "glamour" stats of SEO, there is a fundamental technical element that serves as the bedrock of your site’s visibility, and that is The XML Sitemap.

In this article, we will walk you through everything you need to know about sitemaps: what they are, why they are important for modern SEO, and most importantly, how to generate a sitemap automatically without writing a single line of code using the Fast Sitemap Generator on Apify. Finally, we’ll show you how to submit your sitemap to Google Search Console to get your pages indexed faster than ever before.

What is an XML Sitemap and Why Do You Need One?

An XML Sitemap (Extensible Markup Language) is a text file that lists all the URLs on your website that you want search engines to index. But it’s more than just a list; it provides crucial metadata about each URL, including:

Last Modified Date (<lastmod>): Tells Google when the page was last updated, encouraging re-crawling of fresh content.
Change Frequency (<changefreq>): A hint to crawlers about how often the page changes (e.g., "daily" for a news homepage vs. "yearly" for an "About Us" page).
Priority (<priority>): A numerical value (0.0 to 1.0) indicating the relative importance of a page within your site.

The Benefits of XML Sitemaps

You might ask, "Doesn't Google crawl my site anyway?" Yes, but relying solely on Google's crawler (Googlebot) following links has limitations:

Isolated Pages: If a page isn't linked to from another page (an "orphan page"), Googlebot can't find it. A sitemap lists it explicitly.
New Websites: New sites have few backlinks. Without a reputation, crawl budgets are low. A sitemap requests immediate attention.
Large Archives: E-commerce sites or blogs with thousands of pages can confuse crawlers. A sitemap ensures deep pages aren't ignored.
Rich Media: Specialized sitemaps (Video, Image) help your multimedia assets appear in Google Images and Video Search.

The Old Way vs. The Automated Way

Historically, creating a sitemap was a tedious task.

The Manual Method: Opening a text editor and hand-coding <url> tags. For a 5-page site, it’s fine. For a 100-page site, it’s a nightmare. One typo breaks the file.
CMS Plugins: If you use WordPress, plugins like Yoast are great. But what if you have a custom React/Next.js site like this one? Or a static HTML site? Or what if you want to audit a competitor's site structure? Plugins don't work there.
Desktop Crawlers: Tools like Screaming Frog are powerful but resource-heavy. They tie up your computer, rely on your local internet connection (slow), and require manual exporting and uploading.

The Solution: Cloud-Based Automation

Enter Apify Actors. Actors are serverless cloud programs that perform specific tasks. The Apify Fast Sitemap Generator is a specialized Actor developed to solve the sitemap problem effortlessly. Unlike desktop tools, it runs in the cloud, works on any website, and is completely automated and fast.

How to Generate an XML Sitemap with Apify

Let’s get technical. We are going to use the Apify Fast Sitemap Generator Actor to crawl a website and produce a compliant XML sitemap, along with HTML and TXT versions for good measure.

Step 1: Access the Tool

Navigate to the Sitemap Generator Actor on Apify.

Note: You will need an Apify account. The free tier is generous enough for testing and small crawls. Create one by signing up on Apify.

This specific Actor uses a Direct Connection, meaning it crawls directly from the data center without proxies. This ensures high speed and lower costs, though it requires your target site to be accessible to standard web traffic.

Step 2: Configure Your Input

Once you click "Try for free" or "Run", you’ll be taken to the Apify Console. Here is where you tell the crawler what to do.

1. Start URLs:
In the Start URLs field, enter the homepage of the site you want to map.

Example: https://www.your-website.com

2. Crawl Depth (maxCrawlDepth):
This determines how "deep" the crawler goes.

Depth 0: Just the homepage.
Depth 1: Homepage + pages linked directly from it.
Depth 3 (Default): Usually sufficient for most sites to find all content.

Tip: Set this to 10+ if your site has endless pagination or deep category structures.

3. Filtering with Regex (includePatterns / excludePatterns):
This is a superpower. You don't want to index your admin pages, cart pages, or user-specific accounts.

Exclude: Add patterns like .*/admin/.*, .*/login.*, or .*/cart.* to skip these.
Include: Leave empty to crawl everything, or specify patterns to only map a blog section (e.g., .*/blog/.*).

4. Formats:
Select the outputs you need.

xml: Essential for search engines. Recommended.
html: Great for a visible "Site Map" page for human visitors.
txt: A simple list of URLs, often used for content audits or programmatic processing.

Step 3: Run the Generator

Click the green Start button at the bottom. The Actor will now:

Launch: Spin up a container in the cloud.
Crawl: Systematically visit every link on your site, respecting your robots.txt rules (unless you disabled that option).
Construct: Build the XML structure with the correct lastmod dates.
Save: Store the files in a persistent Key-Value Store.

Why is this better?

Pay-Per-Event Pricing: You only pay for the pages successfully discovered. Efficiency is built in.
Speed: It can process thousands of pages in minutes without using your computer's RAM.

Step 4: Retrieve Your Sitemap

Once the run shows "Succeeded", navigate to the Output tab.
You will see a Dataset containing the direct links to your generated files.

It will look something like this:

{
  "format": "xml",
  "url": "https://api.apify.com/v2/key-value-stores/YOUR_STORE_ID/records/sitemap.xml"
}

Click the link to download your sitemap.xml.

Pro Tip: You can also download the sitemap.html file and upload it to your site to instantly create a helpful navigation page for users!

How to Submit Your Sitemap to Google Search Console

Now that you have the file, you need to inform Google where it is located.

Phase A: Upload to Your Server

Download the sitemap.xml from Apify.
Upload it to the root directory of your website via FTP or your hosting file manager (e.g., public_html).
Your sitemap should be accessible at: https://www.your-website.com/sitemap.xml.

Phase B: Submit to GSC

Log in to Google Search Console.
Select your property (website) from the dropdown.
In the left sidebar, click on Sitemaps (under the "Indexing" section).
In the "Add a new sitemap" field, enter the filename (e.g., sitemap.xml).
Click Submit.

Phase C: Verify

Google will process the submission instantly. You should see a status of "Success".

If you see "Could not fetch", wait a few hours or double-check that your URL is publicly accessible.
Clicking on the submitted sitemap will show you the "Discovered URLs" count. Does this match the number of pages the Apify Actor found? If so, you’re golden!

Advanced Automation: Set It and Forget It

The beauty of using the Apify Sitemap Generator lies in its ability to automate tasks. Your website content changes, you add blog posts, remove products, and update pages. Your sitemap must reflect these changes, or Google will be indexing "dead" content.

Use the Google Indexer & Instant SEO Submitter Actor

The Google Indexer & Instant SEO Submitter Actor on Apify enables you to automatically submit your generated sitemap to Google programmatically, instead of having to go through the above-listed phases (A-C). We wrote a detailed guide on How to Submit Sitemap to Google Programmatically.

Wrapping Up

An XML sitemap is a small file with a massive impact. It is the bridge between your content and the search engines that deliver your audience. By moving away from manual creation and using automated, intelligent tools like the Sitemap Generator Actor, you ensure that this bridge is always sturdy, accurate, and open for traffic.

Happy Crawling!

How to Scrape Real Estate Data from Zillow in 2026 (Step-by-Step Guide)

Emmanuel Uchenna — Fri, 19 Dec 2025 09:30:53 +0000

How to Scrape Real Estate Data from Zillow in 2026 (Step-by-Step Guide)

Data is the oxygen of the modern real estate market. Whether you are an investor looking for the next big opportunity, a realtor analyzing market trends, or a developer building a property aggregator, a student, or a professional, having access to accurate, up-to-date listing data is a game-changer. And when it comes to US real estate, Zillow is the undisputed king of data for the real estate market.

However, extracting bulk data from Zillow is notoriously difficult. The platform relies on complex map-based interfaces and employs sophisticated anti-scraping measures to block automated access. Zillow currently employs several measures to prevent scraping, including IP blocking, CAPTCHA challenges (HUMAN Security), and dynamic content loading.

In this article, we will walk you through how to bypass these challenges and scrape thousands of rental listings, including prices, addresses, and full photo galleries- using the Zillow Rental Data Scraper on the Apify platform.

Why scrape Zillow data?

Web scraping transforms Zillow's vast data into structured datasets (Excel, CSV, JSON) for different downstream applications. It enables professionals to automate the collection of listings, price history, and market stats, unlocking insights that manual searching cannot match.

Key Use Cases

Market Research: Scraping real estate data allows you to monitor local trends, supply/demand balance, and regional pricing to gauge market health.
Investment Analysis: Identify undervalued assets, calculate ROI, and track value appreciation to build profitable portfolios.
Competitor Intelligence: Analyze how similar properties perform, track listings, and understand market positioning.
Location Analytics: Assess neighborhood demographics, amenities, and property characteristics for development or relocation planning.
Proptech & App Development: Feed real-time real estate data into new platforms, apps, or AI models for predictive analytics.
Rental Management: Analyze rental demand, set optimal pricing, and understand tenant preferences.

What Data Is Collected?

The Zillow Rental Data Scraper extracts detailed information for every listing found in your search area:

Property Identity: Full address, building name, Zillow Property ID (zpid), and precise GPS coordinates.
Rental Details: Current pricing, available units, bedroom/bathroom counts, and square footage.
Rich Media: High-resolution image URLs, including the complete photo gallery for each property.
Listing Metadata: Listing status (e.g., FOR_RENT), availability counts, and the direct URL to the Zillow listing page.

The challenge: Why is Zillow hard to scrape?

Zillow does not offer a public API for bulk listing data. Additionally, it utizes a map-based search interface that loads data dynamically as you pan and zoom the map. Standard web scrapers often fail because:

Map-Based Pagination: You can't just page through results; you often need to "move" a virtual map to find all listings in a city.
Sophisticated Anti-Scraping Security: Zillow uses PerimeterX (HUMAN Security), an enterprise-grade solution that analyzes browser fingerprints, mouse movements, and keystrokes. It triggers a difficult CAPTCHA or blocks IPs immediately if it detects bot-like behavior.
IP Blocking: Standard data center IPs (like AWS or Google Cloud) are flagged and blocked almost instantly.
Complex Data Structure: Vital information like high-resolution photos is often hidden behind dynamic user interactions (XHR requests).

The solution: Zillow Rental Data Scraper

The Zillow Rental Data Scraper is a powerful Apify Actor designed specifically to overcome these hurdles. Unlike generic scrapers, the Zillow Scraper mimics the Zillow map-view search behavior. You define a geographic "box" (using latitude and longitude), and the Actor systematically sweeps that area to capture every listing provided by Zillow.

Key capabilities:

Geographic Precision: Scrape listings within a precise map area using a bounding box.
Rich Media Extraction: Captures all available photo URLs for each property, not just the thumbnail.
Granular Filters: Apply min/max price, bedroom/bathroom counts, and property type filters (e.g., houses vs. apartments).
Bypassing Blocks: Built to work seamlessly with Residential Proxies to avoid detection.

Step-by-step guide: How to use the Zillow Scraper

You don't need to write a single line of Python code to get this data. Follow these steps to start scraping in minutes.

1. Find your target area

First, you need to define the geographic area you want to scrape.

Go to Zillow.com.
Search for your target city (e.g., "Austin, TX").
Zoom the map to cover the exact area you are interested in.
You will use the coordinates of this area (North-East and South-West corners) to tell the scraper where to look. You can find these in the URL or use a simple online bounding box tool.

2. Set up the Actor on the Apify platform

Log in to your Apify account or create a new one if you don't have one.
Navigate to the Zillow Rental Data Scraper page.
Click Try for free to open the Actor console.

3. Configure your input

In the Input tab, you will specify exactly what the scarper should look for.

Search Parameters:
Set your filters to match your needs. For example, if you are looking for affordable 1-bedroom apartments:

Min Price: $1,200
Max Price: $2,500
Min Bed: 1
etc

Geographic Boundary:
Enter the coordinates you found in Step 1. This ensures the scraper focuses only on your target market.

ne_lat (North East Latitude)
sw_long (South West Longitude)
etc.

Proxy Configuration (CRITICAL):
Zillow has strict security. To ensure your run succeeds:

Enable Proxy configuration.
Select Residential proxies.
Note: Using Datacenter proxies usually results in empty datasets because Zillow blocks them immediately.

4. Run the scraper

Click the Start button. The Actor will spin up, navigate to Zillow using the residential proxies, apply your filters, and start extracting data.

You can watch the logs to see progress in real-time. The scraper will output "Extracted details" for each property it finds.

5. Download your data

Once the run is finished, go to the Storage tab. You can preview your data or download it in your preferred format:

Excel: Great for manual review and sorting.
CSV: Ideal for importing into databases or CRMs.
JSON: Best for developers integrating the data into apps.

Sample Output Data:
You will get a clean, structured dataset for every property:

Example Output

{
  "zpid": "30.46529--97.60529",
  "id": "30.46529--97.60529",
  "providerListingId": "5c5nn5nycnc5q",
  "imgSrc": "https://photos.zillowstatic.com/fp/7d1a61d3b011516c8a5204f1b398e474-p_e.jpg",
  "hasImage": true,
  "detailUrl": "https://www.zillow.com/apartments/pflugerville-tx/the-beacon-at-pfluger-farm/CjjCGn/",
  "statusType": "FOR_RENT",
  "statusText": "The Beacon at Pfluger Farm",
  "address": "1300 Rauscher Rd, Pflugerville, TX",
  "addressStreet": "1300 Rauscher Rd # f52cfdc49",
  "addressCity": "Pflugerville",
  "addressState": "TX",
  "addressZipcode": "78660",
  "units": [
    {
      "price": "$1,214+",
      "beds": "1",
      "roomForRent": false
    },
    {
      "price": "$1,782+",
      "beds": "2",
      "roomForRent": false
    }
  ],
  "lotId": 2748185226,
  "latLong": {
    "latitude": 30.46529,
    "longitude": -97.60529
  },
  "listCardRecommendation": {
    "flexFieldRecommendations": [
      {
        "displayString": "11 available units",
        "contentType": "frUnitsAvailable"
      }
    ],
    "ctaRecommendations": [
      {
        "displayString": "Request a tour",
        "contentType": "REQUEST_A_TOUR"
      }
    ]
  },
  "isSaved": false,
  "buildingName": "The Beacon at Pfluger Farm",
  "isBuilding": true,
  "canSaveBuilding": true,
  "has3DModel": false,
  "isFeaturedListing": true,
  "isShowcaseListing": false,
  "list": true,
  "relaxed": false,
  "carouselPhotosComposable": {
    "baseUrl": "https://photos.zillowstatic.com/fp/{photoKey}-p_e.jpg",
    "communityBaseUrl": null,
    "photoData": [
      "https://photos.zillowstatic.com/fp/7d1a61d3b011516c8a5204f1b398e474-p_e.jpg",
      "https://photos.zillowstatic.com/fp/6ef9281fdb0d7e13b9fb90a3cec20987-p_e.jpg",
      "https://photos.zillowstatic.com/fp/1069c481833fdd6fa45c27fe7296ece7-p_e.jpg",
      "https://photos.zillowstatic.com/fp/3ae52944a1839e27510901aa70f03bb6-p_e.jpg",
      "https://photos.zillowstatic.com/fp/538f0be96c53927b7a1e3f536d05fed0-p_e.jpg",
      "https://photos.zillowstatic.com/fp/ba77a295f04bc79a14d93ed34dcc1e5f-p_e.jpg",
      "https://photos.zillowstatic.com/fp/2f5c50213efb1e57356fc2d9eb332089-p_e.jpg",
      "https://photos.zillowstatic.com/fp/bc2f048dd1d895baabd564e6f733b07c-p_e.jpg",
      "https://photos.zillowstatic.com/fp/90d8a596e849d8a67d3b05dcad9a038c-p_e.jpg",
      "https://photos.zillowstatic.com/fp/c3db6fee321ab87e0aec5574e2b96798-p_e.jpg",
      "https://photos.zillowstatic.com/fp/66f87c6c89345569fa9969bc7752e69c-p_e.jpg",
      "https://photos.zillowstatic.com/fp/9b4d966520512aafe6cc452a9e3c03a2-p_e.jpg",
      "https://photos.zillowstatic.com/fp/b3e544dcadcd50e9909c13cda614409c-p_e.jpg",
      "https://photos.zillowstatic.com/fp/88b022e258a3689392794869bce2c8b5-p_e.jpg",
      "https://photos.zillowstatic.com/fp/04aa9848fd82a6af259b8ee4057ddada-p_e.jpg",
      "https://photos.zillowstatic.com/fp/64f3e72700fa174491609240a2772348-p_e.jpg",
      "https://photos.zillowstatic.com/fp/e2024d6f788006f6f646e908030db1f6-p_e.jpg",
      "https://photos.zillowstatic.com/fp/7b9ec973819dcde5d354fe5fb396fb35-p_e.jpg",
      "https://photos.zillowstatic.com/fp/c2f38c90c4628d1a472a445b1b79b87f-p_e.jpg",
      "https://photos.zillowstatic.com/fp/010b8274d97cb58d03f43f33511e1c16-p_e.jpg",
      "https://photos.zillowstatic.com/fp/c83aab0ab0194622a262e019044b40eb-p_e.jpg",
      "https://photos.zillowstatic.com/fp/8488b9de4f2ac78b1ab8d661495205e1-p_e.jpg",
      "https://photos.zillowstatic.com/fp/3758a24dc4ff6f2df99d24a10b948499-p_e.jpg",
      "https://photos.zillowstatic.com/fp/0198c3f273d9369db18ccb371870397d-p_e.jpg",
      "https://photos.zillowstatic.com/fp/6245e857a3d8f784f3cf8d7e85b8b037-p_e.jpg",
      "https://photos.zillowstatic.com/fp/340b5363f062cc09a7876d1d6772fa96-p_e.jpg",
      "https://photos.zillowstatic.com/fp/ddd360f0f1db438298ea438c35afdc16-p_e.jpg",
      "https://photos.zillowstatic.com/fp/8b2392852d76970f4e38bb444dde9118-p_e.jpg",
      "https://photos.zillowstatic.com/fp/7c93c4adaeff5e718db6a1878515300a-p_e.jpg",
      "https://photos.zillowstatic.com/fp/402ff64e1592cff104968d216228fd15-p_e.jpg",
      "https://photos.zillowstatic.com/fp/41fc7e98b78f44a1b098479003c15c0f-p_e.jpg",
      "https://photos.zillowstatic.com/fp/1e96a1b3214f06e3352aa6e9267f9a7a-p_e.jpg",
      "https://photos.zillowstatic.com/fp/729f2ba54acab8ef7454af2fa4435239-p_e.jpg",
      "https://photos.zillowstatic.com/fp/f17991597d432633505cd1bc16b19017-p_e.jpg",
      "https://photos.zillowstatic.com/fp/c9e726647b8d47aa0afe17b5e88e1cd1-p_e.jpg",
      "https://photos.zillowstatic.com/fp/214b658c9de15f14982974d07663111d-p_e.jpg",
      "https://photos.zillowstatic.com/fp/e7f14898a3770034f6a713bcea34f4c7-p_e.jpg",
      "https://photos.zillowstatic.com/fp/4819e0d2edc6d2d32badeba4fdea044b-p_e.jpg",
      "https://photos.zillowstatic.com/fp/68406fbf2c016fd38fdec02e411b155d-p_e.jpg",
      "https://photos.zillowstatic.com/fp/2111ff9c000d2d5f3c37b1d838f902d6-p_e.jpg",
      "https://photos.zillowstatic.com/fp/e12b2a8952536f45e55c6716de2ef273-p_e.jpg",
      "https://photos.zillowstatic.com/fp/a0e7c07753f64ebbdf9e2b186e3f368f-p_e.jpg",
      "https://photos.zillowstatic.com/fp/f8168301283fe4477153eac9c1320fc8-p_e.jpg",
      "https://photos.zillowstatic.com/fp/1971944837586c1bca34fa72f088ef85-p_e.jpg",
      "https://photos.zillowstatic.com/fp/7f0cd466eb612c36955aabd3346020ae-p_e.jpg",
      "https://photos.zillowstatic.com/fp/f2159903758d257c046e8c4db8b9630e-p_e.jpg",
      "https://photos.zillowstatic.com/fp/eb3f16643202f129ec81c8997138f392-p_e.jpg",
      "https://photos.zillowstatic.com/fp/87fbce9721b2c588e3d27080e3deacfa-p_e.jpg"
    ],
    "communityPhotoData": null,
    "isStaticUrls": false
  },
  "availabilityCount": 11,
  "marketingTreatments": [
    "trustedListing",
    "paid",
    "zillowRentalManager",
    "multiFamilySalesListing",
    "paidOrUnpaidMultifamily",
    "paidMultifamily",
    "multifamilyPremium"
  ],
  "listPriceIncludesRequiredMonthlyFees": false,
  "isInstantTourEnabled": false,
  "isContactable": true,
  "isPaidBuilderNewConstruction": false
}

Is scraping Zillow legal?

Scraping publicly available factual data (like addresses, prices, and facts about a property) is generally considered legal in the US, provided you do not breach login barriers or copyright protections (like re-publishing creative descriptions or photos without permission).

However, you must always respect Zillow's Terms of Service and specific laws like GDPR or CCPA if personal data is involved. If you plan to use the data commercially or at scale, we recommend consulting with a legal professional.

Wrapping Up

We've covered why Zillow data is a goldmine for real estate professionals and the significant technical hurdles—like map-based pagination and sophisticated anti-scraping defenses (PerimeterX), that make extracting it so difficult.

The Zillow Rental Data Scraper on Apify offers a robust, no-code solution to bypass these barriers. By leveraging residential proxies and mimicking real user behavior, you can reliably harvest rich datasets, including listings, high-resolution photos, and pricing history.

Ready to start scraping? Get the Zillow Rental Data Scraper!

How to Scrape LinkedIn Job Postings with Python: A Step-by-Step Guide

Emmanuel Uchenna — Thu, 18 Dec 2025 19:17:45 +0000

LinkedIn is the world's largest professional network, making it a goldmine for data on job market trends, company hiring patterns, and employment opportunities. Whether you are a recruiter, a market researcher, or a developer building a job board aggregator, automating the collection of this data can provide a significant competitive advantage.

In this article, we will walk you through how to build a robust LinkedIn Job Postings Scraper using Python. We will cover how to handle common challenges, such as infinite scrolling and anti-bot protections, using the Apify Residential Proxies.

Tip: Don't want to build it from scratch? specific Check out the ready-to-use production-grade LinkedIn Job Postings Scraper Actor on the Apify Store.

Why Scrape LinkedIn Job Data?

Scraping LinkedIn data opens up a wide range of powerful use cases:

Lead Generation: Create highly targeted lists based on job titles, industries, or specific technical skills to drive personalized and effective outreach campaigns.
Competitor Intelligence: Gain insights into the competitive landscape by tracking your competitors' hiring patterns, growth trajectories, and organizational structures.
Recruitment & Sourcing: Go beyond basic keyword matching to discover passive candidates and build deep talent pipelines filled with the exact skills you need.
Market Research: Monitor emerging industry trends, skill demand shifts, and the geographical distribution of talent.
Academic Studies: Gather data for analyzing labor market dynamics, professional migration patterns, and economic correlations.

In short, if valuable data exists on a public LinkedIn page, ethical web scraping is a scalable method for aggregating it for business or research insights.

The Challenges of Scraping LinkedIn

LinkedIn is known for its strict anti-scraping measures. If you try to scrape it with a simple script, you will likely face:

IP Bans: Frequent requests from the same IP address will trigger rate limits.
Infinite Scrolling: Job lists load dynamically as you scroll, complicating pagination.
Login Walls: Many pages require authentication, which risks flagging your personal account.

To overcome these challenges, we will utilize the Apify SDK for Python and Residential Proxies, which enable us to route requests through legitimate devices, making our traffic indistinguishable from real users.

Prerequisites

Before we start, ensure you have:

Python 3.8+ installed on your machine.
An Apify Account (you can sign up for free).
Basic knowledge of CSS selectors.

Step 1: Setting Up the Environment

We will use the Apify Python SDK to manage our scraper's execution and storage. You can start by using the Apify CLI to create a new boilerplate project.

npm install -g apify-cli
apify create linkedin-scraper -t python-start
cd linkedin-scraper

Install the necessary Python libraries:

pip install apify httpx beautifulsoup4 httpx-socks

apify: For managing the Actor's lifecycle and storage.
httpx: A modern, asynchronous HTTP client.
beautifulsoup4: For parsing HTML content.

Step 2: Handling Infinite Scrolling and Pagination

LinkedIn's job search page uses infinite scrolling. Instead of trying to simulate scroll events (which is slow and flaky), we can reverse-engineer the hidden internal API used by the frontend.

# The base URL pattern
list_url = f"https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search"

This approach allows us to scrape thousands of jobs without ever needing to render the full page in a browser, significantly speeding up the process.

Step 3: Implementing Residential Proxies

This is the most critical part. To avoid getting blocked, you must use high-quality proxies. The LinkedIn Job Postings Scraper is robustly designed to use Apify Residential Proxies when available, with a fallback mechanism for local testing.

Here is how we configure the httpx.AsyncClient to use a specific proxy country (e.g., US) to ensure we see relevant job data:

# Handle Proxy Configuration
proxy_country = actor_input.get('proxyCountry')
proxy_url = None

try:
    # Attempt to use specific Residential Proxies (e.g., US)
    proxy_configuration = await Actor.create_proxy_configuration(
        groups=['RESIDENTIAL'],
        country_code=proxy_country
    )
    proxy_url = await proxy_configuration.new_url()
except Exception as e:
    Actor.log.warning(f"Residential proxy unavailable: {e}. Falling back...")
    # ... fallback logic handles local execution safely ...

# Configure the HTTP client
client_kwargs = {'timeout': 30.0}
if proxy_url:
    client_kwargs['proxy'] = proxy_url

async with httpx.AsyncClient(**client_kwargs) as client:
    # ... scraping logic ...

By allowing users to specify a proxyCountry, the scraper can view job listings exactly as they appear to a user in that region.

Step 4: Scraping and Parsing Data

Once we have the HTML, we use BeautifulSoup to extract key details, such as the Job Title, Company Name, Location, and Posting Date.

def get_job_data(soup):
    return {
        "title": soup.find("h2", "top-card-layout__title").get_text(strip=True),
        "company": soup.find("a", "topcard__org-name-link").get_text(strip=True),
        "location": soup.find("span", "topcard__flavor--bullet").get_text(strip=True),
        "job_url": soup.find("a", "topcard__link")['href'],
        # ... more fields ...
    }

Running the Actor

You can run this scraper directly on the Apify platform (recommended). Go to the LinkedIn Job Postings Scraper page.

Go to the LinkedIn Job Postings Scraper page.
Click Try for free.
Fill in your input:
- Keywords: Software Engineer
- Location: United States
- Proxy Country: US (Recommended for US jobs)
Hit Start.

Output Example

The data will be clean, structured, and ready for use.

[
    {
        "title": "software Engineer I",
        "company": "The Walt Disney Company",
        "job_url": "https: //www.linkedin.com/jobs/view/software-engineer-i-at-the-walt-disney-company-3970118620?trk=public_jobs_topcard-title",
        "logo": "https: //media.licdn.com/dms/image/D560BAQGmPH1QqmCzkg/company-logo_100_100/0/1688494223337/fieldguide_inc_logo?e=2147483647&v=beta&t=KqO_GS4C7oH3nVoyvtDCXbbhAvj2OeLbfty0yJYZCbc",
        "date_posted": "4 days ago",
        "pay_range": "$98,000.00/yr - $131,300.00/yr",
        "location": "seattle, WA",
        "organization": "The Walt Disney Company",
        "number_of_applicants": "Be among the first 25 applicants",
        "job_description": "On any given day at Disney Entertainment & ESPN Technology, we’re reimagining ways to create magical viewing experiences for the world’s most beloved stories while also transforming Disney’s media business for the future. Whether that’s evolving our streaming and digital products in new and immersive ways, powering worldwide advertising and distribution to maximize flexibility and efficiency, or delivering Disney’s unmatched entertainment and sports content, every day is a moment to make a difference to partners and to hundreds of millions of people around the world.\n\n\n\nYou will be working on the Developer Experience team under the Consumer Software Engineering organization to build out efficiency tools that bring benefits across developer, QA, product and UX teams. Your contribution will have a broad impact, and it will not only save engineers time and allow them to deliver product features faster, but also help the company save a lot of financial resources. At the same time, you will enjoy fast personal growth while solving exciting and ambitious technical problems!\n\n\n\nYou will have the opportunity to work closely with client developers to build industry leading solutions to enable engineers to remotely access and control streaming devices. You will also build an innovative tool for managing and automating streaming hardware and for creating device labs for Disney engineering teams. Your day to day involves gathering feature requests from users, developing and testing your solutions, and deploying them into production.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBachelor’s degree in Computer Science, Information Systems, Software, Electrical or Electronics Engineering, or comparable field of study, and/or equivalent work experience\n\nThe hiring range for this position in Seattle, WA and NY, NY is $98,000 to $131,300 per year based on a 40 hour work week. The amount of hours scheduled per week may vary based on business needs. The base pay actually offered will take intoaccount internal equity and also may vary depending on the candidate’s geographic region, job-related knowledge, skills, and experience among other factors. A bonusand/or long-term incentive units may be provided as part of the compensation package, in addition to the full range of medical, financial, and/or other benefits,dependent on the level and position offered.",
        "seniority_level": "Mid-Senior level",
        "employment_type": "Full-time",
        "job_function": "Information Technology",
        "industries": "Entertainment Providers"
    },
    {
        "title": "software Engineer",
        "company": "Fieldguide",
        "job_url": "https: //www.linkedin.com/jobs/view/software-engineer-at-fieldguide-3961092714?trk=public_jobs_topcard-title",
        "logo": "https: //media.licdn.com/dms/image/D560BAQGmPH1QqmCzkg/company-logo_100_100/0/1688494223337/fieldguide_inc_logo?e=2147483647&v=beta&t=KqO_GS4C7oH3nVoyvtDCXbbhAvj2OeLbfty0yJYZCbc",
        "date_posted": "2 weeks ago",
        "pay_range": "$125,000.00/yr - $167,000.00/yr",
        "location": "san Francisco, CA",
        "organization": "Fieldguide",
        "number_of_applicants": "Be among the first 25 applicants",
        "job_description": "About Us:\n\nFieldguide is establishing a new state of trust for global commerce and capital markets through automating and streamlining the work of assurance and audit practitioners specifically within cybersecurity, privacy, and ESG (Environmental, Social, Governance). Put simply, we build software for the people who enable trust between businesses.\n\nWe’re based in San Francisco, CA, but built as a remote-first company that enables you to do your best work from anywhere. We\"re backed by top investors including Bessemer Venture Partners, 8VC, Floodgate, Y Combinator, DNX Ventures, Global Founders Capital, Justin Kan, Elad Gil, and more.\n\nWe value diversity — in backgrounds and in experiences. We need people from all backgrounds and walks of life to help build the future of audit and advisory. Fieldguide’s team is inclusive, driven, humble and supportive. We are deliberate and self-reflective about the kind of team and culture that we are building, seeking teammates that are not only strong in their own aptitudes but care deeply about supporting each other\"s growth.\n\nAs an early stage start-up employee, you’ll have the opportunity to build out the future of business trust. We make audit practitioners’ lives easier by eliminating up to 50% of their work and giving them better work-life balance. If you share our values and enthusiasm for building a great culture and product, you will find a home at Fieldguide.\n\n\n\nAs a Software Engineer at Fieldguide, you’ll be an early member of the team, taking a front-row seat as we build both the company and the engineering organization to tackle the massive and archaic audit and advisory industry.\n\n\n\n\n\n\n\n\n\n\n\nFieldguide is a values-based company. Our values are:\n\n\n\n\n\n\n\nCompensation Range: $125K - $167K",
        "seniority_level": "Entry level",
        "employment_type": "Full-time",
        "job_function": "Engineering and Information Technology",
        "industries": "software Development"
    },
    // ... more jobs 
]

If you need the data in formats other than JSON, Apify allows you to export your dataset to various formats, including CSV, Excel, XML, and more.

Wrapping Up

Building your own scraper is a great learning experience, but maintaining it against the constant changes to the LinkedIn website and anti-scraping measures can be a full-time job.

If you need a reliable, maintenance-free solution that handles proxy rotation, scaling, and data parsing for you, try the LinkedIn Job Postings Scraper on Apify today. It is designed to be fast, efficient, and easy to integrate into your existing workflows.