DEV Community: SerpApi.Org

How to scrape google local pack results without getting blocked

SerpApi.Org — Wed, 02 Sep 2026 02:31:31 +0000

Building a robust pipeline to extract business listings from search results requires more than just a basic script. While a simple Python request might work for a few dozen queries, scaling to thousands of requests without hitting CAPTCHAs or 429 status codes requires mastering three core technical pillars: geo-localization, anti-bot mitigation, and resilient parsing.

Mastering Geo-Localization with UULE

To get accurate results for specific regions, you need to master the UULE parameter. This base64-encoded string instructs the search engine to return results for a specific location, overriding your proxy's actual IP location.

The format is w+CAIQICI[Length][Canonical-Name]. The Length character is calculated based on your canonical location string length using a custom ASCII-based index. By constructing these parameters, you can simulate search intent from virtually any neighborhood, which is significantly more efficient than attempting to spoof GPS coordinates via browser-based tools.

Avoiding Automated Traffic Detection

High-frequency scraping will trigger security filters if your fingerprints don't match standard human behavior. Simple user-agent rotation is no longer enough. You must focus on:

JA3 Fingerprinting: Your HTTP/2 TLS handshake must match the specific cipher suites and extensions of a real browser. If your request structure doesn't align with a standard Chrome desktop signature, the server will flag the connection immediately.
Residential Proxy Rotation: Datacenter IPs are easily identified and blocked. Using a pool of residential proxies allows you to rotate your outgoing traffic, mimicking domestic web usage.
Headless Browser Hardening: If you use Puppeteer or Playwright, you must disable the navigator.webdriver flag and hide variables like window.cdc_adoQyv8763_Array to prevent detection. Always use stealth plugins to normalize your environment’s hardware concurrency and canvas rendering.

Resilient Data Extraction

Google frequently updates its front-end code, often using randomized alphanumeric class names (e.g., u4698b). If your scraper relies on these classes, your parser will inevitably break. Instead, build your selectors using:

JSON-LD Payloads: Check for application/ld+json script blocks in the HTML. These contain structured business data (name, ratings, address) in a machine-readable format that rarely changes, even when the UI layout is updated.
Structural XPath: Target semantic HTML paths or immutable attributes like data-cid instead of CSS classes. This creates a "structural anchor" that remains stable even when the page design is modified.

Scalability and Maintenance

When scaling to a high volume of requests, you must shift toward a distributed architecture. Implement a task queue using tools like Redis or RabbitMQ to manage concurrency, and always enforce a 5–15 second buffer per residential IP to avoid rate limits.

While maintaining an in-house infrastructure offers full control, the engineering overhead is significant. You must weigh the "build vs. buy" cost; for many teams, integrating a dedicated SERP API is far more cost-effective than dedicating weeks of developer time to fixing broken selectors, rotating proxies, and debugging TLS fingerprinting logic. Whether you choose to build a custom engine or outsource the parsing, focus on building a system that treats the DOM as a dynamic, volatile structure rather than a static document.

Originally published at How to scrape google local pack results without getting blocked

Free web scraping api options for developers in 2026

SerpApi.Org — Tue, 01 Sep 2026 04:41:16 +0000

Managing scrapers with Puppeteer or Playwright often turns into a nightmare of proxy maintenance and IP rotation, costing you more in engineering hours than a managed service ever would. Over the last decade of building data pipelines, I’ve found that the real "cost" of scraping isn’t the subscription fee—it’s the time spent debugging 403 errors and rotating blacklisted IPs.

The Hidden Costs of Managed Services

When you opt for a service tier, you need to understand the difference between a raw HTTP request and a headless browser render. A standard GET request is cheap, but triggering a Chromium instance to execute JavaScript is essentially renting cloud compute.

Most providers use a credit multiplier for these renders. One page load might consume 20 times the credits of a standard request. If your site relies on SPAs (Single Page Applications), your monthly quota will vanish in minutes if you aren't careful.

Why "No-Card" Tiers Are Essential

I always advise startups to stick to providers that don't require credit card details during signup. This acts as a natural "hard stop." When you hit your 1,000 or 5,000 request limit, the service simply returns an error instead of charging your company card for overages. It keeps your development sandbox risk-free.

Here is how the current landscape looks for developers who want to test without financial commitment:

Provider	Monthly Free Requests	Card Required	Best For
ScraperAPI	1,000	No	General Proxy Rotation
ZenRows	1,000	No	Anti-bot Bypassing
Crawlbase	1,000	No	Static HTML Parsing
SerpApi	100	No	Structured Search Data

Optimizing Your Implementation

The biggest mistake I see junior engineers make is keeping JavaScript rendering enabled by default. Always check the source code (view-source) of your target. If the data you need is present in the raw HTML, turn off the rendering parameter immediately. This simple change can preserve 90% of your monthly credits.

Furthermore, don't ignore the SDK. A service is only as good as its documentation and native support for your stack (Python, Node, Go). If you spend more time writing custom regex to clean messy HTML than you do writing business logic, your scraping tool is failing you.

When to Go Custom vs. Managed

If you have zero budget but unlimited time for maintenance, open-source libraries like Scrapy or Playwright are the way to go. However, for a professional pipeline, the "engineering cycle" cost of building your own proxy rotation suite will quickly dwarf the cost of a mid-tier subscription.

Ultimately, your strategy should be:

Validate: Use a no-card free tier to confirm your logic works.
Optimize: Disable JS rendering wherever possible to stretch your credits.
Scale: Once you hit the limit, evaluate whether the time saved by the API justifies the cost of a professional plan.

Remember, ethical extraction—respecting robots.txt and maintaining reasonable request delays—isn't just about compliance. It’s about being a good internet citizen and keeping your source platforms healthy so we can all continue building data-driven tools.

Originally published at Free web scraping api options for developers in 2026

How to scrape websites without getting blocked in 2026

SerpApi.Org — Mon, 31 Aug 2026 12:29:15 +0000

Dealing with anti-bot defenses has moved far beyond rotating User-Agent strings. If you are still relying on standard datacenter proxies and basic automation scripts, your requests are likely being dropped at the TCP handshake level. Modern security suites like Cloudflare or DataDome don't just check your IP; they profile your entire connection stack.

The Problem: IP Reputation and TLS Fingerprinting

Most datacenter IP ranges are blacklisted by default because they are easily identified as commercial cloud providers (AWS, Google Cloud). When a firewall sees an incoming connection from these ranges, it triggers an immediate reputation penalty.

However, the real "gotcha" is TLS fingerprinting. When your script initializes an HTTPS connection, it sends a Client Hello packet. Security systems generate a JA3 or JA4 hash from your TLS handshake parameters (cipher suites, extensions, etc.). If this hash doesn't match a standard desktop browser (like Chrome or Firefox), the server terminates the connection instantly. You could have the cleanest residential IP in the world, but if your Python script's SSL library is shouting "I am an automated bot," you will get blocked.

Modern Architecture Choices

To build resilient pipelines, you must choose the right tool for your specific volume and target complexity:

For Lightweight Tasks: Ditch traditional Selenium or standard requests. Use libraries like curl_cffi or nodriver. These tools allow you to perform TLS impersonation and communicate via the Chrome DevTools Protocol (CDP) without leaving the obvious traces of legacy headless browsers.
For High-Volume/Protected Sites: Don't reinvent the wheel. Managed scraping APIs act as an abstraction layer, handling proxy rotation, CAPTCHA solving, and TLS spoofing for you. When evaluating these, always calculate the cost per 1,000 successful pages rather than the base credit price. Many providers hide 10x-25x credit multipliers for features like residential proxies or JS rendering, which can bankrupt a project if not calculated upfront.
For AI-Ready Data: Use specialized tools like Firecrawl if you need to convert raw HTML into clean Markdown for RAG (Retrieval-Augmented Generation). These AI-native parsers excel at structuring data but are not replacements for the network-level unblocking needed to reach the page in the first place.

The "Don't Build" Philosophy

My rule of thumb for engineering teams: never build a custom parser for a platform if a specialized API exists.

For example, if you need to extract search engine results, using a dedicated service like SerpApi is almost always cheaper than the "hidden cost" of maintaining your own proxy pool, rotating IPs, and fixing your parser every time Google shifts their DOM structure.

Key Takeaways for 2026 Pipelines

Stop using Datacenter IPs: They are effectively obsolete for protected targets. Shift to residential proxy pools to mimic human ISP connections.
Impersonate, Don't Modify: Focus on protocol-level TLS impersonation. If your tool doesn't support JA4 fingerprint spoofing, it’s a liability.
Audit Credit Multipliers: Before choosing an API, simulate your actual production usage (JS rendering + Residential Proxy) to see the true cost per request.
Adopt Protocol-Level Drivers: Move away from chromedriver and Selenium. Use nodriver or Camoufox to avoid the detectable footprints left by legacy automation.

By offloading the "cat-and-mouse" game of firewall evasion to specialized providers, your team can focus on what actually drives value: the data itself, not the infrastructure used to fetch it.

Originally published at How to scrape websites without getting blocked in 2026

Cheap residential proxy api for scraping: 2026 guide

SerpApi.Org — Sun, 30 Aug 2026 01:28:15 +0000

Dealing with anti-bot systems is the most frustrating part of web scraping. Once a target server flags your datacenter IP ranges, your scripts become useless. While premium enterprise providers are tempting, they often lock developers into expensive long-term contracts. In my experience building scrapers, you don’t need a massive budget to bypass these blocks; you need to understand how to leverage residential IP pools efficiently without overpaying.

Why Residential IPs are Non-Negotiable

Datacenter IPs are cheap, but they are easy to spot. Modern firewalls check the Autonomous System Number (ASN) of your traffic. If your requests originate from an Amazon or DigitalOcean block, you’re instantly flagged as non-human.

Residential proxies, however, route traffic through real household ISPs. Because these IPs represent actual users, anti-bot systems can't block them without causing collateral damage to legitimate visitors. For a successful scrape, your pipeline must mimic human fingerprints, which starts with an IP that has a high reputation score and a legitimate ISP ASN match.

What to Look for in Budget-Friendly Providers

When scouting for an affordable proxy API in 2026, don’t just look at the price per gigabyte. Consider these three metrics:

True Pay-As-You-Go: Avoid providers that require high upfront commitments. You want the flexibility to scale your usage based on your actual extraction volume.
Pool Health: A "100 million IP" network is useless if the IPs are sluggish or frequently drop. Look for services that provide low latency and clear health status codes.
Sticky vs. Rotating Sessions: If you are scraping a simple catalog, per-request rotation is fine. But for flows requiring login or multi-step navigation, you need "sticky" sessions that hold an IP for 10–30 minutes to maintain your session state.

Quick Comparison Table (2026 Estimates)

Provider	Cost/GB (PAYG)	Min. Commitment	Key Strength
Webshare	$1.00 – $1.50	None	Best for ultra-low costs
Smartproxy	$2.20 – $3.00	None	Highly stable, massive pool
Asiaproxy	$1.50 – $2.00	None	Great for regional targeting

Technical Integration Strategy

When using these APIs in Python, you must account for the reality that residential nodes are less stable than commercial servers. Your code should never assume a request will succeed on the first try. Always implement a retry logic with a timeout.

import requests

# Gateway config
PROXY_URL = "http://username:password@gateway.provider.com:port"
proxies = {"http": PROXY_URL, "https": PROXY_URL}

def fetch(url):
    try:
        # Use a 10s timeout to avoid hanging on slow residential nodes
        response = requests.get(url, proxies=proxies, timeout=10)
        response.raise_for_status()
        return response.text
    except Exception as e:
        print(f"Request failed: {e}")
        return None

Avoiding "Hidden" Bandwidth Costs

One rookie mistake is ignoring header overhead. Some providers bill you for every byte of data passing through the socket, including TLS handshakes and headers. If you are only grabbing small snippets of data, this overhead can inflate your bill by 15-20%.

Pro-tip: Always check if your provider bills per successful response only. If a proxy node drops midway or returns a 403, you shouldn't be charged for that wasted bandwidth. Finally, never touch "free" proxy lists. They are almost always honey pots designed to sniff your traffic and hijack your credentials. Stick to verified providers, monitor your usage, and keep your scraper logic simple.

Originally published at Cheap residential proxy api for scraping: 2026 guide

Best free serp scraper chrome extensions for 2026

SerpApi.Org — Sat, 29 Aug 2026 10:39:14 +0000

After spending years refining my data collection workflows, I’ve learned that relying on browser-based tools for SEO is often a game of diminishing returns. While they seem convenient for one-off tasks, their architecture is fundamentally flawed for anything beyond a few manual queries. If you are looking to automate your workflow, here is how to navigate the landscape without burning your IP or wasting hours in Excel.

Vetting Security and Transparency

The biggest mistake I see developers make is installing extensions without auditing the manifest.json. If a tool requests read access to your entire browsing history or sensitive account data, uninstall it immediately. My golden rule: if it isn’t open-source, treat it as a black box that might be harvesting your data. Always check the Chrome Web Store for recent updates; if an extension hasn’t been patched in six months, it’s likely to break the moment Google tweaks its DOM structure.

The IP Reputation Trap

Most free tools suffer from a fatal flaw: they perform requests from your local machine. Because you lack proxy rotation, Google flags your home or office IP address almost instantly. Once you hit the threshold of roughly 20–50 requests, expect a wall of CAPTCHAs.

Local Execution: Your IP is your identity. Once flagged, your productivity drops to zero.
Tab Lifecycle: Most of these tools require an active tab, killing any chance at background automation.
DOM Instability: Without active maintenance, these tools return null values as soon as the SERP layout shifts.

Export Quality and Data Hygiene

A tool is only as good as the file it outputs. Top-tier extensions provide clean, CSV-ready exports with consistent columns. Lower-end tools often dump raw HTML or nested JSON, turning a 5-minute task into an hour of data cleansing. Before committing to a plugin, perform a test export. If you find yourself re-mapping columns manually, you are doing it wrong—at that point, it is more efficient to pull data directly via the Google Search Console API.

When to Pivot to APIs

The "extension ceiling" is real. Once you need to track more than 50 keywords daily, manual extraction becomes a bottleneck. APIs, such as SerpApi, abstract away the headaches of proxy management and CAPTCHA solving. You aren't just paying for the data; you are paying for the reliability of a managed infrastructure.

If you have the dev bandwidth, self-hosted solutions like SerpBear are an excellent middle ground. They give you total control over your own proxies and scraping logic without the recurring cost of an enterprise API. However, be prepared to maintain your own code; if Google updates its structure, your server-side script will stop working until you fix it.

Ethical Extraction

Regardless of the method, stay within the lines:

Public Data Only: Focus on rank positions, meta descriptions, and snippets. Never target private profiles or behind-login content.
Rate Limiting: Even if you use a script, be a good netizen. Excessive requests can be seen as a minor DDoS attempt, which is both unethical and a quick way to get your proxy pool burned.

If your growth depends on consistent, high-quality data, save yourself the frustration of manual browser-based tools. Move to an API-first approach or a well-maintained, self-hosted instance to ensure your data pipeline stays stable as your SEO projects scale.

Originally published at Best free serp scraper chrome extensions for 2026

How to export Google Search Console API data to Google Sheets

SerpApi.Org — Fri, 28 Aug 2026 07:13:15 +0000

There have been countless times I've found myself stuck on a Friday afternoon, painstakingly copying data from Google Search Console in 1,000-row increments. It was always frustrating to realize that crucial performance trends, especially for those long-tail keywords, were hidden in the data I couldn't access due to these limitations. The standard export is simply not built for comprehensive analysis.

This is why I started looking for a way to get more data. By connecting directly to the Google Search Console API, we can overcome these restrictions and pull significantly larger data sets. This approach has been a game-changer for my workflow, allowing me to automate data collection and gain deeper insights without manual intervention.

Unlocking More Data

The 1,000-row limit in the standard export interface is a major bottleneck. It forces teams to work with incomplete datasets, leading to potential blind spots in understanding site performance. The Search Console API, however, allows programmatic requests for up to 25,000 rows in a single call. This unlocks access to a much richer set of site performance metrics that are simply unavailable through the dashboard.

For sites with thousands of landing pages, relying solely on UI exports creates significant gaps in tracking content performance. This often leads to inaccurate SEO audits because the long-tail data is missing. The API acts as a powerful, free alternative to paid rank tracking tools, providing actual click and impression data rather than relying on estimations.

Feature	Standard UI Export	Search Console API
Row limit/request	1,000 rows	25,000 rows
Automation level	Manual	Programmable
Data access	Sampled/Limited	Raw/Comprehensive

Initial Setup: Google Cloud Project

To begin, you'll need a Google Cloud project. This is where you'll enable the Search Console API and set up the necessary credentials for your scripts.

Create a Project: Go to the Google Cloud Console and select 'Create Project'.
Enable API: Search for 'Google Search Console API' in the API Library and enable it.
OAuth Consent Screen: Configure the consent screen with your email and an application name.
Create Credentials: Choose either 'OAuth client ID' or a 'Service Account' for authentication. For automated scripts, a service account is generally preferred.

A common stumbling block here is the complexity of OAuth flows. For this specific use case, you primarily need read-only access to the Search Console property you want to query. If the Cloud Console interface feels overwhelming initially, simply enabling the API is a good first step. You can refine scopes later as you become more familiar with Google's API ecosystem.

Scripting the Connection: Google Apps Script

The core of this process involves using Google Apps Script to interact with the Search Console API. This script will query the searchAnalytics.query method to pull data directly into your Google Sheet.

Open Apps Script: In a new Google Sheet, go to 'Extensions' > 'Apps Script'.
Add API Service: Paste the GSC API service library code into the editor.
Define Parameters: Set your startDate and endDate within the main function.
Set Row Limit: Crucially, set the rowLimit parameter to 25,000.
Run and Authorize: Execute the function and authorize the script to access your Search Console data.

Ensure your script has the correct authentication scopes enabled, specifically 'readonly' access for the Search Console API. Debugging often involves verifying that your JSON request structure aligns with the API's current requirements, as these can be updated by Google.

Handling Script Timeouts

Google Apps Script has a six-minute execution limit per trigger. For large datasets, this can cause scripts to fail. The solution is to implement pagination.

Instead of requesting all 25,000 rows at once, break the request into smaller chunks. The startRow parameter is key here. By incrementing startRow in a loop, you can fetch data in manageable segments (e.g., 5,000 rows per request) that complete within the time limit.

// Example of pagination logic
var startRow = 0;
var rowLimit = 5000; // Smaller chunks for pagination
var totalRowsRequested = 0;
var maxRows = 25000; // Your desired total

while (totalRowsRequested < maxRows) {
  var query = {
    'startDate': 'YYYY-MM-DD',
    'endDate': 'YYYY-MM-DD',
    'dimensions': 'date,query',
    'startRow': startRow,
    'rowLimit': rowLimit
  };
  // ... make API call with query ...
  // ... process results ...
  totalRowsRequested += results.rows.length;
  startRow += rowLimit;
}

This modular approach ensures script stability, even with API latency. I usually add logging to track progress and identify which specific page might fail if a network issue occurs.

Automating Data Refreshes

To keep your reports up-to-date, set up time-driven triggers in Apps Script.

Open Triggers: Navigate to the 'Triggers' tab in the Apps Script dashboard.
Add Trigger: Click '+ Add Trigger', select your main function, and choose 'Time-driven' as the event source.
Set Schedule: Select your preferred interval (e.g., 'Week timer' or 'Month timer').

Automating this process removes the manual labor from reporting, ensuring your team always has current data for meetings and analysis.

When Sheets Isn't Enough

While Google Sheets is excellent for many use cases, it can become slow with hundreds of thousands of rows. For very large historical datasets, consider migrating to BigQuery. It offers superior performance for querying massive amounts of data using SQL.

Metric	Google Sheets	BigQuery
Storage limit	~10 million cells	Petabytes
Calculation speed	Slow with large sets	High-speed SQL
Complexity	Low (Visual)	High (SQL)

If your sheet starts lagging significantly (often around the 50,000-row mark), it's a signal that your data storage layer needs an upgrade. BigQuery enables much more sophisticated time-series analysis for sites generating millions of impressions.

By leveraging the GSC API and Apps Script, you can bypass the export limitations, automate your data collection, and gain a comprehensive view of your search performance. This is essential for serious SEO work in today's landscape. If you're ready to move beyond manual exports, start by setting up your Google Cloud project and experimenting with these scripts.

Originally published at How to export Google Search Console API data to Google Sheets

How to scrape google maps reviews api without getting banned

SerpApi.Org — Thu, 27 Aug 2026 11:11:14 +0000

Many developers eventually hit a wall when they realize the official Google Places API is designed for UI display, not data science. Limiting output to the five most relevant reviews makes it fundamentally useless for sentiment analysis or large-scale historical tracking. If you’ve spent hours writing scripts only to have them break after a minor UI update, you know exactly what I mean by the "maintenance tax."

Stop Parsing HTML

If you are still using tools like Selenium or Playwright to target CSS classes, your project is a maintenance nightmare. Google’s front-end is constantly A/B tested, meaning your brittle selectors will fail sooner or later.

Instead, shift your strategy to intercepting the internal API calls. By inspecting the Network tab in your browser and filtering by XHR requests, you can often find structured data being transmitted via Protobuf. This is essentially serialized JSON—it is significantly more stable, requires less CPU than a headless browser, and doesn’t break every time a pixel moves on the page.

The Proxy and Fingerprint Reality

Even with stable endpoints, you cannot ignore the infrastructure required to bypass anti-bot triggers. Google tracks request velocity and browser fingerprints with high precision. If you’re using datacenter proxies, you’ll be flagged almost instantly.

To stay under the radar:

Use Residential Proxies: These route traffic through actual home connections, making your requests indistinguishable from a standard user.
Mimic Human Behavior: Never hammer the server. Implement intelligent delays and ensure your request headers are randomized but consistent with a real device fingerprint.
Error Handling: Expect to be challenged. Your pipeline must include robust retries and a back-off strategy that honors 429 status codes.

Build vs. Buy: The Hidden Cost

If your project requires collecting thousands of reviews regularly, the "DIY" route often becomes more expensive than a managed solution. Between maintaining proxy pools, handling CAPTCHA solvers, and fixing broken parsers, you are effectively paying your engineers a premium to do infrastructure work rather than product development.

If you don't have a dedicated DevOps team, relying on specialized third-party scraping APIs is usually the most efficient path. They absorb the cost of maintenance, keeping your pipeline green while you focus on the actual business intelligence.

Data Integrity at Scale

When you start pulling large volumes of data, your database needs to be ready. I always recommend hashing the review ID or a combination of the author ID and timestamp before insertion.

Use a NoSQL approach to store the entire JSON object from the Protobuf response. This keeps your data hierarchical and avoids the performance bottlenecks of complex joins in relational databases. Always perform an "upsert" based on the hash to keep your dataset clean and prevent duplicates as you scrape historical data.

Ethical Considerations

Remember that while public data is technically accessible, scraping still often skirts the edge of terms of service. Always respect robots.txt where possible, avoid high-frequency spikes that degrade user experience, and consult your legal team regarding your specific use case.

Moving away from fragile headless browsers toward structured endpoint extraction is the only way to build a pipeline that survives long-term. Focus on the analysis, not the maintenance.

Originally published at How to scrape google maps reviews api without getting banned

How to scrape google search results using python with stealth

SerpApi.Org — Wed, 26 Aug 2026 06:03:13 +0000

Many developers rely on basic setups with BeautifulSoup and random User-Agents, assuming this is enough to navigate modern search engine protections. After a decade in the industry, I have learned that this approach is rarely sufficient. Between TLS fingerprinting and HTTP/2 handshakes, basic scripts are flagged almost instantly.

Here is a breakdown of how to build a robust, production-grade search data pipeline.

The Modern Tech Stack

Stop using requests and BeautifulSoup. They are too slow and easily detected.

Use httpx for HTTP/2: Modern security engines analyze your TLS handshake. httpx supports HTTP/2, which allows you to multiplex requests and match the connection signature of a legitimate browser, reducing the likelihood of being blocked.
Switch to XPath: CSS selectors are fragile because Google frequently updates class names during A/B tests. parsel with XPath targeting structural elements—like //h3 or specific semantic containers—is much more resilient than brittle CSS selectors.
Asynchronous Processing: Use asyncio to fetch results in parallel. This significantly increases throughput while keeping your footprint low.

Handling Detection and Evasion

The most common reason for failure is a mismatch between your headers and your TLS signature.

TLS Fingerprinting: Security systems look for a "JA3 signature." If you declare a Chrome User-Agent but the handshake indicates a Python urllib client, you will get hit with a CAPTCHA. Ensure your client negotiates ciphers that align with the browser version you are spoofing.
Residential Proxies: Datacenter IPs are often blacklisted entirely. Use a rotating pool of residential proxies. These IPs route your traffic through legitimate consumer ISP nodes, making your requests indistinguishable from real users.

Managing Scale

If your project requires high-volume data (e.g., thousands of keywords per day), maintaining your own infrastructure becomes a burden. You will spend more time fixing broken parsers and managing proxy bans than actually analyzing data.

Task	Self-Built Approach	Managed Service (e.g., SerpApi)
Maintenance	Constant (fix broken XPaths)	Zero
CAPTHCAs	You must solve them	Handled automatically
Output	Raw HTML (requires parsing)	Clean, structured JSON
Cost	Proxy fees + DevOps hours	Predictable usage-based pricing

When to Use Headless Browsers

Playwright with the stealth plugin is powerful, but use it sparingly. It is necessary for capturing dynamic content like Maps or "People Also Ask" sections that rely on JavaScript execution. However, it is resource-intensive, consuming roughly 150MB of RAM per instance. Use standard httpx requests for static SERPs and save Playwright for complex, interactive components.

Final Takeaway

For small-scale experiments, a custom httpx + parsel setup is excellent. However, as your volume grows, the overhead of proxy rotation and infrastructure maintenance becomes a liability. If you find yourself spending more than a few hours a week patching your scraper, it is time to shift to a managed API. This lets you focus on building features rather than fighting search engine security.

Originally published at How to scrape google search results using python with stealth

How to get a google news api key free

SerpApi.Org — Tue, 25 Aug 2026 11:51:20 +0000

If you've ever tried to integrate live news streams into your application, you've likely hit a wall: the official Google News API was deprecated back in 2011. During my early days building news aggregators, I wasted hours trying to authenticate against dead legacy endpoints.

Fortunately, you don't need an official first-party key to fetch real-time search trends. Here are the three most reliable, free workarounds I use in production today.

1. Programmatic RSS Parsing (No Auth)

The fastest way to get started without handling API keys is querying Google’s public RSS endpoints. By targeting news.google.com/rss with specific URL-encoded parameters, you can stream real-time headlines directly.

Here is a quick Python setup using feedparser:

import feedparser
import urllib.parse

query = urllib.parse.quote_plus("artificial intelligence")
# hl: language, gl: region, q: search query
rss_url = f"https://news.google.com/rss/search?q={query}&hl=en-US&gl=US&ceid=US:en"

feed = feedparser.parse(rss_url)
for entry in feed.entries[:5]:
    print(f"Title: {entry.title}")
    print(f"Link: {entry.link}\n")

Pros: Unlimited queries, zero authentication.
Cons: No rich metadata (e.g., full-text body or lead images), and Google occasionally shifts XML schema structures, which can break manual parsers.

2. Google Custom Search Engine (CSE) with a Domain Filter

If you need native JSON but want to stay within the Google ecosystem, you can use the Custom Search JSON API. Google provides 100 free queries per day.

To make this act like a dedicated news engine:

Create a Programmable Search Engine in your Google Cloud Console.
Restrict the search scope to authoritative news domains (e.g., reuters.com, apnews.com).
Generate a free API key from the Google developer credentials dashboard.

While this approach works great for hobby projects, that 100-query daily limit is a hard ceiling that you'll quickly hit in production.

3. Managed SERP APIs (e.g., SerpApi)

When I build micro-SaaS MVPs that require structural stability without the headache of rotating proxies or managing broken HTML scrapers, I offload the parsing to third-party SERP providers.

For instance, SerpApi offers a free tier of 100 searches per month with no credit card required. It returns clean, structured JSON:

{
  "news_results": [
    {
      "title": "Tech giants announce new open-source models...",
      "link": "https://example-news-outlet.com/article",
      "source": "Tech News",
      "date": "2 hours ago"
    }
  ]
}

This completely decouples your application code from Google's front-end layout changes.

Pro-Tip: Optimize with Time Filters

To maximize efficiency and keep your data payloads lightweight under free limits, utilize Google's time-filtering tbs parameters in your queries:

tbs=qdr:h (Past hour)
tbs=qdr:d (Past 24 hours)
tbs=qdr:w (Past week)

By targeting narrow time windows, you avoid processing duplicate historical data and keep your pipelines highly optimized.

Quick Comparison

Method	Free Limit	Auth Required	Output Format	Best For
Google RSS	High (soft IP limits)	No	XML	Rapid prototyping, personal scripts
Google CSE	100 queries/day	Yes	JSON	Low-volume, domain-restricted queries
SerpApi	100 queries/month	Yes	JSON	Production MVPs, zero-maintenance pipelines

Originally published at How to get a google news api key free

How to use Google Search API for free without billing fears

SerpApi.Org — Mon, 24 Aug 2026 04:14:17 +0000

Many developers steer clear of Google Cloud simply because they fear an accidental credit card charge. However, when it comes to the Custom Search JSON API, the fear is largely misplaced. You can integrate search data into your projects without ever touching the billing section, thanks to a hard-coded limit that stops the service once your quota is exhausted.

Why the risk is non-existent

The service provides a permanent, daily allowance of 100 queries. This is not a trial; it is a fixed tier that resets every day at midnight Pacific Standard Time (PST). If your application hits request number 101, the API returns a 403 Forbidden error. Because the service effectively shuts down, there is no mechanism to trigger a payment. Simply put: if you don’t attach a credit card, you cannot be billed.

Getting started safely

To configure this, follow these three essential steps:

Create a Programmable Search Engine: Visit the dashboard and toggle "Search the entire web" to ON. This generates your CX identifier. Skipping this step often leads to empty JSON responses, which wastes your precious daily quota.
Generate a Restricted API Key: Within the Google Cloud Console, enable the Custom Search API. Crucial: Apply HTTP referrers or IP restrictions immediately. Without these, a leaked key allows others to exhaust your daily quota in seconds.
Secure your credentials: Never commit keys to version control. Use .env files or secret management tools to ensure your tokens remain local.

Implementation best practices

When parsing results in Python or your language of choice, always treat the incoming JSON as untrusted input. If a query yields zero results, the items key might be missing. Always include a safety check before iterating through the result set:

# Example of a safe check
response = requests.get(url, params=params)
data = response.json()

if 'items' in data:
    for item in data['items']:
        print(item['link'])
else:
    print("No results found or quota exceeded.")

Pro-tips for stable integrations

Handle 429 Errors: Implement exponential backoff in your code. This prevents your app from hammering the API if you encounter temporary rate limits.
Design for Failure: Since the limit is rigid, your UI should gracefully handle "quota exceeded" states. Don't assume you will always have access to 100 results.
Monitor Usage: Keep an eye on the Google Cloud Dashboard. If you find yourself hitting the 100-query mark daily, consider setting up automated alerts to notify you when you reach 80% usage.

Scaling beyond the limit

When your project outgrows the 100-query limit, you have two primary paths. If you want to maintain control and avoid costs, self-hosting a SearxNG instance offers unlimited, private search results, though it requires maintaining your own server infrastructure.

Alternatively, if you need structured data at scale without the headache of managing proxies or search engine responses, dedicated providers like SerpApi offer robust developer tiers. They handle the infrastructure, letting you focus on the logic of your application rather than troubleshooting request overhead.

By keeping your project separate from the billing portal and applying strict API restrictions, you can leverage Google’s search index as a powerful tool in your tech stack without any financial anxiety.

Originally published at How to use Google Search API for free without billing fears

Free news api key no credit card required: 2026 developer list

SerpApi.Org — Sun, 23 Aug 2026 03:26:15 +0000

As someone who has spent years building data-driven side projects, I have learned one hard truth: adding a payment method just to test an endpoint is a developer’s nightmare. You need reliable pipelines, not a ticking clock toward an unexpected monthly charge.

Finding data sources that respect a "zero-friction" workflow is essential for rapid prototyping. Here is my breakdown of reliable options that provide access without demanding credit card details at signup.

Top Picks for Frictionless Integration

These providers are my go-to choices because they prioritize accessibility, requiring only an email verification to get started.

API	Daily Limit	Best Use Case
NewsAPI	100 requests	Rapid prototyping and quick tests.
GNews	100 requests	Learning and parsing JSON structures.
MediaStack	100 requests	Standard, reliable headline feeds.

My Workflow for Evaluating "Free" Tiers

I have been burned before, so I follow a strict checklist to avoid "predatory" trials.

The Billing Trap: If a signup form asks for a credit card or billing address, it is a trial, not a permanent free tier. Avoid these if you want to prevent surprise charges.
Dashboard Audits: Even with no card attached, I check my account settings quarterly. Business models change; verify that your plan hasn't been migrated to a time-limited status.
Handling 429 Errors: Most free tiers cap you at 100–500 requests per day. When you hit the limit, you will receive a 429 Too Many Requests status code. My advice? Implement exponential backoff in your client-side code. It keeps your app from crashing and makes you a "good citizen" of their infrastructure.
Monitor Headers: Keep an eye on the X-RateLimit-Remaining header in the API response. You can use this to programmatically disable UI components before you hit your limit.

Security First: Protect Your Keys

Never hardcode keys into your repository. I use .env files locally and environment variables on my servers.

Frontend warning: If you are building a SPA (Single Page Application), never expose your key in the browser. Always route your calls through a lightweight backend proxy. If your key leaks, the provider might flag your account, and you will lose access.

When Should You Pivot?

These options are perfect for students or hobbyists, but they have ceilings. If you expect more than 3,000 calls per month, require historical data, or need high-concurrency for a production app, you will eventually outgrow these tiers.

In those cases, I recommend building your architecture in a modular way. Keep your API-calling logic separate from your business logic so that you can swap providers without refactoring your entire codebase.

If you are looking for a more robust, scalable solution for your production-grade data needs, I suggest looking into SerpApi. It is built with a developer-first mindset and handles the complexities of data collection, allowing you to focus on building features rather than managing rate limits or infrastructure.

Originally published at Free news api key no credit card required: 2026 developer list

Scrape google search results python github: Resilient 2026 guide

SerpApi.Org — Sat, 22 Aug 2026 07:50:38 +0000

Having spent years maintaining data extraction pipelines, I’ve learned a hard truth: if your web scraper relies on class names to parse search engine results, it is already broken. Google dynamically updates and randomizes its HTML layout selectors. If you clone an open-source Python scraper from GitHub, chances are it will throw null-pointer errors within weeks of deployment.

To build a truly resilient pipeline, you need to shift from scraping visual layers to targeting underlying structured data.

Bypassing CSS with JSON-LD Extraction

Instead of targeting fragile CSS selectors, target the raw structured data embedded directly within the page source. Google relies on structured JSON-LD schemas to populate its rich snippets and knowledge graphs. This metadata resides inside script tags of type application/ld+json.

Because this data feeds search engine engines, the keys stay consistent even when CSS classes change daily. Here is how I structure selector-free extraction in Python:

import json
from bs4 import BeautifulSoup

def extract_search_metadata(html_content):
    soup = BeautifulSoup(html_content, 'html.parser')
    scripts = soup.find_all('script', type='application/ld+json')
    extracted_data = []

    for script in scripts:
        try:
            data = json.loads(script.string)
            # Traverse the schema dictionary to find search result payloads
            if "itemListElement" in data:
                extracted_data.append(data["itemListElement"])
        except (ValueError, TypeError):
            continue

    return extracted_data

This approach shifts your code's maintenance cycle from weekly emergency hotfixes to simple, bi-annual structure audits.

Simulating Human Behavior with Playwright

When dynamic search elements (like interactive maps or local listings) are required, static HTML parses are insufficient. I use Playwright to run headless browser sessions. However, modern anti-bot systems check for automated signatures.

To execute successfully:

Remove Automation Indicators: Disable the navigator.webdriver flag inside your browser context.
Implement Micro-Scrolls: Simulate human reading patterns by scrolling the page down in small, randomized pixel increments.
De-synchronize Events: Introduce randomized delays between 800ms and 2400ms before clicks or viewport changes.

Overcoming IP Bans and CAPTCHAs

Running automation from standard hosting servers or GitHub Actions default runner IPs will trigger immediate blockages. Datacenter IP ranges are heavily monitored.

Utilize Rotating Residential Proxies: These route requests through real consumer ISPs, maintaining a 98%+ success rate.
Rotate User-Agents Dynamically: Match your rotated IPs with updated browser headers representing modern, active operating systems.
Build Exponential Backoff: If your pipeline encounters a 429 rate limit, dynamically double the execution delay before retrying with a fresh IP address.

Build vs. Buy: The Technical Trade-off

Building custom scraping infrastructure is a rewarding challenge, but scale changes the financial equation. When you calculate developer wages spent on maintaining code, premium residential proxy bandwidth, and running dynamic headless browsers, self-hosting can become incredibly expensive.

For mission-critical production data, leveraging a managed JSON endpoint like SerpApi is often the more efficient engineering decision. It completely abstracts browser automation, IP pool rotation, and DOM changes, allowing you to focus on processing data rather than acquiring it.

Originally published at Scrape google search results python github: Resilient 2026 guide