DEV Community

Cover image for How to Scrape Bing and DuckDuckGo Results with Python and Apify
ReventaData
ReventaData

Posted on Originally published at apify.com

How to Scrape Bing and DuckDuckGo Results with Python and Apify

#webscraping #seo #python #apify

By the end of this tutorial, you will have a short Python script that sends one search to Bing and DuckDuckGo and gets back one row per organic result: engine, rank, title, the real destination URL and the snippet. You will also compare the two engines' rankings side by side.

Both engines wrap result links in tracking redirects, and the DuckDuckGo page in the capture carried an ad in the result list. The Apify Actor Non-Google Search Engines Scraper (reventadata/serp-multi-scraper) drops the ads it recognises and decodes the redirects, so every url points at the page itself. It also reads Ecosia and Brave Search.

The sample rows in this guide are a sample from a capture dated 2026-10-12 (the search python programming on each engine), parsed with the Actor's own parsers.

⏱️ Time estimate: one sitting: install one package, set one token, run four short Python blocks.

Prerequisites

To follow this tutorial, you will need:

  1. Python 3.11+ (apify-client 3.3.0 requires it).
  2. The Apify API client for Python.
  3. An Apify account and its API token, which sits on the API & Integrations page in Apify Console.

Install the client:

pip install "apify-client~=3.0"
Enter fullscreen mode Exit fullscreen mode

The code reads the token from an environment variable:

export APIFY_API_TOKEN="your_apify_token_here"
Enter fullscreen mode Exit fullscreen mode

What you'll achieve

You will turn a list of search queries into one row per organic result, per engine. Each row carries six fields: query, engine (duckduckgo, bing, ecosia or brave), position (1-based rank on that engine), title, url and snippet.

The same URL found by two engines gives two rows, one per engine, so you can line their ranks up.

Note: Per its README, the Actor is not affiliated with Microsoft Bing, DuckDuckGo, Ecosia or Brave Software.

Example use case: Suppose you track where your docs rank for python programming outside Google. Run the query on DuckDuckGo and Bing, then compare position for your domain across engines. In the 2026-10-12 captures, 8 of the 10 URLs appeared on both engines' first page.

Step 1: Access Non-Google Search Engines Scraper

Go to the Non-Google Search Engines Scraper page on Apify Store. The README lists every input and output field. In this tutorial we drive the Actor from Python, so everything after this step happens in code.

Step 2: Configure the search

The main input is queries, a list of search terms; it is required. engines picks which engines run each query: any of duckduckgo, bing, ecosia and brave. Leave it out and you get DuckDuckGo and Bing. maxItemsPerQuery (1 to 100, default 10) caps the rows kept per engine for each query.

Here is the setup. It sends python programming to DuckDuckGo and Bing and keeps up to 10 results from each:

import os
from apify_client import ApifyClient

# 1. Initialize ApifyClient with the token from the environment
client = ApifyClient(os.environ["APIFY_API_TOKEN"])

# 2. Define the search: one query, two engines, up to 10 results each
run_input = {
    "queries": ["python programming"],
    "engines": ["duckduckgo", "bing"],
    "maxItemsPerQuery": 10,
}
Enter fullscreen mode Exit fullscreen mode

💡 Tip: the Actor reads the first results page of each engine, then cuts it to maxItemsPerQuery. Each 2026-10-12 capture held 10 organic results, so a cap above what one page holds returns the page.

Step 3: Optional settings

Parameter Type What it does
country string Two-letter code, default us. Sent as DuckDuckGo's kl, Bing's cc, Ecosia's region and Brave's country parameter.
braveApiKey secret string A key from api.search.brave.com. With it, Brave results come from Brave's JSON API instead of the search page.
proxyConfiguration object Off by default. When set, every engine in the run goes through the same proxy URL.

Three combinations that work:

  • Default pair: queries only. DuckDuckGo and Bing, no proxy.
  • All four engines: engines: ["duckduckgo", "bing", "ecosia", "brave"].
  • Brave through its API: engines: ["brave"] plus your braveApiKey.

⚠️ An engine name outside the four is refused before the run starts. A queries list of only blank strings stops the run with an error at its start.

What a bare two-letter country changes on each engine was not measured for this guide; compare a run with and without it before relying on it.

Step 4: Run the Actor

Now start the run with client.actor("reventadata/serp-multi-scraper").call(run_input=run_input). The call starts the Actor on Apify and waits for it to finish before returning the run:

# 3. Start the Actor run and wait for it to finish
print("Starting search extraction...")
run = client.actor("reventadata/serp-multi-scraper").call(run_input=run_input)
if run is None or run.status != "SUCCEEDED":
    status = run.status if run else "unknown"
    raise SystemExit(f"Run did not succeed (status: {status}).")

# 4. Fetch the saved search results
items = client.dataset(run.default_dataset_id).list_items().items
print(f"Retrieved {len(items)} results.\n")
Enter fullscreen mode Exit fullscreen mode

Behind the scenes:

  • For each query, the Actor fetches each engine's results page with plain HTTP requests, no browser.
  • It drops ad links, decodes DuckDuckGo's uddg= and Bing's &u=a1... redirects, and numbers the remaining results from 1.
  • A challenge page on one engine is logged and skipped; the run moves on to the next engine.

Step 5: View and download results

Print each row:

for item in items:
    print(f"[{item['engine']}] #{item['position']} {item['title']}")
    print(f"  {item['url']}")
Enter fullscreen mode Exit fullscreen mode

Here are two rows, a sample from a capture dated 2026-10-12, parsed with the Actor's own parsers:

[
    {
        "query": "python programming",
        "engine": "duckduckgo",
        "position": 1,
        "title": "Welcome to Python.org",
        "url": "https://www.python.org/",
        "snippet": "Python is a versatile and easy-to-learn programming language that lets you work quickly and integrate systems more effectively. Learn Python basics, download the latest version, access documentation, find jobs, events, success stories and more on the official website."
    },
    {
        "query": "python programming",
        "engine": "bing",
        "position": 1,
        "title": "Welcome to Python.org",
        "url": "https://www.python.org/",
        "snippet": "Experienced programmers in any other language can pick up Python very quickly, and beginners find the clean syntax and …"
    }
]
Enter fullscreen mode Exit fullscreen mode

Now line the engines up by URL:

from collections import defaultdict

# url -> {engine: position}
ranks = defaultdict(dict)
for item in items:
    ranks[item["url"]][item["engine"]] = item["position"]

print("DDG  Bing  URL")
for url, pos in sorted(ranks.items(), key=lambda kv: min(kv[1].values())):
    print(f"{pos.get('duckduckgo', '-'):>3}  {pos.get('bing', '-'):>4}  {url}")
Enter fullscreen mode Exit fullscreen mode

On the 20 captured rows, it prints:

DDG  Bing  URL
  1     1  https://www.python.org/
  2     2  https://www.w3schools.com/python/
  3     3  https://www.programiz.com/python-programming/online-compiler/
  4     4  https://www.python.org/about/gettingstarted/
  5     5  https://www.programiz.com/python-programming/guide
  6     7  https://docs.python.org/3/tutorial/index.html
  7     6  https://en.wikipedia.org/wiki/Python_(programming_language)
  8     9  https://www.geeksforgeeks.org/python/python-programming-language-tutorial/
  -     8  https://www.w3schools.com/python/python_intro.asp
  9     -  https://wiki.python.org/moin/BeginnersGuide
 10     -  https://docs.python.org/
  -    10  https://realpython.com/
Enter fullscreen mode Exit fullscreen mode

Apify datasets export as JSON, CSV, XML, Excel, HTML table, RSS or JSONL.

Understanding your results

The fields:

  • engine and position: the rank on that engine, counted after ads are dropped. In the DuckDuckGo capture, the page held 12 result links; a Udemy ad and a "more info" link about DuckDuckGo's ads were dropped, and the 10 left were numbered 1 to 10.
  • url: the destination, not the tracker. All 10 Bing result links in the capture were bing.com/ck/a redirects; each row carries the decoded address, such as https://www.python.org/.
  • title: the result's headline as the engine shows it.
  • snippet: the engine's own summary, so it differs per engine. In the captures, DuckDuckGo snippets ran 145 to 287 characters; Bing's ran 120 to 133 and all 10 ended in ….
  • query: the search that produced the row, useful when one run holds several queries.

Rows are saved per query, then per engine, in the order you gave. Ecosia and Brave URLs are kept as the page gives them.

Pro tips

  1. Compare, don't merge. The Actor keeps one row per engine for a shared URL. Group by url as in Step 5 to see where engines agree: in the captures, ranks 1 to 5 were the same on both, and they split from rank 6.
  2. Batch queries in one run. Put several terms in queries; each runs on every selected engine, and query tells the rows apart.
  3. Keep the cap small while testing. maxItemsPerQuery: 5 is the Actor's own prefill. Raise it once your code handles the fields.
  4. Use a Brave API key for Brave. Without one, the Actor reads Brave's search page. The Actor's recon notes say that page "requires residential proxy / full browser headers when rate-limited".
  5. Watch Ecosia's transport. The same notes describe Ecosia as "Cloudflare protected on datacenter IPs". The same line names residential proxies as a way in; proxyConfiguration then applies to every engine in the run.

Troubleshooting

  • Run FAILED with zero rows: the run ends FAILED when it saves nothing and at least one engine was blocked or returned no results. It ends SUCCEEDED with an empty dataset and a status message only if every engine returned results and the filters dropped all of them.
  • One engine missing from the rows: that engine failed or returned nothing; the run kept going with the others. The summary line, Done. Total search results pushed: N. Rejections: {...}, names it as engine_failure_<engine> or zero_results_<engine>. From Python: client.log(run.id).get().
  • Input refused before the run starts: an engine name outside duckduckgo, bing, ecosia and brave, or no queries.
  • Fewer results than maxItemsPerQuery: only the first results page is read.

For help, use the Actor's Store page.

Pricing and next steps

For current pricing and details, see the Store page.

To track rankings over time, save the input as a task and add an Apify schedule, which runs an Actor or task on a cron expression (0 8 * * * is every day at 8 AM). To react when a run finishes, add a webhook on ACTOR.RUN.SUCCEEDED; its payload carries the run object, including defaultDatasetId.

Which queries are you planning to track outside Google? Drop a comment! 💬

Top comments (0)