Scale Quora extraction with Python and Apify

Building a reliable web scraper for Quora is a deceptive task. At first glance, it looks like any other modern Q&A site. However, once you start scraping at scale, you run into heavy client-side rendering, aggressive rate limiting, dynamic infinite scrolls, and hidden AI-generated responses.

When your team needs Quora data, you face a classic engineering choice: do you spend a sprint building a custom Playwright or requests script, or do you pay for a pre-built managed service like the quora-scraper Actor?

This article compares the DIY approach against the managed Actor. We will evaluate both options using concrete platform constraints, input schemas, and cost structures so you can make an informed architectural decision. Checked against the Actor's input schema and Apify docs on 2026-09-10.

When should you write a custom Playwright scraper instead?

If you only need to scrape fewer than 100 public URLs once a week and already have an existing scraping infrastructure, write your own script. Building a lightweight Playwright or requests script is highly cost-effective for small, static batches where raw execution speed and infrastructure overhead do not matter.

Quora relies heavily on client-side rendering. If you use Python's requests library, you will miss the bulk of the content because the answers, comments, and stats are hydrated dynamically via JavaScript. This forces you to use a browser automation library like Playwright or Selenium, which introduces high memory and CPU overhead. If your team is already maintaining a cluster of browser instances and has a pool of residential proxies ready, writing 150 lines of Playwright code to extract these elements is a viable option.

However, writing a custom script becomes a liability when you need to handle continuous pagination, search query expansion, or proxy rotation. Quora actively blocks data center IP addresses. If you do not want to manage proxy session stickiness, scroll-bound lazy loading, and dynamic class name changes, a managed tool is the better alternative.

How does the Apify input schema control the scraper?

The quora-scraper behavior is governed by a strict JSON schema. Unlike a custom script where you must parse command-line arguments and handle default state fallbacks manually, this schema defines exactly how the scraper searches, navigates, and targets profiles.

At least one of searchQueries or directUrls must be provided. If you pass searchQueries, the scraper finds relevant Quora question URLs using DuckDuckGo search queries and then scrapes each question without requiring a login. If you pass directUrls, it bypasses search and goes straight to the targeted questions, profiles, topics, or spaces.

Here is an example of a complete, runnable run_input JSON payload that configures search queries, sets limits, and enforces residential proxy usage:

{
  "searchQueries": ["python programming", "machine learning"],
  "directUrls": [
    "https://www.quora.com/What-is-Python-primarily-used-for",
    "https://www.quora.com/profile/Guido-van-Rossum-1"
  ],
  "maxResults": 20,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

A critical platform behavior to keep in mind is how Apify handles inputs. The platform Console UI has a prefill feature, but this is ignored by API calls. Only the default values defined in the schema are applied automatically if a key is missing. When you trigger this scraper programmatically via an API call, you must pass an explicit input dictionary containing all necessary keys to avoid unexpected default behavior.

How do you run the scraper programmatically with Python?

To integrate this scraper into a data pipeline, you can use the official Apify Python API client. This is the standard way to trigger runs, configure timeouts, and download the resulting dataset rows directly into your local environment.

The following Python script initializes the Apify client, starts a run with the input payload, waits for completion, and fetches the structured results:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run_input = {
    "searchQueries": ["machine learning"],
    "maxResults": 15,
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}

# Run the Actor and wait for it to finish
run = client.actor("crawlerbros/quora-scraper").call(global_timeout=240, run_input=run_input)

# Fetch results from the run's default dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    print(f"Type: {item.get('content_type')} | Title: {item.get('title') or item.get('name')}")

This snippet uses a global timeout parameter to abort the run if it hangs. Because Quora pages are resource-heavy, setting a defensive timeout prevents your platform credits from being drained by a stuck browser instance.

What is the 300 second synchronous run limit?

The synchronous run endpoint on Apify has a hard cap of 300 seconds and returns an HTTP 408 status code if the run takes longer. If you are scraping a large volume of keywords or direct URLs, your run will easily exceed this 5-minute threshold, causing your API client to throw an error.

For any production scale workloads, you must execute the run asynchronously. Instead of waiting for the call to finish, you POST to the runs endpoint, receive a run ID, and then poll the status or configure a webhook.

Here is how you handle this pattern programmatically in Python by starting the run asynchronously and polling its status:

import time
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run_input = {
    "directUrls": ["https://www.quora.com/topic/Python-programming-language-1"],
    "maxResults": 10
}

# Start the run asynchronously without blocking
run_info = client.actor("crawlerbros/quora-scraper").start(run_input=run_input)
run_id = run_info["id"]
print(f"Run started with ID: {run_id}")

# Poll the run status until it finishes
while True:
    status_info = client.run(run_id).get()
    status = status_info["status"]
    print(f"Current status: {status}")

    if status in ["SUCCEEDED", "FAILED", "ABORTED", "TIMED-OUT"]:
        break

    time.sleep(15)

if status == "SUCCEEDED":
    results = client.dataset(status_info["defaultDatasetId"]).list_items().items
    print(f"Successfully scraped {len(results)} items.")
else:
    print(f"Run ended with status: {status}")

If you use n8n for workflow automation, you do not need to write this polling loop. The n8n platform has a native Apify Trigger node that fires immediately upon run completion. If you host n8n yourself, you can use your standard Apify API key, whereas OAuth2 credentials are restricted to n8n Cloud instances.

How to parse and handle different Quora content schemas?

The quora-scraper returns a flat array of dataset rows, but the fields present in each row depend entirely on the content_type value. Your downstream data pipeline must inspect this field before loading data into a database to avoid schema validation errors.

Each row can represent a question, answer, profile, topic, or space. For example, a profile row contains follower_count and bio, while an answer row contains answer_text and is_ai_answer.

The following Python code demonstrates how to safely parse the raw JSON dataset and route the records based on their schema types:

import json

raw_dataset = """
[
  {
    "content_type": "answer",
    "title": "What is Python primarily used for?",
    "url": "https://www.quora.com/What-is-Python-primarily-used-for/answer/Pratima-Yadav-117",
    "answer_text": "Python is used for various purposes...",
    "is_ai_answer": false,
    "upvotes": 4,
    "scrape_timestamp": "2026-03-08T18:28:03.140078+00:00"
  },
  {
    "content_type": "profile",
    "title": "Guido van Rossum",
    "name": "Guido van Rossum",
    "url": "https://www.quora.com/profile/Guido-van-Rossum-1",
    "follower_count": 3100,
    "answer_count": 42,
    "scrape_timestamp": "2026-03-08T18:28:03.140078+00:00"
  }
]
"""

records = json.loads(raw_dataset)

for record in records:
    content_type = record.get("content_type")

    if content_type == "answer":
        # Handle answer-specific keys
        print(f"Processing Answer: {record['url']}")
        is_ai = record.get("is_ai_answer", False)
        text = record.get("answer_text", "")
        print(f"AI Answer: {is_ai} | Length: {len(text)}")

    elif content_type == "profile":
        # Handle profile-specific keys
        print(f"Processing Profile: {record['name']}")
        followers = record.get("follower_count", 0)
        print(f"Followers: {followers}")

How does Quora block your requests and how do you bypass it?

Quora blocks scraper requests by analyzing IP addresses, browser fingerprints, and request patterns. If you use standard datacenter proxies, Quora's firewall will immediately redirect your requests to a login wall or show a CAPTCHA challenge.

To bypass this restriction, you must use residential proxies. The scraper uses Apify's residential proxy group, which rotates IPs automatically. However, residential proxy sessions are not infinite. On Apify, residential proxy sessions expire and die after approximately 30 minutes, whereas datacenter sessions persist for up to 26 hours. If you run a large scraping job that scrolls a page for over 30 minutes, the underlying proxy session will die, which can lead to connection errors or unexpected page reloads.

To mitigate this, always set a reasonable maxResults cap per run to keep individual jobs under the 30-minute mark. If you need to target specific regions, you can use Apify's country targeting, which supports US state granularity via country-US_XX proxy configurations.

Why are some fields empty in the output schema?

Some fields in your scraped dataset will occasionally be empty. If you see empty strings for follow_count, author_credentials, bio, or description, this is not necessarily a bug in the parser.

Quora pages are highly dynamic and user-dependent. Some authors do not fill out their credentials, some topics do not have descriptions, and certain metrics are hidden depending on whether the page is viewed publicly or through a logged-in session. Because the scraper works entirely without authentication and does not require an account or cookies, it can only extract publicly visible content. Private profiles, restricted answers, and Quora+ paywalled content are completely inaccessible.

Furthermore, answer timestamps in Quora's UI are relative, for example, "2y" or "6mo", instead of exact ISO dates. The scraper returns these raw relative values in the answer_timestamp field. If your downstream database requires strict date formats, you will have to write a parser to convert these relative intervals into absolute timestamps relative to the scrape_timestamp field.

What are the system requirements and memory limits?

Quora's web application is built on top of complex JavaScript frameworks. A single page load downloads multiple megabytes of assets, styles, and tracking scripts, which rapidly inflates browser memory consumption.

To run this scraper successfully, you must allocate a minimum of 1024 MB of memory in your run settings. Allocating less memory than this platform minimum will cause the browser instance to run out of memory and crash mid-scroll.

On the Apify platform, compute costs are calculated using Compute Units (CUs). The formula is CU = (memory_mb / 1024) * duration_hours. Doubling the allocated memory is only CU-neutral for autoscaling runs that process multiple tasks or URLs concurrently for at least 30 seconds. For single-URL runs, doubling the memory will double your compute consumption without necessarily halving the run duration, as the bottleneck is often network latency rather than CPU speed.

How much does it cost to run this scraper at scale?

The cost of running this scraper is determined by two separate variables: the scraper's usage fee of $5 per 1,000 results and the underlying Apify platform resource usage. The platform rates scale across different subscription tiers.

Compute unit and residential proxy bandwidth costs vary by your subscription plan. On the Free and Starter plans, the compute unit rate is $0.20/CU and residential proxy bandwidth is billed at $8/GB. On the Scale plan ($199/mo), the compute rate drops to $0.16/CU and residential proxy bandwidth drops to $7.50/GB. On the Business plan ($999/mo), the compute rate is $0.13/CU and residential proxy bandwidth is $7/GB.

Because Quora pages are asset-heavy, residential proxy bandwidth is a significant cost driver. Every megabyte of images and scripts loaded during browser execution counts against your residential proxy usage. To minimize bandwidth costs on the Free or Starter plans, always specify a strict maxResults limit to prevent runaway page scrolling.

How do you control expenses and prevent runaway runs?

To prevent a single run from consuming your entire monthly credit balance, you should use the maxTotalChargeUsd query parameter. This parameter is exposed directly to the Actor's code as an environment variable named ACTOR_MAX_TOTAL_CHARGE_USD.

When this spend limit is tripped, the run initiates a termination sequence. However, it is not an instant kill. The container continues to consume resources briefly during the shutdown phase, so you may see a tiny overrun beyond the exact limit you set.

You can set this budget limit programmatically using the Apify API client when starting your run:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run_input = {
    "searchQueries": ["web scraping"],
    "maxResults": 50
}

# Start the actor with a strict budget cap of 1.50 USD
run_info = client.actor("crawlerbros/quora-scraper").start(
    run_input=run_input,
    max_total_charge_usd=1.50
)

print(f"Run {run_info['id']} started with a budget cap of $1.50 USD.")

If a run fails or aborts prematurely, you can use Apify's resurrection feature. Resurrecting a run restarts the container with the same storage, excludes the downtime from the total run duration, and restarts the timeout clock. This is highly useful if you hit a proxy failure mid-run and want to resume without losing the data you have already written to your dataset.

How do you automate this scraper with a schedule?

Apify platform schedules allow you to run the scraper at regular intervals. Schedules use a 6-field cron format where the seconds field is optional. The minimum allowed execution interval is 10 seconds.

There are two critical constraints when setting up schedules programmatically:

1. The Actor must have run successfully at least once before you can create a schedule for it.
2. New schedules are created in a DISABLED state by default. You must explicitly enable them after creation.

Here is an example of a configuration payload to schedule your scraper to run daily at midnight:

{
  "name": "daily-quora-scrape",
  "cronExpression": "0 0 0 * * ?",
  "isEnabled": true,
  "actorId": "crawlerbros/quora-scraper",
  "input": {
    "directUrls": ["https://www.quora.com/topic/Python-programming-language-1"],
    "maxResults": 10
  }
}

If you do not name the dataset storage associated with your scheduled runs, they will expire based on your plan limits. On the Free plan, unnamed storages are deleted automatically, keeping only the 10 most recent runs for 4 months. To prevent data loss, always write to a named dataset or offload your data to an external database immediately using webhooks upon run completion.

How do you route scraped data to external APIs using webhooks?

Apify webhooks support exactly one primitive action: sending an HTTP POST request to a target URL. There is no native AWS S3 or Slack integration built directly into the platform. If you want to push your Quora answers to an S3 bucket or a Slack channel, you must route the webhook payload through a middleware platform like n8n, Make, or a custom API gateway.

The following payload structure is an example of a webhook configuration that triggers on run completion. It sends a POST request with the run metadata to your server, allowing you to fetch the dataset programmatically:

{
  "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
  "requestUrl": "https://api.yourcompany.com/v1/apify-webhook",
  "payloadTemplate": "{\n  \"runId\": {{resource.id}},\n  \"datasetId\": {{resource.defaultDatasetId}},\n  \"status\": {{resource.status}}\n}"
}

Your receiving endpoint should parse this payload, verify the status is indeed SUCCEEDED, and use the datasetId to pull the records. To avoid hitting storage rate limits, make sure your pipeline adheres to Apify's limit of 60 requests per second per storage object. If your pipeline exceeds this threshold while fetching individual dataset items, the platform will rate-limit your requests.

What are the limitations and caveats of this scraper?

The quora-scraper has several clear technical boundaries that you must design around to avoid failed runs. It is not a magic solution for accessing restricted data, and its performance depends heavily on external factors.

Below are the key limitations and situations where this tool will fail:

• No Authentication Support: The scraper does not require or support Quora logins or session cookies. Consequently, you cannot scrape private user profiles, closed spaces, or content hidden behind the Quora+ paywall.
• Keyword Search Resolution: The keyword search relies on external search indexing via DuckDuckGo. This means search queries do not perform a direct, real-time database query on Quora; they only return the top relevant URLs indexed by the search engine.
• Heavy Memory Overhead: This tool cannot run on lower memory configurations. Attempting to run it with less than 1024 MB of memory causes the Chromium instance to run out of memory and crash due to Quora's heavy client-side scripts.
• Relative Timestamps: Answer timestamps are pulled exactly as they appear on the frontend (e.g., "3mo" or "1y"). If you require precise calendar dates for analysis, your pipeline must parse these relative strings manually.

The quora-scraper README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]