Username Reconnaissance with Sherlock Scraper and Python

Understanding the Cost of Distributed OSINT Scans

Analyzing the cost of running username reconnaissance at scale requires looking past simple flat-rate estimates and focusing on the underlying platform mechanics. The sherlock-scraper Actor is a wrapper around the open-source Sherlock tool, designed to probe over 400 social networks and websites to check if a specific profile exists. Because this process involves making hundreds of outbound HTTP requests per username, understanding how Apify calculates compute units and handles network traffic is critical to preventing run budget overruns.

Every time you execute this scraper, the platform tracks resource consumption using the standard compute unit formula:

CU = (memory_mb / 1024) * duration_hours

For this specific Actor, memory allocation is a primary cost driver. While doubling the allocated memory keeps the compute unit cost neutral for autoscaling runs, that neutrality only applies to solutions running multiple tasks or URLs for at least 30 seconds each. Because sherlock-scraper processes usernames sequentially, a poorly sized container can lead to unnecessary idle memory charges.

Beyond compute, network routing plays a significant role in your total bill. The Actor makes direct checks against hundreds of external platforms, many of which employ strict rate limiting and anti-scraping defenses. To bypass these blocks, you must factor in Apify's proxy pricing tiers.

On the Free and Starter plans ($19/mo), residential proxy bandwidth is billed at $8/GB. If you upgrade to the Scale plan ($199/mo), the residential proxy rate drops to $7.50/GB, and further decreases to $7/GB on the Business plan ($999/mo). Similarly, compute unit rates vary by plan tier: $0.20 per CU for Free and Starter tiers, $0.16 per CU for the Scale tier, and $0.13 per CU for the Business tier. Because Sherlock makes lightweight HTTP requests, compute consumption is relatively low, but proxy bandwidth can scale quickly if you run wide batch inputs.

Checked against the Actor's input schema and Apify docs on 2026-09-09.

What is the exact schema of a Sherlock run?

Every execution of sherlock-scraper requires a structured JSON payload defining the target identities. The input schema requires an array of strings under the usernames key. You can pass explicit, static usernames or use a wildcard modifier to programmatically expand the search space.

The following JSON example demonstrates a standard input payload containing both static usernames and a wildcard pattern:

{
  "usernames": [
    "johndoe",
    "alice_smith",
    "john{?}doe"
  ]
}

The output returned by the scraper is structured as a series of datasets containing the queried username and an array of verified profile links. This clean key-value separation makes it easy to pipe the results directly into downstream databases or analytical tools:

{
  "username": "johndoe",
  "links": [
    "https://github.com/johndoe",
    "https://www.reddit.com/user/johndoe",
    "https://www.instagram.com/johndoe",
    "https://twitter.com/johndoe",
    "https://www.tiktok.com/@johndoe"
  ]
}

How to trigger Sherlock Scraper programmatically?

To trigger the Actor programmatically, you must send an authenticated POST request to Apify's API endpoint with the required input schema. This approach bypasses the manual configuration steps in the console and allows your custom services to execute target checks programmatically.

Below is a Python script using the official apify-client library. It initializes the client with an API token, triggers the Actor with a target list, and fetches the resulting dataset items:

import os
from apify_client import ApifyClient

# Initialize the client with your Apify API token
client = ApifyClient(os.getenv("APIFY_TOKEN"))

# Prepare the actor input according to the schema
run_input = {
    "usernames": ["sherlock_target", "osint_hunter"]
}

# Run the actor
run = client.actor("crawlerbros/sherlock-scraper").call(run_input=run_input)

# Fetch results from the run's default dataset
dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in dataset_items:
    print(f"Username: {item.get('username')}")
    print(f"Discovered {len(item.get('links', []))} profiles.")

Why does the synchronous run endpoint return an HTTP 408 error?

The synchronous run endpoint returns an HTTP 408 error because it enforces a hard cap of 300 seconds on the request duration. If a search contains multiple usernames or a wide wildcard expansion, the sequential checks over 400+ platforms will exceed 5 minutes, causing the connection to time out.

A single username search typically takes 1 to 2 minutes to probe all 400+ platforms. Because the Actor processes multiple usernames sequentially, any input array containing more than two or three usernames will almost certainly exceed the 300-second threshold.

To prevent HTTP 408 errors, you must run the Actor asynchronously. Instead of waiting for the response in a single transaction, make a POST request to the run endpoint, which immediately returns a run ID, and then poll the status endpoint or utilize a webhook.

How to limit unexpected run charges using API parameters?

To restrict financial risks, pass the maxTotalChargeUsd query parameter directly to the Apify run endpoint. This parameter stops the run when the specified dollar limit is reached, though the container may keep consuming resources briefly during its shutdown sequence.

An unhandled loop or an accidentally massive wildcard expansion can run up a significant bill before you notice the activity. The platform provides a maxTotalChargeUsd query parameter, which is also exposed to the Actor's runtime environment as the ACTOR_MAX_TOTAL_CHARGE_USD environment variable.

The following curl command demonstrates how to trigger a run with a strict budget cap using the maxTotalChargeUsd parameter:

curl -X POST "https://api.apify.com/v2/acts/crawlerbros~sherlock-scraper/runs?token=$APIFY_TOKEN&maxTotalChargeUsd=0.50" \
     -H "Content-Type: application/json" \
     -d '{
       "usernames": ["john{?}doe"]
     }'

When does the wildcard expansion break your run budget?

Wildcard expansion breaks your budget when multiple wildcards or large arrays multiply the total count of sequential HTTP requests. Because Sherlock searches each variation sequentially, the duration and compute consumption scale linearly with each added separator mutation.

The usernames input field supports a special wildcard pattern: {?}. This wildcard automatically expands a single base string into three distinct variations using common separators: underscore, hyphen, and period. For example, inputting john{?}doe instructs the underlying engine to run checks for john_doe, john-doe, and john.doe.

While this is incredibly useful for finding name variations, the work multiplies if you combine multiple wildcards or pass a large array of them. To handle this safely, you should write a pre-flight validator in your application to check the size of the array and the presence of wildcards before calling the Apify API:

import sys

def calculate_estimated_runs(username_list):
    total_runs = 0
    for username in username_list:
        if "{?}" in username:
            # Each wildcard expands to 3 variations
            total_runs += 3
        else:
            total_runs += 1
    return total_runs

usernames_to_test = ["alice{?}smith", "bob{?}jones", "charlie_brown"]
estimated_checks = calculate_estimated_runs(usernames_to_test)

print(f"Total target usernames to process: {estimated_checks}")

# Impose an arbitrary safety ceiling of 10 checks for standard runs
if estimated_checks > 10:
    print("Error: Input exceeds safe batch execution limits. Reduce wildcards.")
    sys.exit(1)

What are the limitations and failure modes of Sherlock Scraper?

This Actor cannot access private, restricted, or deleted accounts, and it experiences false positives when targets block scraping IPs or redirect to registration screens. Additionally, Apify's residential proxies rotate approximately every 30 minutes, which can disrupt active sequential checks mid-run.

First, Sherlock cannot find private, deleted, or restricted-visibility accounts. It operates entirely on unauthenticated HTTP requests, scanning only what is publicly visible to an anonymous visitor. If a target username exists but the profile is set to private or restricted by regional laws, the site may return a 404 or a redirect, causing Sherlock to miss the account.

Second, rate limiting on target sites is a constant point of failure. Sherlock makes one request per site per username. Some target platforms may flag the scraping IP address. If a website starts blocking requests, it might return false negatives (reporting the account does not exist because the page was blocked) or false positives (interpreting a redirect to a login screen or an IP block page as a verified profile page).

Third, residential proxy sessions are short-lived. Apify's residential proxies persist for approximately 30 minutes before rotating. If you are running a massive batch scan that runs for hours, the connection will rotate sessions mid-run. While this helps bypass target IP blocks, the transition can occasionally disrupt active HTTP requests within the underlying Sherlock process, leading to incomplete scans for specific platforms.

Finally, you must be aware of storage limits and request limits on the Apify platform. Dataset pushes are capped at 400 requests per second, and unnamed storages on the free plan are subject to deletion. Only the 10 most recent runs are retained on the free tier, and they expire completely after 4 months. To prevent data loss, always write results to named storages or export them immediately upon run completion.

To handle potential false positives or service failures, your processing code should inspect the returned URLs and apply validation rules based on platform-specific behaviors:

import re

def validate_osint_results(dataset_item):
    validated_links = []
    username = dataset_item.get("username")
    links = dataset_item.get("links", [])

    for link in links:
        # 1. Instagram Redirect Validation:
        # Instagram often redirects unauthorized or rate-limited requests to /accounts/login/
        if "instagram.com" in link.lower() and "/login/" in link.lower():
            continue

        # 2. TikTok Specific Anti-Scraping / Challenge Check:
        # TikTok challenges or verification checks redirect to a captcha or trending page
        if "tiktok.com" in link.lower() and ("verify" in link.lower() or "trending" in link.lower()):
            continue

        # 3. Basic sanity check: does the link actually contain the username?
        # Use regex to isolate the path segment and check for exact match
        path_segments = [seg for seg in re.split(r'[/_?=\-+.]', link.lower()) if seg]
        if username.lower() not in path_segments:
            continue

        validated_links.append(link)

    return {
        "username": username,
        "links": validated_links
    }

# Example validation check
raw_result = {
    "username": "johndoe",
    "links": [
        "https://github.com/johndoe",
        "https://www.instagram.com/accounts/login/?next=/johndoe/",
        "https://www.tiktok.com/@johndoe/video/verify"
    ]
}

clean_result = validate_osint_results(raw_result)
print(clean_result)

Why are input schemas ignored by direct API calls?

Input schema prefill values are strictly a UI helper in the Apify Console and are ignored during API executions. Only the properties explicitly defined as default values in the Actor's schema are populated if they are absent from your JSON payload.

If you trigger the Actor programmatically and rely on the platform to fill in missing input fields because you saw them in the UI, your run may fail with a validation error indicating that the required usernames array is missing. To avoid this, never rely on UI-side parameters. Always pass a fully explicit input dictionary in your API payloads.

This behavior highlights a key difference between console testing and production integration. In the visual console editor, a prefill values placeholder provides a guide, but since the API skip-handles these prefilled states, you must ensure that your system-level integrations include the exact array configuration required by the input schema.

How to coordinate multi-run queue configurations?

To run parallel OSINT workers, you must bypass the single-queue limitation by running a custom Redis instance as an external message broker. Because an Apify Request Queue can only be processed by one Actor or task run at a time, trying to share a single platform queue across runs will cause resource locking.

If you need to execute hundreds of concurrent username checks, standard horizontal scaling with a single shared queue will fail. Implementing a custom Redis cluster lets your parent manager coordinate targets and assign them to independent Actor runs.

The following Python model illustrates how an external scheduler can dispatch batches to isolated Actor runs to prevent resource collision on the Apify platform:

import os
import json
from apify_client import ApifyClient

client = ApifyClient(os.getenv("APIFY_TOKEN"))

# Simulated target database (e.g. pulled from Redis)
target_batches = [
    ["user_one", "user_two"],
    ["user_three", "user_four"],
    ["user_five", "user_six"]
]

# Run containers independently to prevent queue contention
run_ids = []
for batch in target_batches:
    run = client.actor("crawlerbros/sherlock-scraper").start(
        run_input={"usernames": batch}
    )
    run_ids.append(run["id"])
    print(f"Dispatched batch {batch} to Run {run['id']}")

# Keep track of individual runs asynchronously
print(f"Active Runs under monitoring: {run_ids}")

How to schedule scans without manual triggers?

To schedule automated scans, you must first execute the Actor successfully at least once manually because the scheduler cannot bind to an inactive Actor. Additionally, all new schedules are created in a disabled state, requiring an explicit activation step.

If you need to monitor username usage over time, you can automate your runs using Apify's scheduling system. Schedules use a 6-field cron syntax (where seconds are optional) and support a minimum run interval of 10 seconds.

Your infrastructure deployment scripts must handle this by first triggering a manual run, creating the schedule, and then explicitly patching the schedule configuration to enable it. This multi-step initiation ensures the target Actor has a validated state before cron execution begins.

How to integrate Sherlock results with external platforms?

To send your scraping results to external services like AWS S3 or Slack, you must use webhooks or integration hubs like n8n because Apify does not offer native integrations. Webhooks act as your primary event-driven tool, sending a single POST request containing run information directly to your target endpoint.

If you are using n8n, you can avoid polling entirely by using the native Apify Trigger node, which fires automatically on run completion. This integration works with an API key for self-hosted n8n instances, while OAuth2 credentials are restricted to n8n Cloud environments. For all other custom architectures, configuring a webhook to hit your own API gateway is the most robust approach.

Using these integration patterns allows you to build highly responsive pipelines that process username findings immediately after the Actor finishes its scan.

The Actor's README is the source of truth for its inputs, outputs and limits. Written with AI assistance. Need a hand wiring this into your stack? Email [e-mail]