DEV Community

Cover image for Company Domain Finder Results Cost More When Emitting Many Result Events
Crawler Bros
Crawler Bros

Posted on

Company Domain Finder Results Cost More When Emitting Many Result Events

Why Company Domain Resolution Costs Scale With Results Rather Than Runtime?

Costs for this Actor are determined by the number of result events generated and the memory allocated for the start event rather than the duration of the container. This pay-per-event model means a company website that takes 30 seconds to resolve costs the same as one that takes 2 seconds. Because compute time is not a factor, you do not need to optimize for execution speed to manage your budget effectively.

The primary driver of cost is the volume of companies you process. Each company name you provide in the companies array typically results in one "result" event. Since the Actor is built for bulk enrichment, the total cost scales linearly with the size of your input list. This structure is ideal for lead generation where network latency on target websites is unpredictable.

How the Actor Start Event and Result Tiers Determine Total Cost?

Every charge this Actor makes is one of two named events: "result" (apify-default-dataset-item) or "Actor Start" (apify-actor-start). The "Actor Start" event is priced at $0.005 per GB of memory allocated to the run. The "result" event price depends on your Apify VOLUME tier: FREE $0.002, BRONZE $0.00167, SILVER $0.00133, GOLD $0.001, PLATINUM $0.001, and DIAMOND $0.001.

Cost therefore scales with the number of events a run emits, not with how long the container runs. There are no separate platform-usage charges or subscription-plan rates on top of these per-event prices. If you allocate 1 GB of memory and resolve 1,000 companies, your cost is the sum of one start event and 1,000 result events at your current tier price. Checked against the Actor's input schema and Apify docs on 2026-09-20.

What Causes the Synchronous API to Return HTTP 408?

The synchronous run endpoint has a hard-cap of 300 seconds, and it returns an HTTP 408 error if the Actor does not finish within that window. This timeout occurs most frequently when processing large arrays of company names that exceed the processing capacity of a single five-minute request. While the connection drops, the Actor continues to run on the platform until completion.

To avoid this failure, you should use the asynchronous execution pattern for any list longer than a few dozen names. You can initiate the run via a POST request and then either poll the status or use a webhook to receive the results.

import requests

# Starting an asynchronous run to bypass the 300s limit
api_token = "YOUR_API_TOKEN"
actor_id = "crawlerbros/company-domain"
url = f"https://api.apify.com/v2/acts/{actor_id}/runs?token={api_token}"

payload = {
    "companies": ["OpenAI", "Stripe", "Anthropic", "SpaceX"],
    "country": "US",
    "autoProxyFallback": True
}

response = requests.post(url, json=payload)
run_id = response.json()["data"]["id"]
print(f"Started run {run_id}. Data will be available in the dataset upon completion.")
Enter fullscreen mode Exit fullscreen mode

Why Input Schema Prefill is Ignored by API Calls?

The prefill values defined in the Actor's input schema are only displayed in the Apify Console UI and are not applied to direct API calls. Only fields with a default property are automatically populated if they are missing from your JSON payload. If you rely on a prefill value without explicitly including it in your API request, the Actor will treat that field as undefined.

For the company-domain Actor, this means you must always include the companies array in your payload. While autoProxyFallback defaults to true and will be applied if omitted, any specific country hint or custom maxSocialLinks must be explicitly sent in your request.

{
  "companies": ["Microsoft", "Apple", "Google"],
  "country": "US",
  "maxSocialLinks": 5,
  "autoProxyFallback": true
}
Enter fullscreen mode Exit fullscreen mode

Targeting Proxy Fallback for Anti-Bot Protection

The Actor uses an autoProxyFallback mechanism that retries a website via a residential proxy only if the initial datacenter-IP fetch is blocked. This logic is more efficient than forcing all traffic through a proxy because it preserves residential data for sites that actually require it. Residential proxy sessions on the platform generally persist for around 30 minutes, which is sufficient for extracting social handles from a homepage.

The autoProxyFallback is enabled by default. It is specifically designed to handle websites like OpenAI or major financial institutions that frequently block datacenter traffic. Using the proxyConfiguration object to force all requests through a proxy is usually unnecessary and more expensive if it burns residential traffic on easily accessible sites.

// Recommended input structure for proxy efficiency
const input = {
  "companies": ["Goldman Sachs", "Morgan Stanley"],
  "autoProxyFallback": true,
  "proxyConfiguration": {
    "useApifyProxy": false 
  }
};
Enter fullscreen mode Exit fullscreen mode

Handling Missing Keys in Downstream Lead Enrichment?

The Actor omits fields entirely when data is not found, meaning you will not see null values for missing social media links in the output. If a company does not have a TikTok or a GitHub account, those keys will simply be absent from the resulting JSON record. This design requires your downstream code to use safe property access to prevent errors during data ingestion.

In Python, using .get() is the safest way to handle these missing keys. In JavaScript, you should use optional chaining or check for property existence. If your pipeline expects a static schema where every key is always present, it will fail when it encounters a company with a minimal digital footprint.

# Safe extraction of social links from Actor results
def parse_results(items):
    for item in items:
        name = item.get("companyName")
        # linkedin may be missing if the company has no profile
        linkedin = item.get("linkedin", "N/A")
        print(f"Company: {name}, LinkedIn: {linkedin}")

# Example output record shape
output_example = {
    "companyName": "OpenAI",
    "officialWebsite": "https://openai.com",
    "linkedin": "https://www.linkedin.com/company/openai",
    "twitter": "https://twitter.com/openai"
    # Note: 'tiktok' or 'facebook' are omitted if not found
}
Enter fullscreen mode Exit fullscreen mode

Why Schedules Start in a Disabled State?

New schedules on the Apify platform are created in a disabled state by default to prevent unintended resource consumption. Before a schedule can be enabled, the Actor must have been run successfully at least once. This ensures that the configuration is valid and the input parameters are producing the expected result events before the automation begins.

Schedules use a six-field cron expression and support intervals as short as 10 seconds. For the company-domain Actor, you should manually enable the schedule in the Console after verifying a manual run. This prevents a scenario where a misconfigured company list consumes your budget on a recurring basis.

# Conceptual schedule configuration
name: lead-enrichment-sync
cron: "0 0 * * 1" # Runs weekly on Monday
isEnabled: true
actorId: crawlerbros/company-domain
input:
  companies: ["Target A", "Target B"]
  country: "US"
Enter fullscreen mode Exit fullscreen mode

Identifying Low Confidence Matches in Your Pipeline?

The Actor includes a matchConfidence: "low" field in the output whenever a domain is selected that cannot be verified against the company name in the page title. This happens when the search returns a result that meets basic name-token requirements but lacks explicit branding in the metadata. High-confidence matches omit this field entirely, allowing you to filter for records that need manual review.

Treating the matchConfidence key as a flag is a reliable way to clean your data. In bulk enrichment, you might choose to discard low-confidence results or send them to a separate queue for human verification.

// Logic for filtering low confidence records
const processItems = (items) => {
  const verified = items.filter(item => !item.matchConfidence);
  const manualReview = items.filter(item => item.matchConfidence === 'low');

  console.log(`Verified ${verified.length} companies.`);
  console.log(`Sent ${manualReview.length} to review queue.`);
};
Enter fullscreen mode Exit fullscreen mode

How to Set a Hard Budget Limit on Actor Runs?

You can use the maxTotalChargeUsd query parameter to set a financial circuit breaker on any run. When the cumulative cost of the "Actor Start" and "result" events reaches this threshold, the run terminates. This is critical when processing dynamic lists of company names where the total count might be higher than anticipated.

When the cap is reached, the run is aborted, but it is not an instant kill. The platform provides a 30-second graceful shutdown window, which means the Actor may still emit a few final results before the container stops. This prevents the dataset from becoming corrupted while still protecting your account balance.

# Starting a run with a $2.00 budget cap via curl
curl -X POST "https://api.apify.com/v2/acts/crawlerbros/company-domain/runs?token=YOUR_TOKEN&maxTotalChargeUsd=2.00" \
     -H "Content-Type: application/json" \
     -d '{"companies": ["Example 1", "Example 2"]}'
Enter fullscreen mode Exit fullscreen mode

Data Retention and Storage Expiration Risks?

Unnamed storages, such as the default dataset created during a run, expire over time depending on your plan. On the free plan, only the 10 most recent runs are retained, and they are deleted after 4 months. If you do not export your company domain data or use named storage, you risk losing the results of your enrichment tasks once the retention limit is reached.

Named datasets are exempt from these automatic deletion rules. For long-term projects or recurring lead generation, it is safer to push your results to a named dataset or immediately move them to an external database via a webhook or n8n integration.

Real Limitations and Technical Caveats

The Actor has specific constraints designed to prevent low-quality data. It will not attempt to resolve 2-letter brand names like "EY" or "TK" because they are too ambiguous and often produce false positives. In these cases, the Actor emits a no_domain_match sentinel. Furthermore, the Actor does not verify social handle ownership; it merely extracts what the official website publishes.

If a company website uses extremely aggressive anti-bot walls that block both datacenter and residential proxies, the social link extraction may fail or be partial. The Actor is also limited to the platforms listed in the schema for its flat columns. While other platforms like VK or Weibo might appear in the socialLinks array, they do not get their own dedicated top-level fields in the JSON output.

Why a Country Hint is Necessary for Global Brands?

Supplying a country hint is the most effective way to disambiguate multinational brands or small businesses with generic names. This parameter biases the search toward local top-level domains like .de or .jp. Without it, the Actor might return a US-based charity's website for a UK-based accounting firm that happens to share the same name.

The country hint uses a string input that accepts full country names or 2-letter codes. It ensures the name-token matching and TLD preference logic prioritize the correct regional entity, which significantly reduces the frequency of the matchConfidence: "low" flag in your final dataset.

Managing Request Queue Constraints in Multi-Run Workflows

A request queue on the Apify platform can only be processed by one Actor or task run at a time. This means you cannot share a single queue of company names across multiple concurrent runs of the Actor to increase speed. While multiple runs can add items to a queue, the actual processing is limited to a single run instance.

If you have a massive list of 50,000 companies, you must split that list into smaller batches before starting the runs. Each Actor instance should be given its own unique set of companies via the companies input array to ensure that the work is distributed correctly without resource contention.

Handling the 60 Requests Per Second Storage Limit

While the Actor is highly efficient, users must stay within platform storage rate limits during high-volume operations. Storage objects have a limit of 60 requests per second, while dataset item pushes are capped at 400 requests per second. These limits are typically not an issue for individual runs but can be reached if you are running dozens of Actors in parallel that all push to the same dataset.

To keep your payloads manageable, you can use the maxSocialLinks integer to cap the number of additional links returned per company. This prevents unusually large websites with hundreds of outbound links from creating oversized result records that might impact storage performance or downstream ingestion speeds.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email info@crawlerbros.com

Top comments (0)