loopnet-scraper Returns Empty Results When Akamai Blocks It

As a senior data engineer, I'm often asked about integrating data acquisition tools into more complex, autonomous systems. The rise of AI agents has shifted this conversation from simple API calls to a richer "tool" paradigm, where the agent needs to understand not just how to call a tool, but what the tool does, its inputs, and its potential failure modes. This is particularly true when dealing with services that expose their capabilities through a Machine Call Protocol (MCP), like Apify Actors.

Let's examine how an AI agent interacts with a specialized tool like the loopnet-scraper via MCP, what its signature looks like, and the practical limits an agent will encounter, especially when dealing with robust anti-bot measures.

How does an AI agent "see" the loopnet-scraper Actor as a tool?

When an AI agent accesses Apify's MCP server at https://mcp.apify.com with a scoped request like ?tools=crawlerbros/loopnet-scraper, the platform generates a tool signature based on the Actor's input schema. This signature translates the schema's structured fields into a format the agent can interpret to construct valid requests. The agent doesn't receive a generic "run Actor" command; it gets a specific blueprint for calling loopnet-scraper.

The core of this blueprint is the input schema. For loopnet-scraper, the agent would perceive a tool with parameters for startUrls, searchType, propertyType, state, city, and various numerical and boolean filters. The description field of each input parameter becomes crucial here, providing semantic context to the AI for appropriate value selection. Default values are also exposed, influencing the agent's decision-making if specific parameters aren't explicitly provided by the user's prompt.

What does the loopnet-scraper tool signature look like to an agent?

The tool signature presented to an AI agent is derived directly from the Actor's input schema, transforming it into a callable function definition. For loopnet-scraper, the agent would interpret something akin to a function signature that accepts an object with specific properties, their types, descriptions, and defaults.

For example, the startUrls parameter, an array of strings, would map to a list of URLs. searchType, an enum, would clearly indicate its allowed string values (for-sale, for-lease, businesses-for-sale, brokers). The numerical fields like priceMin and priceMax would be typed as integers, with their descriptions clarifying they represent USD values and that 0 implies no minimum or maximum. This precise mapping allows the AI to validate its proposed inputs against the tool's capabilities before making a call.

Here's a simplified representation of how an AI agent might internally parse the loopnet-scraper's tool signature:

{
  "name": "loopnet-scraper",
  "description": "Scrape commercial real estate listings, broker profiles, and businesses for sale from LoopNet.com. Extracts title, price, address, property type, size, broker info.",
  "parameters": {
    "type": "object",
    "properties": {
      "startUrls": {
        "type": "array",
        "description": "LoopNet URLs: search pages, listing pages, broker profiles, or business-for-sale pages.",
        "items": { "type": "string" }
      },
      "searchType": {
        "type": "string",
        "description": "Type of LoopNet search when building a URL from filters.",
        "enum": ["for-sale", "for-lease", "businesses-for-sale", "brokers"],
        "default": "for-sale"
      },
      "propertyType": {
        "type": "string",
        "description": "Property type filter.",
        "enum": ["any", "office", "retail", "industrial", "multifamily", "land", "hotel", "health-care", "specialty"],
        "default": "any"
      },
      "state": {
        "type": "string",
        "description": "US state filter (two-letter). 'any' = nationwide (USA).",
        "default": "any"
      },
      "city": {
        "type": "string",
        "description": "Optional city filter. Format: 'los-angeles-ca', 'miami-fl' (city-state code with hyphens)."
      },
      "priceMin": {
        "type": "integer",
        "description": "Minimum listing price. 0 = no minimum.",
        "default": 0
      },
      "priceMax": {
        "type": "integer",
        "description": "Maximum listing price. 0 = no maximum.",
        "default": 0
      },
      "buildingSizeMin": {
        "type": "integer",
        "description": "Minimum building square footage. 0 = no minimum.",
        "default": 0
      },
      "buildingSizeMax": {
        "type": "integer",
        "description": "Maximum building square footage. 0 = no maximum.",
        "default": 0
      },
      "includeListingDetails": {
        "type": "boolean",
        "description": "Follow each listing URL to fetch its detail page (JSON-LD price, description, image, building size, cap rate). Slower but produces richer records.",
        "default": true
      },
      "maxItems": {
        "type": "integer",
        "description": "Maximum listings per run.",
        "default": 25
      }
    },
    "required": []
  }
}

How does an AI agent formulate a run_input for LoopNet?

The agent uses the structured definition to construct its run_input for the Actor. When an agent determines it needs to call loopnet-scraper, it will formulate a POST request to the Apify API endpoint, including its API token for authentication. Crucially, while the Apify Console UI displays prefill values for inputs, these are not applied to API calls.

An AI agent, when constructing its run_input dictionary, must explicitly pass all desired parameters; only default values are applied if a parameter is omitted from the API call.

Here's an example of a run_input JSON an agent might construct to search for industrial properties in California with specific size constraints:

{
  "startUrls": [],
  "searchType": "for-sale",
  "propertyType": "industrial",
  "state": "CA",
  "buildingSizeMin": 5000,
  "buildingSizeMax": 20000,
  "maxItems": 50,
  "includeListingDetails": true
}

To initiate such a run using the Apify API with curl, an AI agent could execute:

curl -X POST -H "Content-Type: application/json" \
     -H "Authorization: Bearify YOUR_API_TOKEN" \
     -d '{
           "startUrls": [],
           "searchType": "for-sale",
           "propertyType": "industrial",
           "state": "CA",
           "buildingSizeMin": 5000,
           "buildingSizeMax": 20000,
           "maxItems": 50,
           "includeListingDetails": true
         }' \
     "https://api.apify.com/v2/acts/crawlerbros~loopnet-scraper/runs"

Why do LoopNet scraping jobs often return few results or fail completely?

LoopNet.com is protected by Akamai Bot Manager, a sophisticated anti-bot system that actively enforces JS-challenge mechanisms. The loopnet-scraper Actor attempts to bypass this using curl_cffi (a Chrome TLS fingerprint) and a rotating Apify RESIDENTIAL US/CA proxy pool. However, even with these measures, a significant rate of 403 responses (forbidden) is common.

Akamai frequently demands browser-level JavaScript execution to issue session tokens, a capability that standard HTTP scraping, even with advanced impersonation, struggles to fully emulate.

When the Actor encounters persistent Akamai blocks, it will emit a loopnet_akamai_blocked sentinel record in its output, detailing the URLs that were attempted but failed. This is a critical failure mode an AI agent must be programmed to recognize. Without explicit handling, an agent might repeatedly call the tool, incurring costs without desired results. For guaranteed scraping, the Actor's README explicitly states that routing through a paid anti-bot service (like ZenRows or ScrapingBee) is necessary, as these services handle the browser-level challenges.

This means an AI agent needing reliable LoopNet data should, in its planning phase, assess whether its task requires the "guaranteed scraping" level of bypass. If so, it would need to incorporate calls to these external anti-bot services, significantly increasing the complexity and cost of the overall data pipeline.

What is the output record shape when loopnet-scraper succeeds?

When the loopnet-scraper successfully navigates Akamai and extracts data, it pushes individual records to the default dataset. Each record represents a commercial real estate listing or other extracted item (like a broker profile). The output shape is consistent, providing structured data for downstream processing by the AI agent or other systems.

The includeListingDetails input parameter is crucial here; when set to true (which is the default), the Actor follows each listing URL to fetch richer data from its detail page. This adds fields like detailed descriptions, additional images, specific building sizes, and capitalization rates, though it also slows down the run and increases the likelihood of encountering Akamai blocks on subsequent requests.

Here's the typical shape of a successful loopnet-scraper output record:

[
  {
    "type": "listing",
    "url": "https://www.loopnet.com/listing/some-property-id/",
    "listingId": "some-property-id",
    "title": "Industrial Flex Space for Lease",
    "priceText": "$12.00 /SF/YR",
    "priceNumeric": 12.00,
    "address": "123 Main St, Anytown, CA 90210",
    "sizeText": "10,000 SF",
    "imageUrl": "https://images.loopnet.com/d2x2/some-image.jpg",
    "scrapedAt": "2026-09-27T10:30:00.000Z",
    "description": "Well-maintained industrial flex space with roll-up doors...",
    "propertyType": "Industrial",
    "capRate": "5.5%"
  }
]

When Akamai blocks occur, the output differs significantly. Instead of listing records, a sentinel record provides crucial debugging information for an AI agent:

[
  {
    "loopnet_akamai_blocked": true,
    "attemptedUrls": [
      "https://www.loopnet.com/for-sale/california/",
      "https://www.loopnet.com/for-sale/industrial/"
    ],
    "description": "Akamai Bot Manager blocked access. Requires browser-level JS execution. Consider using a paid anti-bot service.",
    "scrapedAt": "2026-09-27T10:35:00.000Z"
  }
]

An agent needs to parse for this loopnet_akamai_blocked flag to avoid misinterpreting an empty dataset as a successful run with no results, and instead correctly identify a failure requiring re-evaluation of the strategy.

How can an AI agent detect and handle Akamai blocks?

An AI agent can detect Akamai blocks by examining the Actor's output dataset for the loopnet_akamai_blocked sentinel record. This record indicates a scraping failure due to anti-bot measures, providing crucial context for the agent to adjust its strategy. Upon detecting this, the agent should not assume an empty result means no listings were found, but rather that the scraper was blocked.

A robust AI agent's error handling could look like this in Python:

import apify_client

# Initialize the Apify Client with your API token
client = apify_client.ApifyClient("YOUR_API_TOKEN")

# Example run configuration
run_input = {
    "startUrls": ["https://www.loopnet.com/for-sale/california/"],
    "maxItems": 10
}

try:
    # Run the Actor
    run = client.actor("crawlerbros/loopnet-scraper").call(run_input=run_input)

    # Fetch results from the default dataset
    dataset_items = client.dataset(run["defaultDatasetId"]).list_items().items

    if any("loopnet_akamai_blocked" in item for item in dataset_items):
        print("Scraping failed: Akamai Bot Manager blocked access.")
        # Agent's strategy adjustment: e.g., report failure, suggest using anti-bot service
    elif not dataset_items:
        print("Scraping completed, but no results found (possibly due to Akamai or no matches).")
    else:
        print(f"Scraping successful: Retrieved {len(dataset_items)} items.")
        # Process the scraped data
        for item in dataset_items:
            print(f"Title: {item.get('title')}, Price: {item.get('priceText')}")

except Exception as e:
    print(f"An error occurred during Actor execution: {e}")

How does the synchronous run timeout affect AI agent calls?

The synchronous run endpoint for Apify Actors has a hard-coded timeout of 300 seconds (5 minutes). If a loopnet-scraper run exceeds this duration, the API call will return an HTTP 408 (Request Timeout) error. This is a critical limitation for AI agents, especially when includeListingDetails is true, as following detail pages can significantly extend run times.

For short, targeted searches with maxItems set to a small number, an AI agent might successfully use the synchronous endpoint. However, any complex or broad query is highly likely to exceed the 300-second cap. When this happens, the agent cannot simply retry the synchronous call; it must instead switch to an asynchronous execution model. This involves POSTing to /v2/acts/<actor>/runs and then polling the run's status or setting up a webhook to be notified upon completion.

An AI agent designed for robustness should anticipate this timeout. If an initial synchronous call fails with a 408, the agent should automatically pivot to an asynchronous strategy. This adds complexity, as the agent then needs a mechanism to store the run ID, periodically check its status, and retrieve the results once the run finishes, which could be minutes or even hours later. The agent's internal state management must account for these long-running operations.

What is the cost model for running loopnet-scraper as an AI tool?

The loopnet-scraper Actor operates on a PAY_PER_EVENT pricing model. This means its cost is determined by specific, named events emitted during a run, in addition to the standard Apify platform usage (which is billed separately at your plan's rates). It's crucial for an AI agent to understand this model to manage cost ceilings.

The charged events for loopnet-scraper are:

• "result" (apify-default-dataset-item): This event costs $0.002 per item pushed to the default dataset. This is the primary driver of event-based costs. The number of these events directly scales with the maxItems input parameter and the success rate of scraping. If includeListingDetails is true, each successful listing will count as one result, but the overall time (and thus platform usage) might increase, potentially allowing more records to be found before maxItems is reached. Discount tiers apply to this event: FREE $0.002, BRONZE $0.00167, SILVER $0.00133, GOLD $0.001, PLATINUM $0.001, DIAMOND $0.001.

• "Actor Start" (apify-actor-start): This event costs $0.005 per GB of memory allocated to the run, charged once per run (with a minimum of one event). This covers the overhead of initiating the Actor. Even runs that fail quickly due to Akamai blocks will incur this "Actor Start" charge, along with some platform usage.

An AI agent can use the maxTotalChargeUsd query parameter when initiating a run to set a hard cap on the total event-based cost. When this cap is tripped, the run will terminate. However, termination is not instantaneous; the run will continue to consume resources briefly before shutting down. An agent must factor this slight overshoot into its cost-management strategy. The actual total cost of a run will be the sum of these event charges plus the platform usage (CPU, memory, storage) consumed.

What are the key limitations an AI agent will hit with loopnet-scraper?

Beyond the Akamai challenge, loopnet-scraper has specific limitations that an AI agent must understand to avoid incorrect assumptions or inefficient execution.

First, the Actor has a high 403 rate due to Akamai, as detailed previously. This isn't a minor inconvenience but a fundamental hurdle that can prevent successful data extraction entirely. An AI agent needs robust error handling for the loopnet_akamai_blocked sentinel record, potentially triggering fallback strategies or reporting failure to the user rather than silently returning empty datasets.

Second, the Actor provides no demographic or broker-detail enrichment. LoopNet.com relies heavily on client-side JavaScript to render these deeper details within its Single Page Application (SPA). The current scraping approach, even with curl_cffi and residential proxies, does not execute the full browser-level JS required for this richer data. If an AI agent's task requires broker contact details, specific demographic overlays for properties, or other SPA-rendered information, loopnet-scraper will not provide it. The agent would need to identify alternative data sources or more advanced scraping methods (e.g., full browser automation) to fulfill such requirements.

Third, the Actor explicitly states no pagination beyond the first page of a startUrl. This is a significant constraint. If an AI agent provides a startUrl that is a search results page, it will only process the listings visible on that initial page. It will not automatically navigate to subsequent pages (e.g., page 2, page 3, etc.) to collect more results. To retrieve more listings from a broad search, the AI agent must either:

• Supply multiple startUrls (if it can construct distinct search page URLs for different result sets).
• Strategically adjust other input parameters like state, city, propertyType, or price ranges to generate new startUrls that target different subsets of listings, effectively segmenting the search space.

This limitation means an AI agent cannot simply provide a single broad search URL and expect loopnet-scraper to exhaust all available results beyond the initial view. It requires a more intelligent, iterative approach from the agent to cover large datasets, or an acknowledgment that only the first page's data will be retrieved.

Finally, Apify platform mechanics introduce further constraints. Unnamed storages (like those created by default for loopnet-scraper runs) expire. On the free plan, only the 10 most recent runs are retained for four months. For long-term data retention, an AI agent must either use named storages or proactively download and store results externally. Additionally, residential proxy sessions typically last around 30 minutes. For long-running loopnet-scraper tasks (which are more likely with includeListingDetails=true and high maxItems), the frequent rotation of residential proxies might further contribute to Akamai challenges or slower processing, as new sessions need to establish trust.

Checked against the Actor's input schema and Apify docs on 2026-09-27.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]