Google Earth Data with n8n Pipelines

Orchestrating Data Flows from Google Earth Scraper

The Google Earth Scraper Actor extracts precise geographic coordinates and metadata from Google Earth's internal search API, providing structured output of place names, descriptions, latitudes, longitudes, and Google Place IDs. Connecting this output into a broader data strategy involves addressing practical issues like idempotency and cost management, especially when routing data to a system like n8n or a data warehouse.

What search queries can I use with the Google Earth Scraper?

Define search queries by providing an array of strings to the searchQueries input field. Each string can be a location name or an address, and the Actor attempts to resolve each query to a single, primary location identified by Google Earth. It then extracts detailed information for that specific result.

The searchQueries field is a required array in the Actor's input schema, allowing you to specify multiple locations to be scraped in a single run. For example, if you're tracking specific landmarks, you would list each one. The Actor processes each query sequentially, applying minDelayBetweenRequests and maxDelayBetweenRequests to avoid hitting rate limits. Note that for multiple franchise locations, you must provide highly specific queries for each individual location rather than a broad term.

Here’s an example input structure for the google-earth-scraper:

{
    "searchQueries": [
        "Eiffel Tower",
        "Statue of Liberty",
        "Burj Khalifa"
    ],
    "minDelayBetweenRequests": 2,
    "maxDelayBetweenRequests": 5
}

This input instructs the Actor to search for three distinct locations. For each successful search, the Actor will emit a structured record containing the place's name, coordinates, and other available metadata. It's important to remember that prefill values seen in the Apify Console are not applied to API calls; you must always pass an explicit input dictionary via the API. Only the default values from the schema (like ["Eiffel Tower"] for searchQueries if not specified) are used when a field is entirely omitted from an API call.

How do I prevent duplicate data ingestion on reruns?

Implement a mechanism that tracks processed items, typically using a unique identifier from the source data. The google_place_id field in the Actor's output is ideal for this, allowing downstream systems to check for its existence before inserting new records, ensuring only new, unique location data is processed.

The google_place_id field in the google-earth-scraper output (e.g., 0x47e66e2964e34e2d:0x8ddca9ee380ef7e0) is a stable, unique identifier, ideal for use as an upsert key or uniqueness check in downstream systems.

Consider a scenario where you're pushing the scraped data into a PostgreSQL table. Before inserting a new record, you can query the table to see if a record with that google_place_id already exists.

import psycopg2
import os

# Assume conn and cursor are already established
# Example output item from google-earth-scraper
scraped_item = {
    "search_query": "Eiffel Tower",
    "place_name": "Eiffel Tower",
    "latitude": 48.85837,
    "longitude": 2.294481,
    "description": "Tower in Paris, France",
    "google_place_id": "0x47e66e2964e34e2d:0x8ddca9ee380ef7e0",
    "url": "https://earth.google.com/web/search/Eiffel+Tower/...",
    "data_source": "search_api",
    "scraped_at": "2025-11-28T12:00:00.000000"
}

conn = psycopg2.connect(os.environ.get("POSTGRES_CONNECTION_STRING"))
cursor = conn.cursor()

# Check if the place ID already exists
cursor.execute(
    "SELECT COUNT(*) FROM locations WHERE google_place_id = %s;",
    (scraped_item['google_place_id'],)
)
if cursor.fetchone()[0] == 0:
    # If not exists, insert the new data
    cursor.execute(
        """
        INSERT INTO locations (
            search_query, place_name, latitude, longitude, description,
            google_place_id, url, data_source, scraped_at
        ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s);
        """,
        (
            scraped_item['search_query'], scraped_item['place_name'],
            scraped_item['latitude'], scraped_item['longitude'],
            scraped_item['description'], scraped_item['google_place_id'],
            scraped_item['url'], scraped_item['data_source'],
            scraped_item['scraped_at']
        )
    )
    conn.commit()
    print(f"Inserted new location: {scraped_item['place_name']}")
else:
    print(f"Location already exists, skipping: {scraped_item['place_name']}")

cursor.close()
conn.close()

This Python snippet demonstrates a basic idempotency check. For larger-scale operations or data warehouses, you would use similar logic, leveraging MERGE or UPSERT commands where available, or designing your staging tables to handle duplicates before final insertion.

How do I integrate Actor output with n8n for downstream workflows?

Set up a webhook in Apify to trigger an n8n workflow upon the Actor's successful completion. The webhook sends a POST request containing run details, including a URL to the dataset items, which n8n then fetches and processes.

First, set up your google-earth-scraper Actor task. After it has run at least once (a requirement for scheduling and webhooks in Apify), you can configure a webhook. The webhook payload will contain details about the run, critically including the defaultDatasetId or links.apiDefaultDatasetUrl, which points to the results.

Here's an example of how you might configure a webhook using the Apify API:

import apify_client
import os

apify_client = apify_client.ApifyClient(os.environ.get("APIFY_API_TOKEN"))

# Replace with your Actor ID and n8n webhook URL
actor_id = "crawlerbros/google-earth-scraper"
n8n_webhook_url = "YOUR_N8N_WEBHOOK_URL_HERE"

# Create a webhook that triggers on successful Actor run completion
webhook_data = apify_client.webhooks().create(
    event_types=["ACTOR.RUN.SUCCEEDED"],
    request_url=n8n_webhook_url,
    payload_template="""{
        "runId": {{run.id}},
        "actorId": "{{run.actorId}}",
        "status": "{{run.status}}",
        "defaultDatasetId": "{{run.defaultDatasetId}}",
        "defaultDatasetUrl": "{{links.apiDefaultDatasetUrl}}"
    }""",
    is_enabled=True,
    actor_id=actor_id
)

print(f"Webhook created with ID: {webhook_data['id']}")

In your n8n workflow, the "Apify Trigger" node would catch this incoming POST request. Subsequent nodes in n8n could then use the defaultDatasetUrl (e.g., https://api.apify.com/v2/datasets/<dataset_id>/items) to fetch the actual scraped items. An HTTP Request node in n8n can retrieve the data, and then a function node or a database node can process and store it. This setup ensures your downstream systems receive data only when a scraping run has successfully completed.

Managing Actor Run Duration and Cost

Efficiently managing Actor run duration and cost is crucial for any production pipeline. The google-earth-scraper uses a headless Chromium browser, which consumes compute units (CUs) based on memory and CPU usage over time. Apify provides mechanisms to control these factors, such as run timeouts and cost caps.

The pricing structure on Apify is based on Compute Units (CUs), where 1 CU equals 1024MB memory usage for 1 hour. On the Starter plan, CUs cost $0.20 each. If your Actor uses 1024MB of memory, it consumes 1 CU per hour. The minDelayBetweenRequests and maxDelayBetweenRequests input parameters directly influence the total run time, and thus the cost. Shorter delays mean faster runs but potentially higher risk of rate limiting, which could lead to retries and increased CU consumption. Longer delays reduce the risk but extend the run duration.

The google-earth-scraper uses a headless Chromium instance, which requires memory and CPU. This execution environment contributes to the CU consumption even during delays.

A critical constraint to be aware of is the synchronous run endpoint hard cap. If you call the Actor directly via the /v2/acts/<actorId>/run endpoint and use the waitForFinish query parameter, the request will time out after 300 seconds (5 minutes) with an HTTP 408 status code. For runs expected to exceed this, you must initiate the run with a POST to /v2/acts/<actorId>/runs and then poll the run's status or use a webhook for completion notification.

You can also use the maxTotalChargeUsd parameter when starting an Actor run to set a hard cost limit. When this limit is approached, the Actor run will terminate. This is exposed to Actor code as ACTOR_MAX_TOTAL_CHARGE_USD, allowing the Actor to potentially implement graceful shutdown logic, though termination isn't instantaneous.

When running on the Starter plan ($19/month), CUs cost $0.20. If you need residential proxies (e.g., for very large-scale or geographically distributed searches), they cost $8/GB on the Free and Starter plans. Understanding these rates is essential for predicting and controlling your operational expenses. Always monitor your CU consumption in the Apify console to gauge actual usage against your estimates.

Handling Specific Failure Modes and Data Quality

Despite best efforts, issues can arise. The google-earth-scraper is robust, but external factors or malformed inputs can lead to incomplete data or run failures. Understanding its output schema helps in anticipating and handling these scenarios.

The output schema specifies that "Every field is only included when it was actually found - no placeholder or fabricated values are ever emitted." This is a crucial design choice: if a description or a google_place_id isn't available for a specific search query, that field will simply be absent from the output record, rather than being present with a null or empty string value. This implies that your downstream processing must be resilient to missing fields.

Consider the data_source field, which indicates whether coordinates were obtained via search_api (the reliable path) or url_fallback (parsed from the page URL when the API didn't respond in time). While search_api is preferred for its precision, url_fallback indicates a successful, albeit potentially less ideal, retrieval. Your data quality checks might flag url_fallback sources for manual review if extreme precision is required.

Another common failure mode involves searchQueries that don't resolve to a valid location. The Actor's README states, "One result is produced per search query... or no result at all if the query genuinely didn't resolve to a location." This means your dataset might have fewer items than your initial searchQueries array. Your data pipeline must account for this by either treating unresolvable queries as acceptable omissions or by logging them for further investigation.

Here’s how you might check for expected fields in a Python script processing the output:

output_item = {
    "search_query": "Imaginary City",
    "place_name": "Imaginary City",
    "latitude": 0.0,
    "longitude": 0.0,
    "data_source": "search_api",
    "scraped_at": "2025-11-28T12:00:00.000000"
    # Note: 'description' and 'google_place_id' are missing in this example
}

required_fields = ["place_name", "latitude", "longitude"]
optional_fields = ["description", "google_place_id"]

for field in required_fields:
    if field not in output_item:
        print(f"ERROR: Missing required field '{field}' for query '{output_item.get('search_query')}'")
        # Implement error handling, e.g., move to a dead-letter queue
    else:
        print(f"Field '{field}' present: {output_item[field]}")

for field in optional_fields:
    if field not in output_item:
        print(f"WARNING: Optional field '{field}' is missing for query '{output_item.get('search_query')}'")
    else:
        print(f"Field '{field}' present: {output_item[field]}")

This explicit checking ensures that your downstream systems gracefully handle the absence of optional data, rather than crashing due to unexpected nulls or missing keys.

How do I orchestrate scheduled runs for recurring data needs?

For recurring data needs, scheduling your google-earth-scraper Actor runs automates the data collection process. Apify's scheduling feature allows you to define cron-like intervals, ensuring your pipelines are regularly fed with fresh data.

When setting up a schedule, it's crucial to remember that new schedules are created in a DISABLED state by default. You must explicitly enable them after creation. Also, an Actor must have run at least once successfully before it can be scheduled. This initial run helps validate the Actor's configuration and ensures it can execute without immediate errors.

Schedules use a 6-field cron syntax (seconds, minutes, hours, day-of-month, month, day-of-week), offering granular control over execution times. The minimum interval between scheduled runs is 10 seconds.

Consider a scenario where you want to update location data daily. Here's how you might create a schedule via the Apify API:

import apify_client
import os

apify_client = apify_client.ApifyClient(os.environ.get("APIFY_API_TOKEN"))

actor_id = "crawlerbros/google-earth-scraper"
# Define input for the scheduled run
actor_input = {
    "searchQueries": [
        "Times Square, New York",
        "Golden Gate Bridge, San Francisco"
    ],
    "minDelayBetweenRequests": 3,
    "maxDelayBetweenRequests": 7
}

# Create a schedule to run daily at 3:00 AM UTC
# Cron format: seconds minutes hours day-of-month month day-of-week
# 0 0 3 * * * means 00 seconds, 00 minutes, 03 hours, every day of month, every month, every day of week
schedule_data = apify_client.schedules().create(
    actor_id=actor_id,
    cron_expression="0 0 3 * * *",
    is_enabled=True, # Remember to enable it!
    # Define how the Actor will run, including input
    settings={
        "input": actor_input,
        "memoryMbytes": 1024 # Standard memory setting
    },
    title="Daily Google Earth Location Update"
)

print(f"Schedule created with ID: {schedule_data['id']}")
print(f"Schedule is enabled: {schedule_data['isEnabled']}")

This schedule will trigger the google-earth-scraper every day at 3:00 AM UTC with the specified input. Combining scheduling with webhooks (as discussed earlier) creates a fully automated, hands-off pipeline that regularly refreshes your location data and pushes it to your downstream systems.

Data Retention and Storage Management

Understanding data retention policies is vital, particularly on different Apify pricing tiers, to avoid unexpected data loss. The google-earth-scraper outputs its results to a dataset, which is a key-value store for structured data.

By default, Actor runs create unnamed datasets. On the Free plan, only the 10 most recent runs are retained, and these unnamed storages expire after 4 months. This means if your workflow doesn't explicitly process and move data out of these default datasets, you could lose historical information. For production systems or any requirement to retain data long-term, you should explicitly name your storages (datasets, key-value stores, request queues). Named storages are exempt from deletion.

To ensure long-term data retention, after an Actor run completes (and a webhook has notified your n8n workflow, for instance), your workflow should fetch the data from the run's default dataset and store it in your own persistent storage (e.g., S3, a data warehouse, or a named Apify dataset).

Here's an example of retrieving items from a dataset and potentially pushing them to a named dataset for long-term storage:

import apify_client
import os

apify_client = apify_client.ApifyClient(os.environ.get("APIFY_API_TOKEN"))

# Assume this dataset ID comes from a webhook payload or a run object
run_default_dataset_id = "YOUR_ACTOR_RUN_DEFAULT_DATASET_ID"
target_named_dataset_id = "google-earth-locations-archive"

# Fetch items from the run's default dataset
items = apify_client.dataset(run_default_dataset_id).list_items().items

# Ensure the target named dataset exists or create it
try:
    apify_client.dataset(target_named_dataset_id).get()
except apify_client.ApifyClientAsyncError as e:
    if "Cannot find dataset" in str(e):
        print(f"Named dataset '{target_named_dataset_id}' not found, creating...")
        apify_client.datasets().get_or_create(name=target_named_dataset_id)
    else:
        raise

# Push items to the named dataset for long-term retention
if items:
    apify_client.dataset(target_named_dataset_id).push_items(items)
    print(f"Pushed {len(items)} items to named dataset '{target_named_dataset_id}'.")
else:
    print("No items to push.")

This ensures that even if you're on the Free plan or dealing with transient unnamed storages, your valuable scraped data is preserved in a named dataset that won't be subject to automatic deletion, providing a reliable archive for historical analysis.

Limitations and Caveats

While the google-earth-scraper is a powerful tool, it's essential to understand its specific limitations and the broader context of web scraping.

Firstly, the Actor relies on Google Earth's internal search API. This means its behavior, accuracy, and rate limits are ultimately controlled by Google. While the Actor uses minDelayBetweenRequests and maxDelayBetweenRequests to mitigate rate limiting, aggressive usage could still lead to temporary blocks or CAPTCHAs, even if not explicitly surfaced in the output. The Actor specifically uses a headless Chromium with software WebGL to render Google Earth's 3D interface, meaning it simulates a real browser environment. This is more robust than simple HTTP requests but also more resource-intensive.

Secondly, the Actor returns only one result per search query, which is Google Earth's primary (top-ranked/best-name match) result. For queries like "Starbucks," which have numerous franchise locations, the Actor will return data for only one prominent Starbucks location. If you need to scrape multiple franchise locations for a single brand, you would need to provide highly specific search queries for each individual location (e.g., "Starbucks 123 Main St, Anytown") or chain multiple runs, each with a single highly specific query. The current design does not support discovering and iterating through multiple results for a single broad query.

Thirdly, the output fields description and google_place_id are only included when available. Your data pipelines must be designed to gracefully handle their absence, as discussed in the "Handling Specific Failure Modes and Data Quality" section. This is not an error state but an inherent characteristic of the data source.

Finally, while the Actor does not require a Google Maps API key, it is still operating within Google's ecosystem. Any fundamental changes to Google Earth's internal search API or user interface could potentially impact the Actor's functionality, requiring updates. This is a common characteristic of web scraping tools that rely on specific website structures.

Checked against the Actor's input schema and Apify docs on 2026-09-13.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [e-mail]