Emmanuel Uchenna for Apify

Posted on Sep 9 Originally published at eunit.me

How to Build an Automated SEO Pipeline: Connecting Sitemap Generator and Google Indexer to Notion

#seoautomation #googleindexingapi #technicalseo #webcrawling

Generating an XML sitemap is only the first step in modern technical Search Engine Optimization (SEO). If you have ever published a new blog post, launched a product line on an e-commerce platform, or updated critical landing pages, you know the frustration of waiting days, weeks, or even months for Google to crawl and index your content.

Relying on search engine crawlers to discover changes organically is no longer viable for high-growth websites. If you run a dynamic content platform, an online store with rapidly changing inventory, or a news publication, delayed indexing directly equates to lost traffic, lower visibility, and missed revenue.

In this guide, we will walk through the process of building an automated, enterprise-grade SEO pipeline. We will start by covering automated URL discovery using the Fast Sitemap Generator Apify Actor. Then, we will look at programmatically pushing those URLs to Google using the Google Indexer & Instant SEO Submitter Actor. Finally, we will solve the critical tracking challenge by using Model Context Protocol (MCP) Connectors to stream real-time indexing logs directly into a Notion database.

The evolution of programmatic indexing

Historically, website owners managed SEO by submitting a static Extensible Markup Language (XML) sitemap through Google Search Console. While essential, submitting a sitemap is fundamentally a passive notification. You are effectively dropping a note in Google's mailbox, asking Googlebot to visit your site whenever it gets around to it.

For small, static websites, this passive approach works. However, for modern dynamic sites, it falls short due to several structural challenges:

Crawl budget constraints: Search engines assign a finite "crawl budget" to every domain. On large sites with thousands of pages, Googlebot may exhaust its budget on legacy pages, completely ignoring newly published or updated content.
Orphaned or deep pages: Content buried deep within subdirectories or pagination chains often fails to receive internal link equity, making it nearly invisible to organic crawling.
Time-sensitive content: E-commerce price updates, breaking news, real estate listings, and job postings require instant indexation. Waiting two weeks for organic discovery renders the content stale before it ever reaches search results.
Desktop crawler overhead: Traditional site auditors like Screaming Frog run locally, consuming heavy system resources, tying up developer machines, and requiring manual CSV exports to process updates.

Moving from passive crawling to active push pipelines

To solve these limitations, search engines introduced programmatic indexing APIs. Instead of waiting for Googlebot to discover changes, you can actively send a push notification to Google the instant a page is created or updated.

By combining cloud-based crawling with active API submissions and real-time database logging, you can replace manual SEO workflows with a fully automated "set and forget" pipeline.

Stage 1: Automated URL discovery with Fast Sitemap Generator

Before you can submit URLs to Google, you need an accurate, deduplicated list of active links from your website. Hand-coding XML files or relying on CMS plugins that crash on large databases creates maintenance bottlenecks.

The Fast Sitemap Generator Actor runs on the Apify platform as a serverless cloud program. It uses a Direct Connection to crawl websites at high speeds directly from data centers, without requiring proxy infrastructure.

Configuring the URL crawler

Setting up automated URL discovery requires configuring key parameters inside the Apify Console to balance thoroughness with execution speed.

Here is an example input configuration for a production crawl:

{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxCrawlDepth": 3,
  "maxPagesPerCrawl": 5000,
  "includePatterns": [
    ".*/blog/.*",
    ".*/products/.*"
  ],
  "excludePatterns": [
    ".*/admin/.*",
    ".*/login.*",
    ".*/cart.*",
    ".*/checkout/.*"
  ],
  "sitemapFormats": [
    "xml",
    "html",
    "txt"
  ],
  "respectRobotsTxt": true,
  "changefreq": "daily",
  "defaultPriority": 0.8,
  "includeImages": false
}

Key configuration settings explained

Start URLs (startUrls): The entry point for the crawler. Usually, this is your primary domain homepage.
Max Crawl Depth (maxCrawlDepth): Controls how deep the crawler traverses link structures. A depth of 3 to 5 is typically sufficient to discover all essential content while preventing infinite loops.
Regex Filtering (includePatterns and excludePatterns): Prevents cluttering your index with utility pages. You should explicitly exclude administrative paths like .*/admin/.*, checkout pages .*/cart.*, and query parameters that create duplicate content.
Robots.txt Respect (respectRobotsTxt): Ensures your pipeline adheres to site governance rules and crawler restrictions.

Output storage and Dataset generation

When execution completes, the Actor stores output artifacts in two locations:

Key-Value Store: Contains compiled sitemap.xml, sitemap.html, and sitemap.txt files ready for static hosting.
Apify Dataset: Stores structured JSON records for every discovered page. The Dataset ID produced during this run serves as the input payload for the next stage in our pipeline.

Stage 2: Programmatic submission via Google Cloud infrastructure

With your site URLs discovered and structured into an Apify Dataset, the next phase is notifying Google. This step relies on the official Google Indexing API, which allows site owners to submit batch update requests directly to Google's indexing pipeline.

Step 1: Setting up Google Cloud credentials

Before making programmatic requests, you must configure authentication through the Google Cloud Console.

Create a Google Cloud Project: Log in to Google Cloud Console, click the project dropdown, and select New Project. Name it seo-automation-pipeline.
Enable the Indexing API: Navigate to APIs & Services > Library, search for "Web Search Indexing API", and click Enable.
Create a Service Account: Go to IAM & Admin > Service Accounts, click Create Service Account, name it indexing-bot, and assign it the Project > Editor role.
Generate a JSON Key: Select your new service account, go to the Keys tab, click Add Key > Create new key, select JSON, and download the file. Keep this file secure, as it grants API permissions.

Service Account Email Example:
indexing-bot@seo-automation-pipeline.iam.gserviceaccount.com

Step 2: Granting Google Search Console access

Enabling the API in Google Cloud is not enough on its own. Google requires verified domain ownership to prevent unauthorized indexing requests.

Open Google Search Console.
Select your verified domain property.
Navigate to Settings > Users and permissions.
Click Add user, paste your service account email address, set the role to Owner, and save.

IMPORTANT
Google Search Console requires Owner permissions for Service Accounts calling the Indexing API. Setting lower permission levels such as "Full" or "Restricted" will cause authentication errors during API calls.

Bypassing DIY limitations: Python vs. Apify Actors

Developers often attempt to write custom scripts to interact with the Google Indexing API. Below is a standard Python implementation using official Google authentication libraries:

import json
import requests
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession

SERVICE_ACCOUNT_FILE = "google-key.json"
INDEXING_API_ENDPOINT = "https://indexing.googleapis.com/v3/urlNotifications:publish"

def get_authenticated_session(key_path):
 scopes = ["https://www.googleapis.com/auth/indexing"]
 credentials = service_account.Credentials.from_service_account_file(
 key_path, scopes=scopes
 )
    return AuthorizedSession(credentials)

def submit_url(session, url, action_type="URL_UPDATED"):
 payload = {
        "url": url,
        "type": action_type
 }
 response = session.post(INDEXING_API_ENDPOINT, json=payload)
    return response.status_code, response.json()

if __name__ == "__main__":
 session = get_authenticated_session(SERVICE_ACCOUNT_FILE)
 target_url = "https://example.com/blog/automated-seo-pipeline"
 status, result = submit_url(session, target_url)
    print(f"Status Code: {status}")
    print(f"Response: {json.dumps(result, indent=2)}")

Why custom scripts break at scale

While the DIY Python script works for single URLs, operating it in production exposes major infrastructure hurdles:

Rate limit handling: Google enforces daily quotas (typically 200 URL notifications per day for standard projects). Custom scripts will fail with HTTP 429 Too Many Requests unless you write exponential backoff algorithms.
State management: A basic script does not remember which URLs were submitted yesterday. Running it daily resubmits duplicate URLs, wasting your daily quota.
Execution server maintenance: You must deploy the script to a cloud server, manage cron schedules, and store credentials securely.

The solution: Google Indexer & Instant SEO Submitter

Instead of writing and hosting custom boilerplate code, we use the Google Indexer & Instant SEO Submitter Actor.

Feature	Custom Python Script	Google Indexer Actor
Quotas & Rate Limiting	Manual handling required	Built-in automatic rate limiting & backoff
Input Source	Hardcoded or manual files	Direct integration with Apify Datasets & Sitemaps
Pricing Model	Server hosting costs	Pay-Per-Event
Test Mode	Requires code flags	Built-in dry-run toggle
Monitoring	Custom logging setup	Real-time console logs & dataset outputs

To configure the Actor, pass the JSON credentials and your Dataset ID from Stage 1 directly into the input schema:

{
  "datasetId": "YOUR_STAGE_1_DATASET_ID",
  "serviceAccountJson": {
    "type": "service_account",
    "project_id": "seo-automation-pipeline",
    "private_key_id": "abcdef123456789",
    "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC...\n-----END PRIVATE KEY-----\n",
    "client_email": "indexing-bot@seo-automation-pipeline.iam.gserviceaccount.com"
  },
  "action": "URL_UPDATED",
  "testMode": false
}

Stage 3: The missing link: Logging to Notion via MCP Connectors

Automating crawls and API submissions solves the execution side of SEO. However, engineering teams and SEO leads still face a visibility gap: How do you monitor pipeline results without manually checking console logs after each run?

Exporting CSVs after every run is tedious, and setting up custom webhooks requires maintaining dedicated receiver endpoints. This is where MCP connectors fit into the pipeline.

Understanding MCP connectors

The Model Context Protocol (MCP) is an open standard that lets Actors call third-party services like Notion, Slack, and GitHub on your behalf, using your own credentials, without the Actor ever holding your token directly.

MCP connectors on Apify work differently from the Apify MCP server. The Apify MCP server exposes Apify Actors as tools to external AI clients (such as Claude or Cursor). MCP connectors do the opposite: they let Actors running on the Apify platform call external MCP-compatible services. The two are independent and can be used together.

In this pipeline, we use the Notion Uploader Actor together with an Apify MCP connector to push the Google Indexer's results into a Notion database automatically after each run.

Setting up the Notion database

Create a dedicated database in your Notion workspace titled SEO Indexing Tracker.

Configure the database columns with the following properties:

Column Name	Property Type	Description
URL	Title	The target webpage URL submitted to Google
Submission Status	Select	`Indexed`, `Failed`, or `Quota Exceeded`
HTTP Code	Number	The status response code returned by Google (e.g., `200`, `429`, `403`)
Last Updated	Date	Timestamp of when the request occurred
Source Actor	Text	Name of the Apify Actor that executed the request

Authorizing the Notion MCP connector

MCP connectors are authorized once at the account level, not on individual Actor or task pages. To set one up:

Open Apify Console and go to Settings → API & Integrations → MCP connectors.
Click Add connector and select Notion.
Complete the OAuth flow — you will be redirected to Notion to authorize access to your workspace. The Apify platform never sees your token directly; authentication is handled server-side via the MCP proxy.
Once authorized, the connector appears as a selectable option when running any compatible Actor.

Connecting the Notion Uploader Actor

The Notion Uploader Actor reads any Apify dataset and uploads its rows into Notion. It accepts your MCP connector ID so you do not need to paste a Notion API key anywhere.

When the Notion Uploader runs as an Actor-to-Actor integration (triggered by the completion of the Google Indexer run), it automatically picks up the triggering run's output dataset — no explicit datasetId is needed in the payload template.

Here is an example input for the Notion Uploader when configured as a standalone run:

{
  "notionConnector": "<your-connector-id>",
  "notionDatabaseId": "https://www.notion.so/workspace/SEO-Indexing-Tracker-<db-id>",
  "datasetId": "<google-indexer-output-dataset-id>",
  "dedupeMode": "upsert",
  "keyProperty": "URL",
  "dryRun": false
}

Tip: Enable "dryRun": true on your first run. It resolves the field mapping, validates it against your Notion schema, and shows exactly what would be written — without creating any rows.

Orchestrating the full "set and forget" pipeline

Now that all three Actors are configured, we can chain them into an automated workflow: Fast Sitemap Generator discovers URLs, Google Indexer & Instant SEO Submitter submits them to Google, and Notion Uploader logs the results into Notion.

Step 1: Creating saved Actor tasks

An Actor task is a saved, reusable configuration of an Actor in Apify Console. Tasks let you pre-define all input parameters so they can be run on demand, via API, or on a schedule.

Crawler task: Open Fast Sitemap Generator in Apify Console, configure your startUrls, crawl depth, and regex filters, then click Save as a new task in the top-right corner. Name it sitemap-generator-task.
Indexer task: Open Google Indexer & Instant SEO Submitter, fill in your Google Cloud Service Account JSON key, then click Save as a new task. Name it google-indexer-task.
Uploader task: Open Notion Uploader, set your Notion MCP connector and target database URL, then click Save as a new task. Name it notion-uploader-task.

All saved tasks are listed under Saved tasks in the Apify Console left navigation.

Step 2: Chaining tasks using Actor-to-Actor integrations

The Integrations tab on each Actor or task page lets you chain runs together. When one task succeeds, Apify automatically triggers the next one and passes dynamic run data — such as the output dataset ID — via a payload template.

Chain 1: Sitemap Generator → Google Indexer

Open sitemap-generator-task in Apify Console and click the Integrations tab.
Click Add integration and select Run another Actor or task.
Choose google-indexer-task as the target.
Set the trigger to Run succeeded.
Set the payload template so the indexer receives the crawler's output dataset:

{
  "datasetId": "{{resource.defaultDatasetId}}",
  "action": "URL_UPDATED"
}

Chain 2: Google Indexer → Notion Uploader

Open google-indexer-task in Apify Console and click the Integrations tab.
Click Add integration and select Run another Actor or task.
Choose notion-uploader-task as the target.
Set the trigger to Run succeeded.
Leave datasetId out of the payload template — the Notion Uploader automatically uses the triggering run's dataset when no explicit ID is provided:

{
  "dedupeMode": "upsert",
  "keyProperty": "URL"
}

Now the full chain fires end-to-end on its own. When sitemap-generator-task completes, Apify triggers google-indexer-task with the crawler's dataset. When the indexer finishes, Apify triggers notion-uploader-task, which reads the indexer's output dataset and writes each result row into your Notion database.

Step 3: Scheduling automated pipeline execution

Configure a recurring schedule so the entire chain runs automatically.

In the Apify Console left sidebar, click Schedules.
Click Create new schedule.
Select sitemap-generator-task as the target. The full chain — crawl, index, upload to Notion — fires automatically when this task runs.
Set the frequency using a Cron expression or one of the presets.

For high-frequency publishing (news, job boards): Schedule daily at 2:00 AM UTC.
For standard marketing blogs and commercial sites: Schedule weekly, every Monday morning.

Cron expression example (daily at 2:00 AM UTC):
0 2 * * *

Best practices for automated SEO indexing

To maintain a healthy indexing strategy while avoiding Google API flags, keep these best practices in mind:

1. Submit only modified or new content

Do not resubmit your entire site inventory every day. The Google Indexing API is intended for pages that have new content (URL_UPDATED) or pages that have been deleted (URL_DELETED). Resubmitting unchanged pages wastes daily quotas and can result in rate limiting.

2. Match canonical URLs strictly

Google requires strict URL matching. If your canonical structure uses https://example.com/blog/post-name/ with a trailing slash, ensure your crawler filters and API submissions match that format exactly. Submitting http:// or non-trailing-slash variations can trigger redirect loops and indexing failures.

3. Use Test Mode during initial setup

Before running full-scale API operations, turn on the Test Mode toggle in the Google Indexer & Instant SEO Submitter Actor. Test Mode executes the full crawl, verifies JSON authentication keys, and simulates Notion MCP logging without consuming your daily Google API quota.

4. Monitor quota usage in Google Cloud Console

Standard Google Cloud projects receive an initial quota of 200 Indexing API requests per day. If your site publishes hundreds of new pages daily, submit a quota increase request directly through the Quotas tab in Google Cloud Console.

Wrapping up

Building an automated technical SEO pipeline transforms indexing from a passive waiting game into a predictable, programmatic system.

Combining the cloud crawling capabilities of the Fast Sitemap Generator, the push notifications of the Google Indexer & Instant SEO Submitter, and real-time database logging through Notion MCP Connectors, you establish an automated SEO engine that operates reliably in the background.

Instead of waiting weeks for Google to discover your newest content organically, your site updates are crawled, verified, submitted, and logged in Notion within minutes.

Explore the Fast Sitemap Generator on the Apify Store.
Set up the Google Indexer & Instant SEO Submitter Actor with your Google Cloud Service Account.
Connect your Notion workspace via MCP Connectors to start tracking real-time indexing status today.

DEV Community