DEV Community

Cover image for Bulk Indexing Your URLs, Automated (No More One-by-One API Calls)
Momen Abdelhafez
Momen Abdelhafez

Posted on

Bulk Indexing Your URLs, Automated (No More One-by-One API Calls)

If you've ever tried to get more than a handful of URLs indexed by Google, you already know the pain: Google's Indexing API works fine for one page — and then falls apart the moment you're dealing with hundreds.

Here's what that actually looks like, why it breaks down at scale, and how I ended up automating it.

The One-URL Version Works Fine

Google's Indexing API is straightforward for a single request:

async function submitUrl(url) {
  await jwtClient.authorize();
  const res = await fetch(
    'https://indexing.googleapis.com/v3/urlNotifications:publish',
    {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${jwtClient.credentials.access_token}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ url, type: 'URL_UPDATED' }),
    }
  );
  return res.json();
}
Enter fullscreen mode Exit fullscreen mode

Call it once, get a 200, move on. Nice and clean.

Where It Breaks Down

Now do that for 500 URLs — new blog posts, product pages, backlinks you built last week. A few problems show up fast:

  • Quota limits. The API caps requests per day. Loop over 500 URLs naively and you'll hit the ceiling.
  • No batch status view. The API tells you a request was accepted — not whether the URL is actually indexed. Those are different things.
  • Retry logic. Some submissions fail silently or need a re-try. You end up building your own queue.
  • Tracking over time. Indexing isn't instant. You need something polling status days later, not just firing requests once. None of this is hard, exactly — it's just infrastructure you have to build and maintain for a problem that isn't your core product.

What I Built to Skip This

I ended up wrapping all of this into GoogleIndexer — free, handles the batch/quota/retry/tracking problem so you don't have to:

  • Paste or upload a list of URLs — submits the whole batch
  • Tracks each URL's status automatically (pending → indexed)
  • Confirms the moment each one goes live in search
  • Works for pages and backlinks alike It's the same underlying API problem, just with the queueing and status-polling already handled.

When to Roll Your Own vs Use a Tool

Honestly — if you're only ever submitting a handful of URLs, the raw API snippet above is all you need. Where a tool like this starts to matter is once you're publishing or building links at real volume and don't want to own a whole submission pipeline just to know whether your content is actually discoverable.

If you want to try it: google-indexer.com — free, no credit card.

Would love to hear how others here are handling bulk indexing — anyone built their own queue system for this, or found other tooling that works well?

Top comments (0)