I Built a Discourse Plugin to Automatically Submit New Topics to Search Engines
If you run a Discourse forum, you've probably experienced this: someone publishes a new topic, but it takes a while before search engines even discover the page.
Discourse already has sitemaps, and they work fine. But I wanted something a little more proactive.
Instead of waiting for a search engine to crawl my forum, why not tell it whenever something changes?
That's what got me interested in IndexNow.
I ended up building an open-source Discourse plugin called discourse-indexnow to automate the whole process.
The Problem with Waiting for Crawlers
Discourse forums generate a lot of dynamic content.
People create new topics, edit posts, reply to discussions, and sometimes delete old content.
Traditional XML sitemaps help search engines discover URLs, but they don't immediately notify search engines when something changes.
For smaller forums, especially newer communities, this can be frustrating.
You might publish a useful technical guide today, but have no idea when a search engine will discover it.
I wanted a way to reduce that delay without manually submitting URLs every time I published something.
So, What Is IndexNow?
IndexNow is an open protocol that lets websites notify participating search engines when URLs are added, updated, or deleted.
Instead of relying entirely on periodic crawling, your website can send a notification when something changes.
For example, an IndexNow batch request looks something like this:
{
"host": "forum.example.com",
"key": "your-indexnow-api-key",
"urlList": [
"https://forum.example.com/t/example-topic/123",
"https://forum.example.com/t/another-topic/456"
]
}
Search engines such as Bing and Yandex support the protocol.
One important thing to understand: IndexNow doesn't guarantee indexing or better rankings.
It simply helps participating search engines discover changes sooner.
Also, Google doesn't currently support IndexNow, so you should still maintain your sitemap and existing SEO practices.
Building discourse-indexnow
I wanted the integration to work directly inside Discourse rather than relying on an external cron job or a separate automation service.
The basic idea was simple:
When something happens to a public topic, the plugin determines whether its URL should be submitted and then handles the IndexNow notification in the background.
But once I started thinking about real forum activity, there were a few more things to consider.
1. Automatically submit new and updated topics
This was the main feature I wanted.
When someone creates a public topic, the plugin automatically submits its URL.
When the first post is edited or important topic attributes change, the URL can be submitted again.
It also handles deleted topics, allowing participating search engines to discover that the original URL may no longer be available.
No need to copy URLs or manually open a webmaster dashboard.
2. Handle replies and paginated topics
Here's something that makes forums a little different from regular blogs.
A discussion can grow across multiple pages.
If a new reply appears on page three, submitting only the main topic URL might not be the most useful approach.
So I added optional reply-triggered submissions.
The plugin can submit the URL of the page containing the new reply.
To avoid flooding IndexNow with notifications on busy discussions, there's also a configurable cooldown for each URL.
3. Support multilingual topics
Discourse supports content localization, and I wanted the plugin to work with it.
If a topic has localized versions available to crawlers, the plugin can submit the relevant localized URLs alongside the original.
For example:
https://forum.example.com/t/example/123
https://forum.example.com/t/example/123?tl=es
https://forum.example.com/t/example/123?tl=fr
These are illustrative URLs; the plugin uses the locale parameters actually configured in Discourse.
Rather than making a separate request for every language, the URLs can be grouped into one batch.
This was especially useful for making the integration work with multilingual communities.
4. Don't accidentally submit private content
This was one of the most important design considerations.
A forum isn't just a collection of public blog posts.
Discourse can contain private messages, restricted categories, unlisted topics, and other content that shouldn't be sent to search engines.
The plugin checks whether a URL is eligible before submitting it.
It also supports additional category and tag exclusions.
The goal is to make automatic submission useful without exposing URLs that should remain private.
5. Add rate limiting and retry handling
Automating URL submission sounds easy until you think about what happens on a busy forum.
Imagine hundreds of topics being updated in a short period.
You don't want every event generating unlimited API requests.
So I added:
- Hourly and daily submission limits
- Per-URL cooldowns
- Batch submissions
- Handling for HTTP 429 responses
- Retry delays based on the API response
The plugin also splits large batches to respect IndexNow's 10,000-URL-per-request limit.
All of this happens asynchronously, so submissions don't need to block normal forum interactions.
An Admin Dashboard Makes Things Easier
I also wanted to know whether the integration was actually working.
Just sending API requests and hoping for the best didn't seem like a great solution.
So the plugin includes an admin interface that displays:
- Successful and failed submissions
- Submission history and trigger reasons
- Hourly and daily quota usage
- Seven-day activity trends
- Error categories
- API key verification status
It also supports manual URL submission and historical backfills.
For example, if you install the plugin on a forum that already has hundreds of public topics, you can select a category and date range, preview the matching topics, and submit them in batches.
That's much more convenient than submitting existing URLs one at a time.
How to Install It
The plugin is open source and available on GitHub.
Repository: imlotso/discourse-indexnow
After installing the plugin using the standard Discourse plugin installation process and rebuilding your container, open:
Admin > Plugins > discourse-indexnow
From there, generate an IndexNow API key and enable submissions.
You'll also need to verify that the key file is publicly accessible:
https://your-forum-domain.com/<your-key>.txt
The endpoint should return HTTP 200 and the key as its response body.
The plugin provides a built-in check for this.
I've also written a more detailed guide covering the plugin's features, configuration options, URL filtering, and troubleshooting.
👉 Discourse IndexNow Plugin — Complete Setup Guide and Feature Overview
That guide includes the configuration reference and explains how each submission mechanism works.
Does This Actually Improve SEO?
This is probably the most important question.
IndexNow can help supported search engines learn about new or updated content faster.
But notifying a search engine isn't the same thing as getting indexed.
And getting indexed isn't the same thing as ranking well.
Content quality, crawlability, internal linking, canonical URLs, and many other SEO factors still matter.
I see IndexNow as an additional discovery mechanism rather than a replacement for traditional SEO.
For an active Discourse community, reducing the time between publishing content and notifying search engines seems like a useful improvement.
Whether that translates into better organic search performance is something worth measuring separately.
It's Open Source
I originally built this plugin for my own Discourse forum, SiteTalk, and it's already running in production there.
The project is independently developed, not an official Discourse plugin.
It's available under the MIT license, with automated tests and CI configured for development.
If you're running a Discourse community and want automatic IndexNow submissions, feel free to check it out:
GitHub: https://github.com/imlotso/discourse-indexnow
Issues, suggestions, and contributions are welcome.
One Question for Other Forum Owners
I'm curious how other people approach search engine discovery for community-generated content.
Do you mostly rely on XML sitemaps and normal crawling, or have you tried IndexNow or another push-based solution?
And if you've used IndexNow on a forum, have you noticed any measurable improvement in discovery or indexing times?
I'd be interested to hear what has worked for you.
Top comments (1)
Anyone here running a Discourse forum? 👀
I built this mostly because I got tired of waiting for search engines to discover new posts.
Curious if anyone's tried IndexNow on their own site. Did it actually make a difference, or do you just stick with sitemaps?
Also, if you check out the plugin and think something's missing, let me know. Always looking for ideas!