DEV Community: IMLotso

Automating Discourse SEO with IndexNow

IMLotso — Tue, 20 Oct 2026 03:46:55 +0000

I Built a Discourse Plugin to Automatically Submit New Topics to Search Engines

If you run a Discourse forum, you've probably experienced this: someone publishes a new topic, but it takes a while before search engines even discover the page.

Discourse already has sitemaps, and they work fine. But I wanted something a little more proactive.

Instead of waiting for a search engine to crawl my forum, why not tell it whenever something changes?

That's what got me interested in IndexNow.

I ended up building an open-source Discourse plugin called discourse-indexnow to automate the whole process.

The Problem with Waiting for Crawlers

Discourse forums generate a lot of dynamic content.

People create new topics, edit posts, reply to discussions, and sometimes delete old content.

Traditional XML sitemaps help search engines discover URLs, but they don't immediately notify search engines when something changes.

For smaller forums, especially newer communities, this can be frustrating.

You might publish a useful technical guide today, but have no idea when a search engine will discover it.

I wanted a way to reduce that delay without manually submitting URLs every time I published something.

So, What Is IndexNow?

IndexNow is an open protocol that lets websites notify participating search engines when URLs are added, updated, or deleted.

Instead of relying entirely on periodic crawling, your website can send a notification when something changes.

For example, an IndexNow batch request looks something like this:

{
  "host": "forum.example.com",
  "key": "your-indexnow-api-key",
  "urlList": [
    "https://forum.example.com/t/example-topic/123",
    "https://forum.example.com/t/another-topic/456"
  ]
}

Search engines such as Bing and Yandex support the protocol.

One important thing to understand: IndexNow doesn't guarantee indexing or better rankings.

It simply helps participating search engines discover changes sooner.

Also, Google doesn't currently support IndexNow, so you should still maintain your sitemap and existing SEO practices.

Building discourse-indexnow

I wanted the integration to work directly inside Discourse rather than relying on an external cron job or a separate automation service.

The basic idea was simple:

When something happens to a public topic, the plugin determines whether its URL should be submitted and then handles the IndexNow notification in the background.

But once I started thinking about real forum activity, there were a few more things to consider.

1. Automatically submit new and updated topics

This was the main feature I wanted.

When someone creates a public topic, the plugin automatically submits its URL.

When the first post is edited or important topic attributes change, the URL can be submitted again.

It also handles deleted topics, allowing participating search engines to discover that the original URL may no longer be available.

No need to copy URLs or manually open a webmaster dashboard.

2. Handle replies and paginated topics

Here's something that makes forums a little different from regular blogs.

A discussion can grow across multiple pages.

If a new reply appears on page three, submitting only the main topic URL might not be the most useful approach.

So I added optional reply-triggered submissions.

The plugin can submit the URL of the page containing the new reply.

To avoid flooding IndexNow with notifications on busy discussions, there's also a configurable cooldown for each URL.

3. Support multilingual topics

Discourse supports content localization, and I wanted the plugin to work with it.

If a topic has localized versions available to crawlers, the plugin can submit the relevant localized URLs alongside the original.

For example:

https://forum.example.com/t/example/123
https://forum.example.com/t/example/123?tl=es
https://forum.example.com/t/example/123?tl=fr

These are illustrative URLs; the plugin uses the locale parameters actually configured in Discourse.

Rather than making a separate request for every language, the URLs can be grouped into one batch.

This was especially useful for making the integration work with multilingual communities.

4. Don't accidentally submit private content

This was one of the most important design considerations.

A forum isn't just a collection of public blog posts.

Discourse can contain private messages, restricted categories, unlisted topics, and other content that shouldn't be sent to search engines.

The plugin checks whether a URL is eligible before submitting it.

It also supports additional category and tag exclusions.

The goal is to make automatic submission useful without exposing URLs that should remain private.

5. Add rate limiting and retry handling

Automating URL submission sounds easy until you think about what happens on a busy forum.

Imagine hundreds of topics being updated in a short period.

You don't want every event generating unlimited API requests.

So I added:

Hourly and daily submission limits
Per-URL cooldowns
Batch submissions
Handling for HTTP 429 responses
Retry delays based on the API response

The plugin also splits large batches to respect IndexNow's 10,000-URL-per-request limit.

All of this happens asynchronously, so submissions don't need to block normal forum interactions.

An Admin Dashboard Makes Things Easier

I also wanted to know whether the integration was actually working.

Just sending API requests and hoping for the best didn't seem like a great solution.

So the plugin includes an admin interface that displays:

Successful and failed submissions
Submission history and trigger reasons
Hourly and daily quota usage
Seven-day activity trends
Error categories
API key verification status

It also supports manual URL submission and historical backfills.

For example, if you install the plugin on a forum that already has hundreds of public topics, you can select a category and date range, preview the matching topics, and submit them in batches.

That's much more convenient than submitting existing URLs one at a time.

How to Install It

The plugin is open source and available on GitHub.

Repository: imlotso/discourse-indexnow

After installing the plugin using the standard Discourse plugin installation process and rebuilding your container, open:

Admin > Plugins > discourse-indexnow

From there, generate an IndexNow API key and enable submissions.

You'll also need to verify that the key file is publicly accessible:

https://your-forum-domain.com/<your-key>.txt

The endpoint should return HTTP 200 and the key as its response body.

The plugin provides a built-in check for this.

I've also written a more detailed guide covering the plugin's features, configuration options, URL filtering, and troubleshooting.

👉 Discourse IndexNow Plugin — Complete Setup Guide and Feature Overview

That guide includes the configuration reference and explains how each submission mechanism works.

Does This Actually Improve SEO?

This is probably the most important question.

IndexNow can help supported search engines learn about new or updated content faster.

But notifying a search engine isn't the same thing as getting indexed.

And getting indexed isn't the same thing as ranking well.

Content quality, crawlability, internal linking, canonical URLs, and many other SEO factors still matter.

I see IndexNow as an additional discovery mechanism rather than a replacement for traditional SEO.

For an active Discourse community, reducing the time between publishing content and notifying search engines seems like a useful improvement.

Whether that translates into better organic search performance is something worth measuring separately.

It's Open Source

I originally built this plugin for my own Discourse forum, SiteTalk, and it's already running in production there.

The project is independently developed, not an official Discourse plugin.

It's available under the MIT license, with automated tests and CI configured for development.

If you're running a Discourse community and want automatic IndexNow submissions, feel free to check it out:

GitHub: https://github.com/imlotso/discourse-indexnow

Issues, suggestions, and contributions are welcome.

One Question for Other Forum Owners

I'm curious how other people approach search engine discovery for community-generated content.

Do you mostly rely on XML sitemaps and normal crawling, or have you tried IndexNow or another push-based solution?

And if you've used IndexNow on a forum, have you noticed any measurable improvement in discovery or indexing times?

I'd be interested to hear what has worked for you.

How to Automatically Sync SSL Certificates from 1Panel to SafeLine WAF

IMLotso — Sun, 18 Oct 2026 09:01:31 +0000

Managing SSL certificates across multiple services can quickly become a repetitive maintenance task, especially when your web infrastructure includes a hosting control panel, a reverse proxy, and a Web Application Firewall (WAF).

If you're using 1Panel to manage SSL certificates and SafeLine WAF to protect your websites, you've probably encountered a common issue: certificates renew automatically in 1Panel, but the WAF may still be serving an older certificate.

Fortunately, this process can be automated using Python and SafeLine's OpenAPI.

The Problem: SSL Renewal Doesn't Always Mean SSL Deployment

Let's consider a typical self-hosted infrastructure:

1Panel manages websites and automatically renews SSL certificates.
SafeLine WAF filters malicious traffic before requests reach your applications.
Let's Encrypt provides free SSL certificates.
A scheduled task handles routine infrastructure maintenance.

Even when 1Panel successfully renews a certificate, SafeLine might continue using the previously uploaded version.

This creates an unnecessary operational dependency: someone must manually upload the new certificate to the WAF.

For infrastructure administrators managing multiple domains, this is both inconvenient and error-prone.

A Better Approach: Automate Certificate Synchronization

Instead of manually replacing certificates, we can create a small Python automation that checks whether SafeLine's certificate matches the latest version stored by 1Panel.

The overall workflow is straightforward:

1Panel automatically renews an SSL certificate.
The updated certificate and private key are saved locally.
A Python script reads the certificate files.
The script retrieves the currently configured certificate from SafeLine's API.
Both certificates are compared using their serial numbers.
If they differ, the script updates the certificate through OpenAPI.

This approach avoids unnecessary API updates and eliminates repeated manual uploads.

Why Compare Certificate Serial Numbers?

A simple automation might compare expiration dates or file modification timestamps.

However, these methods are not always reliable.

A certificate's serial number provides a more direct way to identify whether two certificates are the same.

With Python's cryptography library, extracting the serial number is straightforward:

from cryptography import x509

with open("/path/to/fullchain.pem", "rb") as f:
    certificate = x509.load_pem_x509_certificate(f.read())

print(hex(certificate.serial_number))

The script can compare this value against the certificate currently stored in SafeLine.

If the values match, no update is necessary.

If they differ, the certificate can be replaced using the existing certificate ID.

This makes the synchronization process efficient and repeatable.

Using SafeLine's OpenAPI

SafeLine provides certificate management endpoints that can be used for automation.

The relevant API operations include:

Method	Endpoint	Purpose
GET	`/api/open/cert`	List certificates
GET	`/api/open/cert/{id}`	Retrieve certificate details
POST	`/api/open/cert`	Create or update certificates

A Python script can interact with these endpoints using the requests library.

For manually managed certificates, the update request includes the existing certificate ID, certificate content, and private key.

An important detail is to preserve the existing certificate ID rather than creating a new certificate every time.

This prevents unnecessary certificate entries from accumulating in SafeLine.

The automation should also avoid modifying certificates managed internally through ACME.

Full Implementation and Configuration

I documented a working implementation that connects 1Panel certificate renewal with SafeLine WAF certificate management.

The guide includes:

A complete Python synchronization script
Support for multiple domains
Certificate serial-number comparison
SafeLine OpenAPI authentication
Dry-run testing before applying changes
Scheduled execution through 1Panel
Troubleshooting API requests blocked by SafeLine
Security considerations for API tokens

You can find the complete implementation here:

👉 Automating SafeLine WAF SSL Certificate Updates with 1Panel — Full Python Script and Setup Guide

The guide also explains how to handle an issue that may occur when the SafeLine management interface is protected by its own WAF.

Common Issue: HTTP 467 When Calling the API

One interesting challenge with this setup is that SafeLine can block automation requests to its own management interface.

In some configurations, API requests may return HTTP 467 rather than the expected API response.

This typically indicates that a WAF protection rule intercepted the request before it reached the API.

A practical solution is to create a narrowly scoped allow rule:

Restrict the source to the automation server's public IP.
Limit the permitted path to /api/open/.
Avoid creating unrestricted exceptions for the entire management interface.

If a CDN or reverse proxy is involved, ensure SafeLine is configured to identify the real client IP correctly.

Avoid disabling TLS verification as a routine workaround. A properly trusted HTTPS endpoint is preferable.

Scheduling the Automation

Once the script is tested, it can run through 1Panel's scheduled task system.

A weekly schedule may be sufficient for many deployments, provided it runs after certificate renewal and the acceptable deployment delay is understood.

For more demanding environments, an event-triggered workflow or more frequent checks may be appropriate.

A good synchronization script should be idempotent: running it repeatedly should not cause unnecessary certificate changes.

Security Considerations

Certificate automation handles sensitive private keys and privileged API credentials.

Before deploying it to production:

Keep API tokens outside publicly accessible source code.
Restrict configuration file permissions.
Use HTTPS with proper certificate validation.
Limit access to SafeLine's management API.
Avoid logging private keys or authentication tokens.
Test the automation with a dry run before enabling updates.

Automation should reduce operational risk, not introduce new security vulnerabilities.

Final Thoughts

SSL certificate renewal is only one part of certificate lifecycle management.

When multiple services terminate HTTPS connections, renewed certificates must also be propagated to the components that actually serve them.

By combining 1Panel, Python, and SafeLine OpenAPI, you can build a lightweight certificate synchronization workflow without adding another complex infrastructure service.

For the complete working script and detailed configuration steps, see the original 1Panel and SafeLine WAF automation tutorial on SiteTalk.

If you're managing self-hosted infrastructure, this is a useful way to eliminate one more recurring maintenance task.

Why Is My Forum So Quiet?

IMLotso — Tue, 13 Oct 2026 03:17:04 +0000

Hey everyone! This is my first post on DEV 👋

I run a forum where I’d love to get developers and sysadmins chatting, but right now… it’s mostly just me posting 😂

Honestly, it’s not just about traffic. I’d love to have a few regulars who come back, share what they’re working on, ask questions, or just hang out. Instead, it sometimes feels like I’m writing a personal blog with a comment section.

Maybe the topics aren’t clicking, or maybe people already have their favorite places to hang out. I’m still trying to figure it out.

Anyway, I’m hoping to meet some fellow devs and Linux/DevOps folks here, swap stories, and talk about things we’re building—or breaking.

What makes you stick around in a tech community? If you’ve ever started a forum or community, I’d love to hear how it went for you.