DEV Community: Vincent

I Built an Open-Source SEO Indexing Infrastructure with Rust

Vincent — Fri, 28 Aug 2026 02:37:38 +0000

A website being online does not mean search engines understand it

As developers, we usually think:

If my website is deployed and users can access it, the job is done.

But after building content-heavy websites, I realized there is another layer that developers often overlook:

How do search engines discover, crawl, and understand my website?

After deployment, many invisible problems can appear:

Are pages returning the correct HTTP status codes?
Is the sitemap valid?
Are URLs structured correctly?
Are there unexpected 404 pages?
Are search engines still crawling the website?
Which URLs are indexed and which are not?

These problems usually do not appear during local development.

They only appear after your website is live.

The problem I experienced

While building my own content website, I encountered unexpected indexing issues.

At first, everything seemed fine.

The website was running.
The pages were accessible.
The application worked.

However, search engine crawling behavior changed after some deployment and website structure changes.

Lesson 1: Infrastructure stability affects search engine crawling

The website was initially deployed on Cloudflare Workers.

Because of runtime limitations, some requests occasionally resulted in 5xx errors.

For normal users, an occasional failure may only be a small inconvenience.

But for search engines, repeated failures can affect how they evaluate a website.

A website that frequently fails to respond correctly is harder for search engines to crawl reliably.

Eventually, I noticed that search engine crawling activity decreased significantly.

This made me realize:

Website reliability is not only a user experience problem. It is also a search engine visibility problem.

Lesson 2: URL changes are more dangerous than they look

Later, I changed parts of my URL structure to make it cleaner and more SEO-friendly.

However, changing URLs is not just a simple string replacement.

It affects:

redirects
sitemap generation
URL discovery
indexing status
search engine understanding

Without proper monitoring, it is difficult to know whether these changes are helping or hurting.

SEO starts before keywords

When people talk about SEO, they often think about:

keyword research
ranking optimization
backlinks
content strategy

These are important.

But for developers, there is a more fundamental question:

Can search engines correctly access and understand my website?

Before optimizing keywords, we should make sure:

pages are accessible
URLs can be discovered
technical SEO issues are detected
website structure is healthy

Building IndexFlow

This experience led me to build IndexFlow-core.

IndexFlow-core is an open-source search engine indexing infrastructure designed for developers.

The goal is not to replace SEO professionals.

Instead, it focuses on the technical foundation between websites and search engines.

Current focus areas:

Sitemap management
URL discovery
SEO quality checks
Indexing workflow management
Search engine submission infrastructure

GitHub:

https://github.com/IndexFlowing/IndexFlow-core

Website:

https://www.indexflowing.com

Why Rust?

I wanted IndexFlow to be:

lightweight
self-hostable
reliable
suitable for long-running services

Rust fits this kind of infrastructure project very well.

During development, I also extracted some reusable components into standalone Rust libraries.

The idea is simple:

If a component can be useful outside IndexFlow, it should not be locked inside the project.

Open-source direction

IndexFlow-core is moving toward an open-source search engine indexing infrastructure.

The long-term vision is to provide developers with better tools for managing the relationship between their websites and search engines.

Future improvements include:

better SEO quality gates
smarter URL scheduling
multi-site management
improved indexing monitoring

Final thoughts

IndexFlow was not created because I wanted to build another SEO tool.

It started because I encountered a real problem while building websites.

Deployment issues, URL changes, and indexing problems made me realize:

Search engine visibility also needs infrastructure.

That is why I started building IndexFlow.

If you are interested in technical SEO, Rust, or open-source infrastructure, I would love to hear your feedback.

Thanks for reading.

How I built a high-performance Chinese learning platform with 100k+ TV clips using Rust and Next.js 15

Vincent — Mon, 20 Jul 2026 02:44:30 +0000

Hi dev.to community! 👋

Today, I’m incredibly excited to share MandarinClips, a side project I’ve been pouring my soul into. It's a high-performance language learning platform that indexes and organizes sentence-sized dialogue segments from authentic Chinese movies and TV series to help language learners transition from sterile textbooks to real-world immersion.

👉 Check it out here: MandarinClips.com

Here is a deep dive into the engineering decisions, performance bottlenecks, and the technical architecture that powers the platform under the hood.

🏗️ The High-Level Architecture

I wanted the platform to be blindingly fast, cost-effective, and highly available. Here is the stack:

Frontend: Next.js 15 (React 19) + Tailwind CSS + next-intl (for multilingual support).
Backend: Rust (Axum framework) + Tokio async runtime.
Database: PostgreSQL with pg_trgm (trigram) extension.
Storage & CDN: Backblaze B2 proxied through Cloudflare CDN (S3 API compatible, 0-cost egress).

⚡ Backend: Why Rust & Axum?

The backend is responsible for high-frequency search requests, serving sitemaps, and tracking user quota usage.

Using Rust was a no-brainer for several reasons:

Safety & Zero-Cost Abstractions: No garbage collection means zero GC pauses, resulting in extremely predictable latency.
Axum & Tokio: Axum is a highly ergonomic, thin routing wrapper on top of Tokio. It handles asynchronous tasks with minimal CPU overhead, allowing our single 2-core VPS to handle thousands of concurrent queries without breaking a sweat.

🗄️ Database: Sub-millisecond Fuzzy Search with PG GIN Indexes

With over 100,000 video clips and subtitles indexed, a standard ILIKE '%query%' search would quickly degrade and result in full table scans, locking up the CPU under high traffic.

To solve this, we utilized the PostgreSQL pg_trgm (trigram) extension to build GIN (Generalized Inverted Index) on both Chinese characters and no-tone pinyin strings.

CREATE EXTENSION IF NOT EXISTS pg_trgm;

CREATE INDEX idx_clips_text_zh_trgm 
ON clips USING GIN (text_zh gin_trgm_ops);

CREATE INDEX idx_clips_pinyin_search_trgm 
ON clips USING GIN (pinyin_search gin_trgm_ops);

By querying against these indices, fuzzy matches for queries like "ni hao" or "明天见" take under 1 millisecond locally. We achieve pure O(1) list scans.
🎥 Media Delivery: 0-Cost Egress with Backblaze B2 & Cloudflare
Hosting over 100,000 MP4 video clips can quickly become a bandwidth billing nightmare on standard cloud providers.
We solved this by storing all .mp4 video files and .jpg posters in Backblaze B2, which is part of the Cloudflare Bandwidth Alliance. By proxying the B2 bucket through Cloudflare's global edge network:
All public egress bandwidth is $0.
We configured aggressive Cloudflare Cache Rules (using Cache-Control: public, max-age=86400, s-maxage=604800 headers) so that sitemaps and video assets are served straight from Cloudflare's edge memory, dropping origin VPS bandwidth and IOPS to near-zero.
🌐 Frontend: Next.js 15 & Native 308 Permanent Redirections
For the frontend, we used Next.js 15. One of our recent refactoring milestones was migrating legacy URLs from underscores (_) to SEO-friendly dashes (-).
To preserve SEO link juice, we avoided heavy JS-based Edge middleware and leveraged Next.js's native redirects config in next.config.ts to issue 308 Permanent Redirects at the router level:
// next.config.ts
async redirects() {
return [
{
source: '/sitemap.xml',
destination: 'https://www.mandarinclips.com/sitemaps/sitemap.xml',
permanent: true,
}
];
}
This is processed natively before any middleware execution, ensuring zero-latency transitions for Googlebot.
🎁 Wrapping Up & Feedback
Building MandarinClips has been an incredible full-stack journey.
I’d love to hear your feedback on the architecture, the Rust Axum setup, or the database indexing! If you're currently learning Chinese (or are just curious about the UI), please check out MandarinClips and let me know your thoughts in the comments below!
Happy hacking! 🚀