DEV Community: kavela

Turning World Bank Data Into 50K+ Searchable Pages with WordPress

kavela — Tue, 24 Mar 2026 11:28:45 +0000

What if you could make decades of World Bank and IMF economic data actually accessible and browsable - not buried in spreadsheets and PDF reports that nobody reads?

That's what we built with historysaid.com: a programmatic SEO site that transforms raw international development data into 50,000+ structured, searchable pages. Every country, every indicator, every year - all queryable, all browsable, all indexed by Google.

This post covers the architectural thinking behind it and what we learned building it.

The Data Problem

The World Bank and IMF publish some of the richest economic datasets on the planet:

GDP, inflation, trade balances, debt levels for 200+ countries
Time series spanning 60+ years (some indicators go back to the 1960s)
Hundreds of unique economic indicators covering everything from agricultural output to internet penetration rates
Regular updates as new data gets published quarterly or annually

But the official portals are designed for researchers and economists who already know what they're looking for. You need to know specific indicator codes and use clunky query builders to extract data into spreadsheets.

There's no way to just... explore. To browse. To stumble upon interesting economic stories by clicking around.

We wanted to change that.

Why Not Just Build a Dashboard?

We considered building a single-page dashboard app with interactive charts and filters. But dashboards have a fundamental SEO problem: they're one URL. Google can't index the state of your filters. If someone searches "Turkey GDP growth history", a dashboard app won't rank because that specific view doesn't have its own URL.

Programmatic SEO solves this. Each unique combination of country + indicator gets its own page, its own URL, its own title, and its own meta description. Google can index all 50K of them.

We chose WordPress for the same reasons we used it for startup-cost.com (see our previous post): cheap hosting, familiar ecosystem, and a powerful rewrite engine that nobody uses to its full potential.

The Architecture - Overview

We built a custom WordPress plugin that handles everything from data ingestion to page rendering.

Data Pipeline

The data flows through several stages:

World Bank API --> Fetch & Parse --> Validate --> Normalize --> MySQL
IMF Data Portal --> Fetch & Parse --> Validate --> Normalize --> MySQL
                                                                  |
MySQL --> Virtual URL Routing --> Template Engine --> HTML Page

Each data source has its own ingestion logic because the API formats differ significantly. The World Bank provides a well-documented REST API with JSON responses, while IMF data comes in a different structure. We wrote adapters that normalize both into a common internal format.

The pipeline runs on a scheduled basis. When new data is published by either source, our next run picks it up automatically and updates the relevant records.

Data Quality Challenges

Working with international economic data is messier than you'd expect:

Missing values everywhere - Some countries don't report certain indicators for certain years. We handle nulls gracefully rather than showing zeros.
Delayed reporting - Some nations publish data 2-3 years late. Our pages show the most recent available data and clearly indicate the time period.
Unit inconsistency - Some values are in current USD, some in constant USD, some in percentages. Each indicator carries its unit metadata.
Country code mismatches - The World Bank uses ISO 3166-1 alpha-3 codes, the IMF sometimes uses its own codes. Our normalization layer handles the mapping.

Database Design Principles

We use custom MySQL tables (not wp_posts) following the same pattern from our startup-cost.com engine. The key design decisions:

Proper normalization - Countries, indicators, and data points are separate tables with foreign key relationships
Appropriate data types - We use high-precision decimal types for economic values because the data ranges from tiny percentages to trillion-dollar GDP figures. Floating point would introduce precision errors that data-savvy users would notice.
Strategic indexing - Our most common query patterns (all data for a country+indicator, all countries for an indicator+year) each have compound indexes that resolve in single-digit milliseconds
Roughly 4 million data points in the main table, all queryable in under 10ms thanks to proper indexing

Page Types

Our routing creates four types of pages:

Country pages - Overview of all available indicators for a country. Shows key stats, latest values, and links to explore each indicator in depth.
Country + Indicator pages - The core of the site. Detailed time series data with charts, data tables, summary statistics, and trend analysis. This is the bulk of our 50K pages.
Indicator pages - Global comparison view. Shows all countries ranked by a specific indicator, with the ability to see how they compare.
Comparison pages - Side-by-side country comparisons for a given indicator. Perfect for searches like "Japan vs South Korea GDP."

Zero rows in wp_posts. Everything is computed from data tables on each request (with caching for popular pages).

Charts and Data Display

Each data page includes an interactive chart (using a lightweight client-side charting library) and a full data table. The chart data is embedded as JSON in the page - fast, cacheable, and SEO-friendly since the actual values are also present in the HTML table.

We also calculate and display summary statistics: latest value, historical min/max, average, and trend direction. These make each page genuinely informative rather than just a raw data dump.

SEO Strategy

Every page gets unique, data-driven SEO elements:

Dynamic titles that include the country name and indicator name
Meta descriptions that include actual data values ("Turkey's GDP was $X in 2024. Explore the full trend from 1960 to 2024...")
Schema.org Dataset markup so Google understands these are data pages with temporal and spatial coverage
Breadcrumb navigation for clear site hierarchy

Each meta description contains real numbers from the data, making every page genuinely unique in Google's eyes.

Caching

With 50K+ pages, not everything can be pre-cached. We use a tiered approach:

Popular combinations (major countries + major indicators) get pre-cached with longer TTLs
Medium-traffic pages are cached on demand
Long-tail pages have shorter TTLs and are generated fresh when needed

We also cache aggregated data (like country rankings and regional averages) at the query level since multiple pages reference the same aggregations.

Internal Linking

Strong internal linking is essential for a site this large. Without it, search engines would never discover most of the pages:

Each country page links to all available indicators for that country
Each indicator page links to top countries for that indicator
Breadcrumbs on every page create clear hierarchy
Related content suggestions based on geographic and thematic proximity
Comparison links suggest relevant country pairs

The internal link graph ensures that any page on the site is reachable within 3-4 clicks from the homepage.

Results After 12 Months

50,000+ pages indexed in Google Search Console
Average TTFB: ~150ms on shared hosting
Database queries consistently under 10ms
Zero editorial work - the site runs itself, updated automatically from source data
Growing organic traffic from long-tail searches like "Nigeria inflation rate 2015" or "Vietnam GDP per capita history"

The Reusable Pattern

This is the same architectural pattern we use across multiple sites at Kavela:

Find an interesting, structured dataset
Design custom tables optimized for the specific data model
Build a data pipeline that keeps the database fresh
Use WordPress virtual routing to create SEO-friendly URLs
Render pages dynamically from the data
Generate chunked sitemaps and build strong internal links

The key is making sure every page offers genuine value - real data, real calculations, real insights. Template spam with swapped-out city names won't work. Search engines are smart enough to detect that. But if every page genuinely answers a different question with different data, you've built something valuable.

Explore it yourself: historysaid.com

Built by Kavela Ltd - turning data into discoverable web experiences.

How We Built a Programmatic SEO Engine Serving 80K+ Pages on WordPress (Without Using wp_posts)

kavela — Tue, 24 Mar 2026 11:28:20 +0000

When we set out to build startup-cost.com, we knew traditional WordPress wouldn't cut it. We needed to serve 79,000+ unique pages - one for every combination of 479 cities and 167 business types - with real cost data, real-time calculations, and solid performance.

Most people hear "80K pages on WordPress" and assume we're crazy. WordPress is a blogging platform, right? Well, yes - but under the hood it's a flexible PHP framework with a powerful rewrite engine. We just had to throw away the parts that don't scale and build our own.

Here's the story of how we did it without a single row in wp_posts.

The Problem with wp_posts at Scale

WordPress stores all content in a single table called wp_posts. For a blog or a small business site with a few hundred pages, this works fine. But when you start pushing tens of thousands of rows into that table, things fall apart quickly:

Query performance degrades - WordPress joins wp_posts with wp_postmeta for almost every query. With 80K posts, each with 10+ meta fields, you're looking at 800K+ rows in postmeta alone. Queries that used to take 5ms now take 500ms.
The admin panel becomes unusable - Try loading the "All Posts" screen with 80K entries. WordPress paginates, sure, but even counting the total takes forever.
Revision history eats disk - WordPress auto-saves revisions. With programmatic content that gets regenerated, you end up with 3-4x the actual content in revision rows.
XML sitemaps choke - Popular sitemap plugins try to query all posts at once. With 80K rows, they either timeout or consume all available memory.
Imports and exports break - WordPress export generates a single XML file. Good luck with an 80K-post WXR file.

We needed a fundamentally different approach.

Why Not Use a Static Site Generator or a Different CMS?

Fair question. We considered Hugo, Next.js, and even a custom Node.js app. But WordPress gave us specific advantages:

Hosting is dirt cheap - Shared WordPress hosting costs a few dollars a month and handles our traffic fine
The plugin ecosystem - We still use various utility plugins for SEO settings, caching, and other tasks
Familiar deployment - Our team knows WordPress inside out. No learning curve, no new CI/CD pipeline needed
PHP is actually fast enough - With opcache and a clean query pattern, PHP 8 serves pages in double-digit milliseconds

The key insight was: we don't have to use WordPress the way it was designed. We can use it as a routing and rendering framework while storing our data however we want.

The Architecture - High Level

We built a custom WordPress plugin that bypasses wp_posts entirely. The concept has three pillars:

1. Custom Database Tables

Instead of stuffing everything into posts and postmeta, we created dedicated tables with proper schemas, data types, and indexes. Think of it as designing a mini-application database that lives inside WordPress's MySQL instance.

The key principle: each entity type (cities, business types, cost metrics) gets its own table with columns that match the actual data model - not the generic key-value pairs that postmeta forces you into. This means proper indexing, proper normalization, and queries that hit exactly the data they need.

The performance difference is massive. A meta-based lookup on 80K posts might take hundreds of milliseconds. A direct indexed query on a purpose-built table returns in single-digit milliseconds.

2. Virtual URL Routing

WordPress has a powerful but underused rewrite API. Most developers only interact with it through the permalink settings screen. But under the hood, you can register completely custom URL patterns that map to your own rendering logic.

We define URL patterns that WordPress recognizes and routes to our plugin. When a request comes in, WordPress matches the URL against our patterns, extracts the relevant slugs, and hands control to our code. No post is created. No database row in wp_posts is touched. The URL exists purely because we told WordPress to recognize the pattern.

This approach gives us full control over URL structure while still benefiting from WordPress's request lifecycle, caching hooks, and plugin compatibility.

3. Dynamic Content Generation

This is what makes programmatic SEO actually work. We don't just slap the same text on 80K pages with city names swapped - each page has genuinely different data. Real cost figures, real calculations, real comparisons.

The rendering layer pulls data from our custom tables, runs computations specific to each city-business combination, and outputs a fully formed HTML page with all the SEO elements (unique title, meta description, schema markup, etc.).

Every page displays different numbers because the underlying data is different. Google is smart enough to detect thin, templated content. Genuine value on each page is what makes programmatic SEO work long-term.

Performance Results

We benchmarked both approaches on the same server (shared hosting, PHP 8.1, MariaDB 10.6):

Metric	wp_posts approach	Our custom approach
Average page load (TTFB)	~800ms	~120ms
Database query time	~200ms (with meta joins)	~15ms (indexed lookup)
Admin dashboard load	30+ seconds	Instant (no admin overhead)
Sitemap generation	Timeout at 60s	~2 seconds
Memory usage per request	~64MB	~12MB

The difference is dramatic. Custom tables with proper indexes make WordPress perform like a purpose-built application.

Sitemap Strategy

Google's sitemap limit is 50,000 URLs per file and 50MB uncompressed. With 80K pages, we need multiple sitemaps plus a sitemap index.

We generate sitemaps programmatically, chunking URLs into manageable files, and create a sitemap index that references all of them. The generation runs on a weekly cron. Google Search Console picks them up without issues, and we can track indexing progress per sitemap chunk.

Internal Linking

With 80K pages, internal linking is critical both for SEO and for helping users navigate:

Each city page links to all business types available in that city
Each business page links to top cities for that business type
Each detail page links to related pages based on geographic and thematic proximity
Dense internal link graph helps search engines discover and understand the site structure

Caching

We use a multi-tier caching strategy. Not all 80K pages get equal traffic - popular city/business combinations are pre-cached with longer TTLs, while long-tail pages are cached on demand with shorter TTLs. We also cache at the database query level for frequently accessed aggregations.

What We Learned

WordPress can handle massive scale - but only if you step outside its default content model. The platform is more flexible than people give it credit for.
Custom tables are not "hacky" - They're the right tool when your data doesn't fit the post/meta pattern. WordPress itself uses custom tables for comments, users, and options.
Virtual routing is powerful - WordPress rewrite rules can handle complex URL patterns. You don't need a custom framework just because your URLs don't map to posts.
Programmatic SEO needs real value - Google detects thin, templated content. Every page needs genuinely different, useful data. That's the difference between spam and a real product.
Start with the data model - We spent more time designing our database schema than writing rendering code. Good data modeling pays off at every layer.
Monitor, then optimize - We started without caching and added it only where monitoring showed bottlenecks. Premature optimization would have added complexity we didn't need.

Check out the result: startup-cost.com - startup cost estimates for 167 business types across 479 cities worldwide.

Built by Kavela Ltd - we build data-driven web tools at scale.