Moving 530k Jobs from Supabase to Postgres on a Free Oracle VM (14s -> 47ms)

#postgres #apify #career #sideprojects

I built a job feed. It pulls jobs straight from company career pages into one searchable list. No login, no redirects through job boards.

In a few months it grew from 10,000 jobs to over 530,000. Every time the number jumped, something broke. Here's what broke and how I fixed it.

Career pages, then ATS platforms

I started by writing a scraper for each company's career page. One company, one scraper. Slow.

Then I noticed most companies don't build their own job board. They use an applicant tracking system (ATS) like Greenhouse, Lever or Ashby, and these have public job data. One integration covered thousands of companies. The count went 10.5k → 16.5k → 100k → 300k → 530k, and it's still growing.

GitHub as the database

At first, GitHub was my database. A scheduled GitHub Action fetched jobs, saved one JSON file per company, and committed it. The site read those files. Free, versioned, nothing to manage.

Up to around 100k jobs, it was fast.

The problem: a browser can only filter data it already has. So the first load had to download everything. As data grew, first paint went to 13–15 seconds. Filters were instant after that, but nobody waits 15 seconds to see them.

Moving to Supabase

So I moved to Postgres on Supabase. The GitHub Action synced data into it every night. The page loaded 50 jobs at a time, and filters became SQL queries.

First load got better. But every filter now took 5–7 seconds. Then at 530k jobs, the database crossed the 500 MB free limit.

When I checked, two queries were the real problem: counting all jobs, and grouping jobs by company for the filter list. Each took about 14 seconds. That's not Supabase's fault. A small shared free instance just isn't made to scan half a million rows on every request.

Getting a free Oracle server

Oracle Cloud's free tier gives you an ARM server with up to 2 cores and 12 GB RAM. For a 532 MB database, that's more than enough.

But it's hard to get. Every time I clicked Create, I got:

Out of host capacity

Instead of refreshing all day, I wrote a script. It calls the Oracle CLI to create the server, and if the error is "no capacity" or a rate limit, it waits and tries again. A GitHub Action ran it every few minutes.

for i in 1 2 3 4 5; do
  timeout 60 oci --no-retry compute instance launch ... && break
  sleep 20
done

It failed 124 times overnight. The 125th try worked, and I woke up to a free 12 GB server in Mumbai.

Setting up Postgres

The next morning, it took me about 3–4 hours to get Postgres running on the new VM and start syncing the data.

I ran one or two of the slow queries on it, and that was enough to convince me. I quickly pointed Vercel and my Cloudflare Worker (which serves my public API) to the new database and tested the site.

The results were mind-blowing.

The result

Same queries, run directly on each database:

Query	Supabase (free)	Oracle VM
Count all jobs	13.7 s	26 ms
Group jobs by company	13.9 s	47 ms
Keyword search	177 ms	5.5 ms
First page of results	22 ms	0.3 ms

In the browser, pages that took 16–48 seconds now load in under a quarter second.

To be fair, a lot of this is hardware. The whole database now fits in memory on a 12 GB machine, while the free instance was small and shared. Same Postgres, just more room.

What I learned

Find the slow query before blaming the platform. Two queries caused almost everything.
Check what a free tier actually limits. Mine capped storage, but compute is what hurt.
If you have to wait for something, let a script do the waiting.

You can try the job-feed here (non-profit): link.
If you want the data in your own app, it's also on Apify as an API: link.