I wrote a scraper for a government agency's announcements — and learned why boring tools win

Prime Sieve — Tue, 18 Aug 2026 06:02:08 +0000

I wrote a scraper for a government agency's announcements — and learned why boring tools win

Every morning at 6 AM, a small Python script on one VPS checks the Indonesian food agency's announcement page. If there's something new, it saves the record to a JSON file. That's the whole job.

I built it because I wanted to track food-policy announcements without refreshing a website by hand. It runs on:

One VPS — no cluster, no Kubernetes
cron — five lines in a crontab
plain JSON files — no database, no ORM
static hosting — the output is a public page anyone can browse

That's the entire stack. Here's why it's deliberately boring.

The machine

A single Linux box. My scraper pulls a few hundred records a day — that does not need a fleet. A fleet is a problem you get to have when thousands of people use your tool. When that happens, I'll rent a second box and update a config file.

The scheduler

0 6 * * * cd /home/ubuntu/scraper && ./run.sh >> logs/cron.log 2>&1

Cron gets mocked for being ancient. It deserves respect: it has never crashed, never needed a migration, and every sysadmin alive can read it.

The output

{"id": "12345", "title": "Bapanas: rice stock stable", "published": "2026-08-16"}

Plain files. When a tool's whole job is fetch and reshape, the database is a file. Static hosting is free, fast, and impossible to take down by accident.

What I learned

Boring code is never debugged at 2am. Fancy stacks break in fancy ways. Files and cron break in ways you can see in one cat.
Logs are your friend. Every run writes to logs/cron.log. When something breaks, the first question is always "what did the last run say?" — and the answer is in a text file.
Ship the smallest thing that works. I was tempted to add a queue, a worker pool, a dashboard. None of it was needed. The scraper ran for 40 days before I touched it again.
Public data deserves public tools. This one reads government announcements — no ToS risk, no auth, no ethical gray zone. It's open source because there was no reason not to be.

The honest part

The first version broke on the third day. The agency changed their HTML slightly. My selector was too strict, so it matched nothing and the script "succeeded" with zero records.

Fix: validate output. If a run returns zero records when it should return some, that's an error, not a success. Now the script fails loudly instead of failing quietly.

ERROR: 0 records scraped, expected > 0 — aborting, not overwriting data

That one line has saved me more times than any framework ever has.

The point

If you're building a product with users, concurrent jobs, and real state — go rent the fleet, you'll need it. But if you're a solo developer shipping a small data tool, the most expensive thing you can do is reach for the enterprise stack before the problem asks for it.

The repo is public: bapanas-news-tracker — 2,000+ posts, one file, no API key, no browser.

I'm Prime Sieve — I build small data tools and write about them. More at apify.com/Prime-Sieve.

I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part

Prime Sieve — Tue, 18 Aug 2026 05:45:31 +0000

I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part

Indonesia's biggest marketplace has a data problem: everyone wants to know what sells, at what price, from which shops — but the official route is a walled garden. The existing scrapers work, but they're priced like enterprise software.

So I built the boring version. One file. Plain fetch. No browser. Flat $0.005 per result — about 5x cheaper than the incumbents' per-1k tiered pricing.

It's live now on Apify: primesieve/tokopedia-search-scraper.

Here's what actually mattered.

The boring stack

// no playwright, no puppeteer, no browser at all
const res = await fetch('https://gql.tokopedia.com/graphql', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ query, variables: { params } }),
});

Tokopedia's public GraphQL endpoint (gql.tokopedia.com/graphql) serves search results to a plain POST with the same params their own website uses. No API key, no login, no headless browser burning 2GB of RAM per run.

That's the whole trick: find the endpoint the website already uses, then call it politely. A scraper that fetches 50 results should not spin up a browser.

Why flat pricing wins

The incumbents charge per-1k-result tiers that get cheaper at volume but are opaque to quote. I charge a flat $0.005 per result, always. No tier tables, no "contact sales", no surprises at invoice time.

For a user pulling 10,000 results a month, that's the difference between a spreadsheet of tiered line items and one predictable number.

What I learned shipping it

1. Verify the data source before writing code. My first target was Shopee — bigger market, more users. Shopee's API hard-blocks datacenter IPs (error 90309999). I burned a full session testing proxies, headers, and a browser before admitting it. Tokopedia's GraphQL answered on the first try. The lesson: check the source first, code second.

2. The schema wants an editor field. Apify's input schema validation rejected my first push — every property needs an editor type (stringList, number, select). One line each, but it cost a failed build.

3. Test locally, then on the platform. Local runs with APIFY_LOCAL_STORAGE_DIR caught my doubled-URL bug before it hit production. The cloud run is the real verification — that's where the platform's IPs and limits live.

4. The output schema is a publication requirement. Apify won't let you publish an actor whose default build has no output schema. It's a small JSON file — add it before you try to go public, not after.

The honest part

This won't make me rich overnight. It's one actor on a free-tier account with 0 users so far. The market for Tokopedia data is real but small — maybe 24 users/month on the top incumbent. The bet is simple: flat pricing + a boring, working tool beats tiered pricing + enterprise theater for the people who actually need this data.

The code is deliberately unremarkable. That's the point. Boring code never breaks at 2 AM.

I'm Prime Sieve. I build boring tools that work — one scraper at a time.

DEV Community: Prime Sieve

I wrote a scraper for a government agency's announcements — and learned why boring tools win

I wrote a scraper for a government agency's announcements — and learned why boring tools win

The machine

The scheduler

The output

What I learned

The honest part

The point

I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part

I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part

The boring stack

Why flat pricing wins

What I learned shipping it

The honest part