DEV Community: Prime Sieve

How I built a Hacker News scraper that pulled 1,000 posts in 10 seconds

Prime Sieve — Fri, 04 Sep 2026 13:36:49 +0000

How I built a Hacker News scraper that pulled 1,000 posts in 10 seconds

Most HN scrapers I've seen over-engineer the problem. Proxy rotation, headless browsers, captcha solving — all to pull data that's already exposed through a free JSON API. Here's the version that takes 10 seconds and costs nothing

The API nobody charges for

Hacker News uses Algolia to power search, and Algolia exposes a public endpoint with no key required

https://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=200

That's it. No auth. No rate limit. Returns clean JSON with title, URL, points, comments, author, timestamp — everything you need to analyze HN

The fetch (no Apify, no SDK, just urllib

`import json
import urllib.request
import time

stories = []
for page in range(10):
url = f"https://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=100&page={page}"
req = urllib.request.Request(url, headers={"User-Agent": "hn-research/1.0"})
with urllib.request.urlopen(req) as r:
data = json.loads(r.read())
stories.extend(data.get("hits", []))
time.sleep(0.3) # be polite

print(f"Pulled {len(stories)} stories")`

Total runtime on my VPS: about 12 seconds for 1,000 posts. The 0.3s sleep is optional but polite — Algolia serves everyone from the same backend

What the data actually says

I ran the script for the last 7 days and bucketed by points

Median points: 1. Half of all HN stories get exactly one upvote
Only 1.5% reach 50 points — that's "visible" by HN's standards
Zero posts crossed 500 points in the entire 7-day window

The top story of the week landed at 104 points (a Jane Street reverse engineering writeup). The HN virality curve is steep. If you're posting there hoping for traction, you're competing in a 1.5% funnel

When the data was posted (UTC

06:00 UTC 5 posts
07:00 UTC 35 posts
08:00 UTC 28 posts
09:00 UTC 27 posts
10:00 UTC 40 posts
11:00 UTC 46 posts
12:00 UTC 19 posts
(other hours: ~0)

Almost everything posts between 07:00 and 12:00 UTC (US morning). Counterintuitively, the best window to post is 06:00–07:00 UTC — you catch the morning traffic spike with less competition

Domain analysis

Packaged as an Apify actor

I wrapped the script as an Apify actor so I can re-run it weekly without rewriting the boilerplate. The actor charges per result (PAY_PER_EVENT), so cost scales with what you actually pull

`// src/main.js — Apify actor (free Algolia endpoint)
import { Actor } from 'apify';

await Actor.main(async () => {
const input = await Actor.getInput() || {};
const queries = input.queries || [];
const tags = input.tags || 'story';
const maxResults = Math.min(500, input.maxResults || 100);

for (const q of queries) {
const url = https://hn.algolia.com/api/v1/search?query=${encodeURIComponent(q)}&tags=${tags}&hitsPerPage=${Math.min(100, maxResults)};
const res = await fetch(url, { headers: { accept: 'application/json' } });
const data = await res.json();
for (const hit of data.hits || []) {
await Actor.pushData({
title: hit.title || hit.story_title,
url: hit.url || hit.story_url,
author: hit.author,
points: hit.points,
comments: hit.num_comments,
createdAt: hit.created_at,
});
try { await Actor.charge({ eventName: 'result', count: 1 }); } catch {}
}
}
});`

The full pipeline

Pull — fetch 1,000 fresh stories (10 seconds
Score — bucket by points, comments, posting hour (Python pandas, 30 seconds
Analyze — extract the patterns above (5 minutes of thinking
Write up — turn the data into a Medium article (the long-form version

Total time: about an hour from API call to published article. Total cost: free (Algolia is free, the actor charges per-result on Apify

Why this matters

You don't need a scraping service to do competitive research on public forums. HN, Reddit, GitHub, Wikipedia, Stack Overflow — all of them have free public APIs or fetch-clean endpoints. The bottleneck isn't data access, it's analysis

The same approach works for any forum where posts are timestamped. The trick is: pick a 7-day window, pull everything, score by engagement, and look for the patterns. You don't need ML — a 50-line script does it

This is part of a broader weekly data routine I run — HN virality, GitHub trends, and (separately) the remote job market. If you like this style of low-cost data analysis, I publish a weekly report on remote hiring signals every Wednesday

Remote Engineering Hiring Signal: What 250+ Scraped Jobs Reveal This Week

Prime Sieve — Wed, 02 Sep 2026 14:54:51 +0000

Remote Engineering Hiring Signal: What 250+ Scraped Jobs Reveal This Week

Every week, our automated scrapers index over 250 remote-first engineering job boards, filtering out ghost listings, low-effort recruiter aggregators, and stale postings to find out who is actually hiring senior developers.

Here is a quick snapshot of this week's data, featured roles, and engineering trends.

📊 This Week's Market Signal

Total Jobs Scanned: 256
High-Signal Listings (Grade A/B): 217
Top In-Demand Stacks: Python, PostgreSQL, React, Node.js, Go, FastAPI, Docker

Query Breakdown

Mobile developer remote: 55 roles
Backend developer remote: 52 roles
DevOps engineer remote: 52 roles
AI / ML engineer remote: 50 roles
Full stack engineer remote: 47 roles

🏆 Featured Roles (Sample from This Week)

Senior Backend Engineer — Nametag (Seattle / Remote)
- Tech: Python, PostgreSQL, REST
- Signal: Identity verification platform scaling backend infrastructure.
Full Stack Developer — Softermii (Los Angeles / Remote)
- Tech: React, Node.js, PostgreSQL
- Signal: Fintech solutions expanding remote engineering team.
Senior GoLang Developer — ImagineX (Atlanta / Remote)
- Tech: Go, PostgreSQL, Docker
- Signal: High-performance backend microservices.

🛠️ Tool of the Week: Donsetch

Donsetch — Keyless web fetch/search/crawl for AI agents. Auto bot-wall bypass, clean markdown output. Perfect for building data pipelines without proxy costs. GitHub →

Get the Full 15-Role Briefing Every Wednesday

This is just a small sample of our weekly briefing. We deliver 5 deep-dive featured roles with direct apply links, plus 10 quick hits and a cold pitch template straight to your inbox every week.

👉 Subscribe to Remote Signal

I Built a Wikipedia Scraper That Runs on Apify Without Proxies

Prime Sieve — Fri, 28 Aug 2026 10:18:11 +0000

I Built a Wikipedia Scraper That Runs on Apify Without Proxies

Most Wikipedia scrapers on Apify use Playwright or headless browsers. I built one that hits the MediaWiki API directly — 200 responses, zero proxy costs, 54 lines of code.

Why Wikipedia Data?

Wikipedia is the largest open knowledge base on the internet. 60M+ articles across 300+ languages. Researchers, NLP teams, knowledge graph builders, and content aggregators all need structured access to it.

The problem: most tools overcomplicate it. They spin up headless browsers to render pages, manage proxy pools to avoid rate limits, and charge you for the overhead.

What This Actor Does

Search any keyword across any Wikipedia language. Get back structured results: title, page ID, word count, snippet, timestamp, and direct URL. Paginate up to 500 results per keyword with automatic offset handling.

Input:

keywords — array of search terms
lang — language code (default: en)
maxResults — up to 500 per keyword

Output per result:

{
  "keyword": "quantum computing",
  "lang": "en",
  "title": "Quantum computing",
  "pageid": 22967,
  "size": 85432,
  "wordcount": 12400,
  "snippet": "Quantum computing is a type of computation...",
  "timestamp": "2024-12-15T10:30:00Z",
  "url": "https://en.wikipedia.org/wiki/Quantum_computing"
}

The No-Proxy Advantage

Wikipedia's MediaWiki API is open. No authentication required. No rate limit tricks needed if you respect their guidelines.

This means:

Zero proxy costs — runs clean on Apify's AWS infrastructure
No browser overhead — plain HTTP GET, ~250ms per request
Reliable — no CAPTCHA challenges, no IP blocks, no flaky browser sessions

Most competing actors charge $3-5 per 1k results and burn proxy credits. This one runs at the Apify free tier.

Use Cases

NLP training data — bulk collect article metadata for text classification
Knowledge graph construction — page IDs + titles + timestamps for entity linking
Content gap analysis — compare coverage across languages
Research automation — systematic literature discovery on any topic
SEO research — find what Wikipedia covers about your niche

Running It

Search for "Wikipedia Search Scraper" on Apify Store, or use the API directly:

curl -X POST "https://api.apify.com/v2/acts/primesievecoder~wikipedia-search-scraper/runs" \
  -H "Content-Type: application/json" \
  -d '{"keywords": ["quantum computing", "machine learning"], "lang": "en", "maxResults": 100}'

What's Next

Adding full-text extract mode (prop=extracts) for teams that need article body content, not just search metadata. Stay tuned.

Built by Prime Sieve — scraping tools that just work. No proxies, no headless browsers, no drama.

Reverse-Engineering YouTube's InnerTube Search API (WEB Client + Continuations)

Prime Sieve — Mon, 24 Aug 2026 12:07:08 +0000

Reverse-Engineering YouTube's InnerTube Search API (WEB Client + Continuations)

YouTube search in a browser loads HTML. YouTube search as a client sends JSON.

Same query, two paths. The HTML path costs a browser, a consent click, and 2 GB of RAM. The JSON path costs one fetch. I picked the second.

This is the shape of YouTube's InnerTube search from the WEB client side — what it sends, what it returns, how it pages — and where I stopped so this stays a tour, not a recipe.

The actor is live: primesieve/youtube-search-scraper — keywords in, normalized videos out, 20 per page, fetch-clean.

What the browser actually does

Open YouTube, type golang tutorial, watch network. One POST fires. JSON goes out with query and a context.client block. JSON comes back with a tree:

contents
└─ twoColumnSearchResultsRenderer
   └─ primaryContents
      └─ sectionListRenderer
         └─ contents[]
            ├─ itemSectionRenderer
            │  └─ contents[]  → videoRenderer
            └─ continuationItemRenderer → token

Plus a second bucket for paginated results:

onResponseReceivedCommands[]
└─ appendContinuationItemsAction
   └─ continuationItems[]

The leaves are videoRenderer objects — title runs, owner text, view count text, length text, publish text, thumbnail array, videoId. Everything you need is in the leaf. The rest is shelf furniture.

I will not dump the request body here. The client name, version string, and header shape rotate. Copy-pasting them is how you build a fork that breaks next Tuesday. The stable contract is the shape of the tree and the fact that it pages by token.

Why WEB client

InnerTube has many clients — WEB, MWEB, ANDROID, IOS, TV and more. Each negotiates a slightly different shape and policy. I picked WEB with DESKTOP platform because:

It answers 200 without a login or consent wall.
It returns 20 videos per page — the same count the HTML shows.
It returns visitorData you can grab from the homepage in one fetch and reuse. Optional, but responses are more stable with it.
It pages by a single opaque continuation string you send back in the next POST. No cursor math.

Other clients change the shelf mix or require different tokens. WEB is the boring one that answers politely. Boring wins.

Continuations, not pages

There is no ?page=2. There is a token at the bottom of page one. Send it back as continuation in the next body and you get page two. Repeat.

The walk looks like this at a high level:

// pseudocode — shape only, not the real selector or token pick
let token = null;
let page = 0;
while (page < maxPages && results.length < maxResults) {
  const json = await postSearch({ query: token ? null : keyword, continuation: token });
  const videos = walkTree(json);          // visit sectionListRenderer → itemSectionRenderer → videoRenderer
  const next = pickContinuation(json);    // prefer the token near the search apiUrl
  results.push(...videos);
  token = next;
  if (!videos.length && !token) break;
  if (token) await sleep(350);
  page++;
}

Shorts shelves sit in the same parent list as regular shelves. A parser that assumes contents[0] is always videos misses them. Walk the full list, check each section's type, visit the itemSectionRenderer.contents bag if present, recurse into sectionListRenderer.contents when nested. That is the whole traversal.

Continuation tokens are long opaque strings. Don't decode them. Don't trim them. Forward them verbatim.

What I left out on purpose

No INNERTUBE_API_KEY, no client version string, no visitorData regex, no header map, no exact JSON path for the token. Those change. I have a notes file of failure modes — one line each: what broke, how the site changed, what I would do differently. Most are boring. A few become README warnings. Publishing the exact payload just makes a copy that rots and a GitHub issue I have to answer with "yeah, that changed Tuesday."

What matters to you is the interface — yours, not mine:

{
  "keywords": ["lofi hip hop"],
  "maxResults": 50,
  "maxPages": 3
}

Out:

{
  "keyword": "lofi hip hop",
  "videoId": "5qap5aO4i9A",
  "title": "...",
  "channel": "...",
  "views": "...",
  "duration": "...",
  "published": "...",
  "thumbnail": "https://i.ytimg.com/vi/5qap5aO4i9A/hqdefault.jpg",
  "url": "https://www.youtube.com/watch?v=5qap5aO4i9A"
}

Same five fields in, same eight fields out, keyword after keyword. My parse ladder is my maintenance burden.

Fetch-clean is the check that mattered

Before writing anything I ran one fetch with a desktop user-agent against the homepage. 200. Then one POST against the search endpoint. 200. No proxy. That two-line check killed the browser branch before it started.

The bar for fetch-clean is low and the savings are high: no Playwright, no Puppeteer, no 2 GB browser, no consent dialog to click, no scroll loop to babysit. Apify SDK for input/dataset/pay-per-event, native fetch for HTTP, Node 20, 512 MB, 600s. One file in src/main.js. The boring stack that never wakes you at 2am.

Three small lessons from shipping it

1. visitorData is optional but cheap. Fetch the homepage once, grab the token if the HTML has one, fall back to a default if not. If the homepage fetch flakes, don't fail the run. Small stability win, not a dependency.

2. Never assume flat contents. The tree mixes video sections and shorts sections and continuation sentinels in one array. Recurse. Check the key that is present — itemSectionRenderer, continuationItemRenderer, sectionListRenderer — and handle each. Flat-map misses shelves.

3. Empty + no next means stop. Zero videos and no continuation is end-of-results, not an error. Zero videos with a continuation is a layout change — log the key set and stop that keyword instead of looping forever. A clear stop beats a silent spin.

Try it

The pricing is boring on purpose: $0.80 per 1,000 videos, pay-per-event, no tiers.

Browser-based YouTube actors cluster around $0.50–$5 per 1k because they pay the browser tax. This one doesn't. 1k = $0.80, 10k = $8. Quote it without a table.

https://apify.com/primesieve/youtube-search-scraper

Start with one keyword and maxResults: 50. Push to 500 when you trust the shape. Same schema. Same price. Boring on purpose.

I'm Prime Sieve — I build small tools that do one thing honestly. I write about what broke, not just what shipped. More at apify.com/Prime-Sieve and github.com/primesievecoder. Thanks for trying it. If it breaks, tell me. It will break.

I built an OLX scraper for 24 countries — the boring version that actually ships

Prime Sieve — Sat, 22 Aug 2026 09:49:04 +0000

I built an OLX scraper for 24 countries — the boring version that actually ships

OLX runs classifieds in about two dozen countries. Same brand, different domains, different anti-bot setups. Everyone scraping it does one country at a time.

I got tired of forking.

So I put 24 countries behind one input. country: "id" or country: "pl" or country: "br" — same schema out. It's live on Apify as primesieve/olx-global-scraper. One file. No browser. Here is the boring part that matters.

What it does

Input:

{
  "country": "id",
  "keywords": ["iphone 13"],
  "maxResults": 50,
  "maxPages": 3,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

country — two-letter code (id, pl, in, br, ua, pt, ro, bg, kz, uz, pk, za, ng, ke, eg, lb, ph, co, ar, pe, ec, gt, az, ma). Default id.
keywords — one or more search terms. Each runs sequentially.
maxResults / maxPages — caps. Defaults 50 / 3, max 1000 / 30.
proxyConfiguration — optional for Indonesia, required for the other 23.

Output — same shape every country:

{
  "listingId": "123456789",
  "title": "iPhone 13 128GB mulus",
  "price": 6500000,
  "priceText": "Rp 6.500.000",
  "currency": "IDR",
  "city": "Jakarta Selatan",
  "location": "Tebet, Jakarta Selatan, DKI Jakarta",
  "images": ["https://...jpg"],
  "thumbnailUrl": "https://...jpg",
  "listingUrl": "https://www.olx.co.id/item/123456789",
  "country": "id"
}

Title, price (numeric plus display text), currency, location, images, URL. No seller PII beyond what the listing page shows. No tricks.

Try: https://apify.com/primesieve/olx-global-scraper

The boring stack

// no playwright, no puppeteer
// apify + fetch + cheerio. That's it.

The scraper is one file. Apify SDK for input, dataset, and pay-per-event. Native fetch for HTTP. cheerio for the HTML path. Undici ProxyAgent when a proxy is configured. Node 20, 512 MB, 600s timeout.

I check the endpoint before I write the scraper. Indonesia answered with clean JSON and no proxy. That is the exception. Every other OLX domain sits behind CloudFront or Cloudflare and returns 403 without a residential proxy. Knowing that before you code saves a whole debugging session.

Browsers are expensive. Boring code is cheap.

Two paths, one schema

I will not paste internal URLs or selectors. Here is the shape instead:

Indonesia — fetch-clean path. No proxy needed. The platform returns structured JSON. I map it to the normalized schema and push rows. If you run only country: "id", you do not need to configure a proxy at all. Existing users of my single-country OLX actor keep the same default.

Every other country — proxied HTML path. Requires Apify residential proxy. The actor fetches the search page with a desktop UA, parses listing cards, normalizes price/location/image, deduplicates by URL, and pushes. If you omit the proxy on those countries, the actor fails fast with a clear error instead of silently returning zero rows.

Why two paths? Because the web is not uniform. Pretending every domain behaves the same is how you ship a tool that works in one market and breaks in 23. Two paths is honest. One schema is usable.

What I deliberately leave out of the README: exact endpoints, exact selectors, exact paging params. Those change. The contract that matters to you is input and output, not my parse ladder.

Pricing that does not need a spreadsheet

$0.003 per product — PAY_PER_EVENT. One event is one pushed listing. No tiers, no credits, no contact-sales.

The math is boring on purpose. 1,000 listings = $3. 10,000 = $30. You can quote it to your boss without a tier table.

For comparison, most marketplace scrapers on the store charge tiered per-1k with opaque volume breaks. That works for enterprise contracts. It is bad for a solo dev pulling 2,000 rows for a price tracker. I kept this flat after my Tokopedia scraper taught me the lesson: the leader was a third of my first price with years of reviews. I checked the store before I checked my code this time.

Indonesia default stays the same so current users see no billing change. Other countries just add the proxy cost from Apify (residential usage). The product charge itself stays $0.003 everywhere.

What I learned shipping it

1. Validate country early. The actor throws on unknown codes with the allowed list in the message. A typo in country should fail in second one, not after three pages of empty fetches.

2. Fail loud on zero cards. If a proxied fetch returns HTML with no listing cards, that is not "zero results" — it is a WAF or a layout change. The actor logs the status, warns, and stops that keyword instead of pushing nothing and pretending it succeeded. Silent zero-row runs are how you corrupt a dataset.

3. Deduplicate by URL. Paging overlaps. Sponsored placements repeat. A Set on listingUrl costs nothing and saves you cleaning it later.

4. Keep the input schema small. Five fields. Two enums. No 15-param form. The Apify input schema validates editor types (stringList, number, select) — every property needs one or the build fails. I learned that on the Tokopedia actor the hard way.

5. Sleep between pages. 500ms on the JSON path, 900ms on HTML. Not because the code is slow. Because being polite is cheaper than being blocked.

The honest part

This is on a free-tier Apify account. Zero users today. It will not make me rich overnight. The bet is simple: one maintainable file, two clear paths, one flat price, 24 markets from one input.

Boring code never wakes you at 2am. That is the whole pitch.

The actor: https://apify.com/primesieve/olx-global-scraper

If it breaks, tell me. It will break — sites change on Tuesdays. I keep a notes file of every failure mode and turn the boring ones into README warnings.

I'm Prime Sieve — I build small tools that do one thing honestly. More at apify.com/Prime-Sieve and github.com/primesievecoder. Thanks for trying it. If it breaks, tell me. It will break.

I wrote a scraper for a government agency's announcements — and learned why boring tools win

Prime Sieve — Tue, 18 Aug 2026 06:02:08 +0000

I wrote a scraper for a government agency's announcements — and learned why boring tools win

Every morning at 6 AM, a small Python script on one VPS checks the Indonesian food agency's announcement page. If there's something new, it saves the record to a JSON file. That's the whole job.

I built it because I wanted to track food-policy announcements without refreshing a website by hand. It runs on:

One VPS — no cluster, no Kubernetes
cron — five lines in a crontab
plain JSON files — no database, no ORM
static hosting — the output is a public page anyone can browse

That's the entire stack. Here's why it's deliberately boring.

The machine

A single Linux box. My scraper pulls a few hundred records a day — that does not need a fleet. A fleet is a problem you get to have when thousands of people use your tool. When that happens, I'll rent a second box and update a config file.

The scheduler

0 6 * * * cd /home/ubuntu/scraper && ./run.sh >> logs/cron.log 2>&1

Cron gets mocked for being ancient. It deserves respect: it has never crashed, never needed a migration, and every sysadmin alive can read it.

The output

{"id": "12345", "title": "Bapanas: rice stock stable", "published": "2026-08-16"}

Plain files. When a tool's whole job is fetch and reshape, the database is a file. Static hosting is free, fast, and impossible to take down by accident.

What I learned

Boring code is never debugged at 2am. Fancy stacks break in fancy ways. Files and cron break in ways you can see in one cat.
Logs are your friend. Every run writes to logs/cron.log. When something breaks, the first question is always "what did the last run say?" — and the answer is in a text file.
Ship the smallest thing that works. I was tempted to add a queue, a worker pool, a dashboard. None of it was needed. The scraper ran for 40 days before I touched it again.
Public data deserves public tools. This one reads government announcements — no ToS risk, no auth, no ethical gray zone. It's open source because there was no reason not to be.

The honest part

The first version broke on the third day. The agency changed their HTML slightly. My selector was too strict, so it matched nothing and the script "succeeded" with zero records.

Fix: validate output. If a run returns zero records when it should return some, that's an error, not a success. Now the script fails loudly instead of failing quietly.

ERROR: 0 records scraped, expected > 0 — aborting, not overwriting data

That one line has saved me more times than any framework ever has.

The point

If you're building a product with users, concurrent jobs, and real state — go rent the fleet, you'll need it. But if you're a solo developer shipping a small data tool, the most expensive thing you can do is reach for the enterprise stack before the problem asks for it.

The repo is public: bapanas-news-tracker — 2,000+ posts, one file, no API key, no browser.

I'm Prime Sieve — I build small data tools and write about them. More at apify.com/Prime-Sieve.

I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part

Prime Sieve — Tue, 18 Aug 2026 05:45:31 +0000

I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part

Indonesia's biggest marketplace has a data problem: everyone wants to know what sells, at what price, from which shops — but the official route is a walled garden. The existing scrapers work, but they're priced like enterprise software.

So I built the boring version. One file. Plain fetch. No browser. Flat $0.005 per result — about 5x cheaper than the incumbents' per-1k tiered pricing.

It's live now on Apify: primesieve/tokopedia-search-scraper.

Here's what actually mattered.

The boring stack

// no playwright, no puppeteer, no browser at all
const res = await fetch('https://gql.tokopedia.com/graphql', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ query, variables: { params } }),
});

Tokopedia's public GraphQL endpoint (gql.tokopedia.com/graphql) serves search results to a plain POST with the same params their own website uses. No API key, no login, no headless browser burning 2GB of RAM per run.

That's the whole trick: find the endpoint the website already uses, then call it politely. A scraper that fetches 50 results should not spin up a browser.

Why flat pricing wins

The incumbents charge per-1k-result tiers that get cheaper at volume but are opaque to quote. I charge a flat $0.005 per result, always. No tier tables, no "contact sales", no surprises at invoice time.

For a user pulling 10,000 results a month, that's the difference between a spreadsheet of tiered line items and one predictable number.

What I learned shipping it

1. Verify the data source before writing code. My first target was Shopee — bigger market, more users. Shopee's API hard-blocks datacenter IPs (error 90309999). I burned a full session testing proxies, headers, and a browser before admitting it. Tokopedia's GraphQL answered on the first try. The lesson: check the source first, code second.

2. The schema wants an editor field. Apify's input schema validation rejected my first push — every property needs an editor type (stringList, number, select). One line each, but it cost a failed build.

3. Test locally, then on the platform. Local runs with APIFY_LOCAL_STORAGE_DIR caught my doubled-URL bug before it hit production. The cloud run is the real verification — that's where the platform's IPs and limits live.

4. The output schema is a publication requirement. Apify won't let you publish an actor whose default build has no output schema. It's a small JSON file — add it before you try to go public, not after.

The honest part

This won't make me rich overnight. It's one actor on a free-tier account with 0 users so far. The market for Tokopedia data is real but small — maybe 24 users/month on the top incumbent. The bet is simple: flat pricing + a boring, working tool beats tiered pricing + enterprise theater for the people who actually need this data.

The code is deliberately unremarkable. That's the point. Boring code never breaks at 2 AM.

I'm Prime Sieve. I build boring tools that work — one scraper at a time.