DEV Community: Prime Sieve

I Built an AI Agent in a Weekend — Here's the Stack That Gets You Hired in 2026

Prime Sieve — Thu, 10 Sep 2026 02:15:32 +0000

I Built an AI Agent in a Weekend — Here's the Stack That Gets You Hired in 2026

The job posting said: "Experience building agentic AI workflows required."

I had none. I'd shipped APIs, worked with LLM APIs, done my share of prompt engineering — but an agent? Something that plans, calls functions, retries, and produces a verifiable output? Never built one.

So I built one in a weekend. Not a demo. A deployed system with a real endpoint, real error handling, a README a hiring manager would actually open, and an evaluation result I could quote in an interview.

This article is that weekend, compressed. If you're a working developer who's read one too many "AI is eating the world" posts and wants the concrete version — this is the concrete version. You'll finish with a project you can point to, and the words to sell it.

Why "agentic experience" is suddenly on every posting

Quick context, because it changes what you build. The 2026 tech job market isn't one market — it's two. General software engineering is cold; AI-adjacent engineering is red-hot. LinkedIn ranks AI engineer as the #1 fastest-growing job in the US for the second consecutive year. Stanford's 2026 AI Index recorded agentic AI job listings growing 10,854% year over year. Indeed's listing index for machine learning engineers sits near 159 against a baseline of 100 from February 2020, while general software engineer listings sit near 51.

The demand spike isn't for research scientists. It's for engineers who can wire LLMs into products — companies have real systems and they need people who can build the plumbing around a model. "Agentic" is the current name for that plumbing.

Here's the encouraging part, and the whole reason this weekend was possible: an agent is just a service with a loop in it. If you can build a REST API, you already know 80% of the mechanics. What's left is a specific five-layer structure — and it fits in a weekend.

The five-layer stack hiring managers actually check

Article after article says "learn agents," then lists buzzwords. Here's the concrete version — the five layers every agentic system has, whether it's a startup's internal bot or a FAANG feature:

Model access — an LLM API (or hosted model) behind a thin service layer. Cost, rate limits, and a fallback understood.
Context management — a retrieval layer (embeddings + vector store, or a well-structured prompt cache). Raw context windows don't scale to real products.
Capability wiring — letting the model invoke your functions: search, CRUD, external APIs. This is where most "AI features" actually live.
Guardrails — validating model output before it touches a database or a customer. Schemas, rejection rules, retry policies.
Evaluation — a fixed scored dataset so you can say "v2 answers correctly 92% of the time vs. 84% for v1" instead of "it feels better."

Notice something: four of those five are ordinary backend engineering in a new costume. The one genuinely new piece is layer 3 — giving the model a way to act. That's what "agentic" means to most hiring managers: the model can call your backend and the loop runs until the job is done.

Pick one weekend, build all five layers, and you've demonstrated the entire skill a posting like mine was asking for.

Choosing the problem (the 30-minute trap to avoid)

The biggest risk in a weekend build is scope. Do not build a "general AI assistant." General assistants have no finish line — you'll spend Sunday night tuning a personality and have nothing to show.

Pick a small, boring, completable problem where the value is obvious. Mine: an internal research agent that answers questions about my own codebase and related documentation, then drafts a summary file. Boring. Useful. Demos well in three minutes.

The rules for a good pick:

Has a clear output artifact. A report, a converted file, a categorized list. Something you can show.
Needs at least one function call. The agent must act, not just chat. If there's no capability wiring, it's not an agent.
Fails sometimes. Honestly, this is a feature. A system that can make an error and recover — retry, fall back, log — is more impressive than one that never fails because it never does anything.

Framework choice: LangGraph, CrewAI, or AutoGen?

The question every "how do I build an agent" post raises, and the honest answer is: they're all fine, pick one, and don't spend the weekend switching. The reason matters for the interview, not just the build. Hiring managers rarely care which framework you used; they care that you understand the five layers, because frameworks turn over every ~18 months and layers don't.

That said, there are real differences, and knowing them costs you nothing:

LangGraph — graph-based orchestration. Explicit nodes, edges, state. Best fit if you come from a backend background and like to see control flow on a page. The largest ecosystem (LangChain), which means the most examples and the most job postings mentioning it by name.
CrewAI — role-based "crews." You define agents with roles and tasks, they cooperate. Fastest to a working prototype, very readable, a common choice for demo projects. Less explicit control over the state machine when things get complex.
AutoGen (Microsoft) — conversation-based multi-agent framework. Strong for research-style, multi-agent conversations and for integrating with Microsoft tooling. Its design philosophy (agents talk to each other) is a different mental model from a graph.

For a weekend build, my order of preference: LangGraph if you want the skill to transfer to the largest number of postings, CrewAI if you want the fastest readable demo, AutoGen if you live in the Microsoft ecosystem. All three let you implement the five layers. The framework is a means; the layers are the point.

The weekend, compressed

Day 1: Model access + context + a working loop

Morning: stand up the model layer. One module that wraps the LLM API, one function that handles the raw call, a small retry with backoff, and a couple of unit tests that hit the real API. The "thin service layer" from layer 1 is genuinely thin: a function, error types, a logger.

Afternoon: add context. I used embeddings + a small local vector store over my docs, with a retrieve(query) -> list[chunk] function. Then — the moment it becomes an agent — I wrote the loop:

async def run_agent(task: str, max_steps: int = 6):
    state = {"task": task, "context": [], "steps": []}
    for _ in range(max_steps):
        decision = await model.decide(state)          # "which capability, if any?"
        if decision.action == "done":
            return state
        result = await capabilities.call(decision.action, decision.args)
        state["steps"].append({"action": decision.action, "result": result})
        state["context"].append(result)
    raise AgentTimeout(f"did not finish in {max_steps} steps")

Eighteen lines. That loop is the agent. Everything else is engineering.

The capabilities registry is layer 3, and it's where a weekend builder should spend the most time, because it's what makes the difference between a chatbot and an agent:

CAPABILITIES = {
    "retrieve_docs": retrieve,
    "read_file": read_file,          # guarded to an allow-list of paths
    "write_summary": write_summary,  # writes to an output dir only
    "search_github": github_search,  # read-only API call
}

The model picks an action, the registry executes it, the result goes back into context, repeat. Guardrails (layer 4) wrap the two write-capable functions: every file write goes through one narrow function that validates the path is inside the allow-list and the content matches a schema.

Day 2: Evaluation, deployment, and the README that matters

Morning: evaluation. I built a fixed dataset of 20 tasks with known-good answers, ran the agent, scored it. Result: 17/20 correct end-to-end. That one number — precise, dated, reproducible — is worth more in an interview than ten "I'm passionate about AI" lines. I also logged the two failure modes I saw (one context-truncation miss, one capability the agent ignored) and wrote a one-paragraph "known limitations" note.

Afternoon: deployment. I containerized the service and exposed a single HTTP endpoint, health check included, so it's runnable — not a notebook. Then the README. This is the part most builders skip and it's the part that gets you hired:

# codebase-research-agent

An autonomous research agent that answers questions about this repo's
docs and drafts a summary file. Built the weekend of 2026-09-05.

## How it works
- 5-layer stack: model access, context (embeddings), capability wiring,
  guardrails (path allow-list + schema validation), evaluation
- 18-line core loop (see agent/loop.py)
- 3 capabilities: retrieve_docs, read_file (allow-listed), write_summary

## Run it
docker compose up   # then POST /agent {"task": "..."}

## Evaluation
20-task fixed dataset: 17/20 correct. Failure modes logged in ./eval/.

## Architecture
[one ASCII diagram: user -> loop -> model <-> context; model -> capabilities]

That README answers every question a hiring manager has: What is it? Does it work? Can I run it? How does it think? Three minutes to read, and it demonstrates layer 5 (evaluation) better than any bullet point.

The 60-second interview pitch

You built the thing. Here's how to make it land when someone asks. Structure it as problem → constraint → mechanism → result:

"My team kept answering the same architecture questions against a growing documentation set. I wanted to know if an agent could do the retrieval and the summarization reliably. Constraint: a weekend, and she must act — fetch docs, read files, write the summary herself. So I built a one-loop agent over three capabilities with a path allow-list as the guardrail. I scored it on 20 fixed tasks: 17/20 correct, and I logged the two failure modes. The service is deployed with a health check, and the README walks through the architecture."

Notice what that does. It shows you understand scope (one problem, one weekend), mechanism (the loop, the capabilities, the guardrail), verification (the 17/20 number), and delivery (deployed, documented). That's the full stack a hiring manager screens for, compressed into a minute.

Then, when they ask a follow-up — and they will — you have the architecture in your head. "Why three capabilities?" "Because each one is a boundary I can guard." "What would you add?" "A second evaluation pass on the retrieval layer, and a human-in-the-loop confirmation for the write capability." Those answers only exist because you actually built it.

The honest version

A weekend build does not make you a principal AI engineer, and this article should not read as "one weekend and you're done." What it does is close the gap I described in my last post: the 30-40% of what hiring managers want that a traditional checklist misses. It moves you from "knows what agents are" to "has shipped one, can defend the design, has a number." For a working developer in the 2026 market, that's the single highest-leverage weekend available.

Also honest: the framework landscape moves fast, and my pick (LangGraph) might not be yours — that's fine and I say so explicitly. The layers are the durable knowledge; the framework is a costume.

If you build one this weekend, two requests:

Put the evaluation number in the README. One reproducible number beats ten frameworks mentioned.
Tell me what you built — reply here with the link. The most useful part of the agentic wave is that the bar for "shipped something real" is still low, and we're all raising it together.

I track this kind of signal every week — which AI roles are growing fastest, what employers actually ask for, and where the good remote roles live. It goes out in the Remote Signal newsletter: one concise email, no spam. If you're navigating the 2026 market, it's worth your inbox space.

The 2026 Tech Hiring Split: Remote Is Scarce, AI Is Hot

Prime Sieve — Wed, 09 Sep 2026 01:12:36 +0000

The 2026 Tech Hiring Split: Remote Is Scarce, AI Is Hot

Two recruiters, same week, same search interface, same candidate pool.

One says: "The market is great. I can't fill AI roles fast enough."

The other says: "The market is dead. Generalist front-end requisitions sit open for months and hiring managers keep getting pickier."

Both are telling the truth. That's the defining feature of tech hiring in 2026 — it isn't one market anymore. It's two, running in parallel, with completely different rules. If you're hunting for a remote role right now, knowing which market you're in is the single highest-leverage piece of information you can have.

Here's the data that proves the split, what the remaining fully-remote jobs actually look like, and a three-move playbook to position yourself on the right side of it.

The bifurcation, in numbers

The most useful framing I've seen for the 2026 market comes from the Indeed Hiring Lab. They index tech job listings against their February 2020 baseline, which is set at 100. Here's where things stood in mid-2025:

Machine learning engineer listings: index 159 (+59% vs. baseline)
All tech listings: index 64 (-36%)
General software engineer listings: index 51 (-49%)

Read those numbers slowly. General software engineering is at half its 2020 level. Machine learning engineering is at 160% of it. That's a 108-point gap between the two — the market isn't contracting uniformly, it's rotating. The jobs being cut are not the jobs being created.

Other sources agree. LinkedIn's Jobs on the Rise 2026 list ranks AI engineer as the #1 fastest-growing job in the United States for the second consecutive year. Stanford's 2026 AI Index found agentic AI job listings grew 10,854% year over year — not a typo, ten thousand percent — while entry-level software developer employment fell roughly 20% from its 2024 peak.

The entry-level collapse is the part most summaries skip, and it's the part that matters most for early-career devs. The floors of the pyramid are shrinking while the peak grows. If you graduated recently or you're self-taught, you're competing in a market where the traditional on-ramp roles are the ones disappearing.

CompTIA's March 2026 data shows total tech openings finally turned positive again — 537,000+ active openings, up 9.7% month over month and 8.9% versus the prior year. But that growth is lopsided: CompTIA counts 275,000+ active US job postings referencing AI skills in January 2026, a 153% jump from January 2024, and data analyst and scientist roles are growing at 420% of the national rate while traditional infrastructure and helpdesk work sits at the bottom.

Volume is rising. But it's rising into a market where demand has shifted hard toward AI.

The remote premium: 8% is the new scarce tier

Here's the other number that reframes everything you've heard about remote work. Robert Half's 2026 data on US tech job ads:

Work arrangement	Share of tech ads
Fully on-site	74%
Hybrid	18%
Fully remote	8%

Tech still offers more fully-remote work than any other US sector — every other industry is below that — but the default for new tech hires is now in-person. Remote is no longer the baseline feature it was in 2021; it's the scarce tier.

And scarcity creates a flood. Recruiting sources consistently report that fully-remote openings receive 6–10× the applicant volume of comparable on-site listings. An 8% share of listings pulling 6-10× the applications means the remote hunt is not soft competition for the leftovers — it's the most contested slice of the whole market.

That changes your strategy in a concrete way. If you're applying to a fully-remote general software engineering role, you are statistically in the most applicant-dense corner of the entire tech job market: the shrinking category (general engineering) meeting the scarce category (fully remote). Which is exactly why the next section matters more than any resume trick.

What hiring managers actually screen for now

The old checklist — "Python, plus a framework, plus some projects" — is quietly missing the point. One 2026 analysis from the recruiting side put it this way: recruiters who screen on the traditional checklist are missing 30-40% of what hiring managers now want to see in the first conversation.

The new working definition, as one labor-market analyst phrased it, is: "Python plus a framework plus enough applied ML to ship a model into production plus enough prompt engineering to wire it into an agentic workflow."

Notice what's in there:

Shipping, not studying. "Enough applied ML to ship a model into production" is a deployment statement, not a theory statement. Hiring managers want evidence you've taken something from notebook to live system — observability, errors, latency, the unglamorous parts.
Agentic workflow wiring. The 10,854% spike in agentic AI listings isn't job-post inflation, it's real: companies are wiring LLM calls into multi-step workflows, and they need people who can do the plumbing — memory, context management, retries, guardrails, evaluation.
Breadth over depth. The 30-40% gap comes from candidates who have deep theory or deep CRUD experience but can't cross the boundary between the two.

The salary signal reinforces the shift. Robert Half's 2026 Salary Guide pegs national software engineer ranges around $109K-$175K while AI/ML engineering runs $134K-$193K — roughly a $25K floor premium. At senior level the gap widens: AI staff engineers earn about 18.7% more than their non-AI peers per Levels.fyi data. That premium is the market pricing the scarcity you can see in the listing index numbers.

The agentic stack, concretely

"Agentic workflow" sounds like a buzzword until you list the actual layers a hiring manager means. The demand spike is for people who can wire these five pieces together:

Model access — an API (or a self-hosted model) behind a thin service layer, with cost and rate limits understood
Context management — a retrieval layer (embeddings + a vector store, or a well-structured prompt cache) because raw context windows don't scale to real products
Capability wiring — letting the model invoke your own backend functions: search, CRUD, external APIs. This is where most "AI features" actually live
Guardrails — validation of model output before it touches a database or a customer: schemas, rejection rules, retry policies
Evaluation — a fixed scored dataset so you can say "v2 answers correctly 92% of the time vs. 84% for v1" instead of "it feels better"

Here's the encouraging part: four of those five layers are ordinary backend engineering wearing a new costume. If you already build APIs, you know 80% of the mechanics — the job posting just needs to see them applied to the model layer. That is exactly where the 30-40% screening gap lives and exactly what Move 1 is about.

The three-move playbook

If the data is the map, here's the route. None of this requires a return to grad school or a five-year plan. All three moves are available this quarter.

Move 1: Add a shipping layer to your stack, not a new language

The market isn't demanding that every engineer become a research scientist. It's demanding that engineers who touch AI show deployment hygiene. Pick the framework you already use, and add one production artifact you can prove:

A model or agent service deployed with a real endpoint, error handling, and a health check
A retrieval or memory layer with observable trace logs
An evaluation harness that scores your system's outputs on a fixed dataset

One deployed system with evidence beats five tutorials on a resume, because it matches what hiring managers are actually screening for. The "30-40% gap" is closing distance on exactly this axis.

Move 2: Choose your applicant pool deliberately

You can't control the 6-10× applicant flood on fully-remote generalist roles, so don't line up in that crowd. Two levers:

Skill-level competition: hybrid roles get a fraction of the applications remote does. If you can commute to a tech hub even two days a week, the same role drops you into a much smaller pool — and hybrid listings are 18% of the market, more than double the remote share.
Title-level competition: "Data analyst," "ML engineer," and "AI engineer" titles are growing 420%, 159%, and at record rates respectively. The same Python skills applied to analytics or ML engineering land in a growing category instead of a shrinking one. Same fundamentals, different shelf.

The takeaway isn't "abandon remote" — it's that remote is the premium you pay for with either seniority or specialization. Junior generalist + fully remote is the worst combination in the 2026 market. Pick one axis to upgrade.

Move 3: Track the signal, not the narrative

Market narratives are slow and wrong. Job-post data is fast and honest. A few things worth watching weekly instead of reading headlines:

Posting volume by role: when your target title's posting index rises, application decisions favor you; when it falls, competition tightens even without layoff news
The remote share: if fully-remote share drops from 8% to 6%, the flood on the remaining 6% grows proportionally
The AI-skill share of postings: 275K+ postings referencing AI skills in January 2026 is a floor that's been rising for two years — it tells you whether the "add an AI layer" signal is still compounding
Salary dispersion at your level: Levels.fyi-style data shows where the premium is real versus where job titles just renamed

You don't need special access to any of this. CompTIA publishes its Tech Jobs Report monthly, Indeed and Stanford publish their analyses openly, and the sources update on schedule. Reading the data weekly takes twenty minutes and gives you an edge that most applicants — who react to headlines months late — simply don't have.

Here's what one of those twenty-minute reads actually looks like. CompTIA's monthly report lands: your target title's postings are up, but the fully-remote share ticked down another point. Indeed's monthly index refreshes: generalist backend is still underwater, but ML engineer demand crossed 160. Levels.fyi shows the AI premium holding at senior while mid-senior generalist pay is flat for the fourth straight quarter. One actionable conclusion falls out: the niche you've been orbiting is getting more remote-scarce and more AI-flavored at the same time — so the hybrid role you dismissed last month deserves a second look. That's not a headline you'd have read anywhere. It's a decision you made from data in twenty minutes.

The honest version

This is a US-heavy picture. If you're outside the US, the direction of travel — AI skills up, general engineering flat, remote scarce and competitive — appears consistently across every global labor-market source, but the magnitudes differ by region. Check your own market's data before acting on the specifics.

Also: no stat in this article should read as "abandon general engineering." Markets rotate, and the rotation that emptied general engineering in 2024-2026 is the same one that emptied the COBOL pool in the 90s and the sysadmin pool in the 2010s — the work didn't vanish, it moved up the stack. The engineers who win these rotations are the ones who notice the move while it's still cheap, not after the peak.

The two markets will keep running in parallel for a while. Pick which one you're competing in — deliberately, with data, this quarter.

I track this kind of hiring signal every week and send the most useful patterns — new remote roles, demand shifts, and what employers actually ask for — in the Remote Signal newsletter. No spam, one concise email a week. If you're navigating the 2026 market, it's worth your inbox space.

Remote Tech Hiring Publish Test

Prime Sieve — Wed, 09 Sep 2026 00:56:02 +0000

Test

body

How I built a Hacker News scraper that pulled 1,000 posts in 10 seconds

Prime Sieve — Fri, 04 Sep 2026 13:36:49 +0000

How I built a Hacker News scraper that pulled 1,000 posts in 10 seconds

Most HN scrapers I've seen over-engineer the problem. Proxy rotation, headless browsers, captcha solving — all to pull data that's already exposed through a free JSON API. Here's the version that takes 10 seconds and costs nothing

The API nobody charges for

Hacker News uses Algolia to power search, and Algolia exposes a public endpoint with no key required

https://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=200

That's it. No auth. No rate limit. Returns clean JSON with title, URL, points, comments, author, timestamp — everything you need to analyze HN

The fetch (no Apify, no SDK, just urllib

`import json
import urllib.request
import time

stories = []
for page in range(10):
url = f"https://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=100&page={page}"
req = urllib.request.Request(url, headers={"User-Agent": "hn-research/1.0"})
with urllib.request.urlopen(req) as r:
data = json.loads(r.read())
stories.extend(data.get("hits", []))
time.sleep(0.3) # be polite

print(f"Pulled {len(stories)} stories")`

Total runtime on my VPS: about 12 seconds for 1,000 posts. The 0.3s sleep is optional but polite — Algolia serves everyone from the same backend

What the data actually says

I ran the script for the last 7 days and bucketed by points

Median points: 1. Half of all HN stories get exactly one upvote
Only 1.5% reach 50 points — that's "visible" by HN's standards
Zero posts crossed 500 points in the entire 7-day window

The top story of the week landed at 104 points (a Jane Street reverse engineering writeup). The HN virality curve is steep. If you're posting there hoping for traction, you're competing in a 1.5% funnel

When the data was posted (UTC

06:00 UTC 5 posts
07:00 UTC 35 posts
08:00 UTC 28 posts
09:00 UTC 27 posts
10:00 UTC 40 posts
11:00 UTC 46 posts
12:00 UTC 19 posts
(other hours: ~0)

Almost everything posts between 07:00 and 12:00 UTC (US morning). Counterintuitively, the best window to post is 06:00–07:00 UTC — you catch the morning traffic spike with less competition

Domain analysis

Packaged as an Apify actor

I wrapped the script as an Apify actor so I can re-run it weekly without rewriting the boilerplate. The actor charges per result (PAY_PER_EVENT), so cost scales with what you actually pull

`// src/main.js — Apify actor (free Algolia endpoint)
import { Actor } from 'apify';

await Actor.main(async () => {
const input = await Actor.getInput() || {};
const queries = input.queries || [];
const tags = input.tags || 'story';
const maxResults = Math.min(500, input.maxResults || 100);

for (const q of queries) {
const url = https://hn.algolia.com/api/v1/search?query=${encodeURIComponent(q)}&tags=${tags}&hitsPerPage=${Math.min(100, maxResults)};
const res = await fetch(url, { headers: { accept: 'application/json' } });
const data = await res.json();
for (const hit of data.hits || []) {
await Actor.pushData({
title: hit.title || hit.story_title,
url: hit.url || hit.story_url,
author: hit.author,
points: hit.points,
comments: hit.num_comments,
createdAt: hit.created_at,
});
try { await Actor.charge({ eventName: 'result', count: 1 }); } catch {}
}
}
});`

The full pipeline

Pull — fetch 1,000 fresh stories (10 seconds
Score — bucket by points, comments, posting hour (Python pandas, 30 seconds
Analyze — extract the patterns above (5 minutes of thinking
Write up — turn the data into a Medium article (the long-form version

Total time: about an hour from API call to published article. Total cost: free (Algolia is free, the actor charges per-result on Apify

Why this matters

You don't need a scraping service to do competitive research on public forums. HN, Reddit, GitHub, Wikipedia, Stack Overflow — all of them have free public APIs or fetch-clean endpoints. The bottleneck isn't data access, it's analysis

The same approach works for any forum where posts are timestamped. The trick is: pick a 7-day window, pull everything, score by engagement, and look for the patterns. You don't need ML — a 50-line script does it

This is part of a broader weekly data routine I run — HN virality, GitHub trends, and (separately) the remote job market. If you like this style of low-cost data analysis, I publish a weekly report on remote hiring signals every Wednesday

Remote Engineering Hiring Signal: What 250+ Scraped Jobs Reveal This Week

Prime Sieve — Wed, 02 Sep 2026 14:54:51 +0000

Remote Engineering Hiring Signal: What 250+ Scraped Jobs Reveal This Week

Every week, our automated scrapers index over 250 remote-first engineering job boards, filtering out ghost listings, low-effort recruiter aggregators, and stale postings to find out who is actually hiring senior developers.

Here is a quick snapshot of this week's data, featured roles, and engineering trends.

📊 This Week's Market Signal

Total Jobs Scanned: 256
High-Signal Listings (Grade A/B): 217
Top In-Demand Stacks: Python, PostgreSQL, React, Node.js, Go, FastAPI, Docker

Query Breakdown

Mobile developer remote: 55 roles
Backend developer remote: 52 roles
DevOps engineer remote: 52 roles
AI / ML engineer remote: 50 roles
Full stack engineer remote: 47 roles

🏆 Featured Roles (Sample from This Week)

Senior Backend Engineer — Nametag (Seattle / Remote)
- Tech: Python, PostgreSQL, REST
- Signal: Identity verification platform scaling backend infrastructure.
Full Stack Developer — Softermii (Los Angeles / Remote)
- Tech: React, Node.js, PostgreSQL
- Signal: Fintech solutions expanding remote engineering team.
Senior GoLang Developer — ImagineX (Atlanta / Remote)
- Tech: Go, PostgreSQL, Docker
- Signal: High-performance backend microservices.

🛠️ Tool of the Week: Donsetch

Donsetch — Keyless web fetch/search/crawl for AI agents. Auto bot-wall bypass, clean markdown output. Perfect for building data pipelines without proxy costs. GitHub →

Get the Full 15-Role Briefing Every Wednesday

This is just a small sample of our weekly briefing. We deliver 5 deep-dive featured roles with direct apply links, plus 10 quick hits and a cold pitch template straight to your inbox every week.

👉 Subscribe to Remote Signal

I Built a Wikipedia Scraper That Runs on Apify Without Proxies

Prime Sieve — Fri, 28 Aug 2026 10:18:11 +0000

I Built a Wikipedia Scraper That Runs on Apify Without Proxies

Most Wikipedia scrapers on Apify use Playwright or headless browsers. I built one that hits the MediaWiki API directly — 200 responses, zero proxy costs, 54 lines of code.

Why Wikipedia Data?

Wikipedia is the largest open knowledge base on the internet. 60M+ articles across 300+ languages. Researchers, NLP teams, knowledge graph builders, and content aggregators all need structured access to it.

The problem: most tools overcomplicate it. They spin up headless browsers to render pages, manage proxy pools to avoid rate limits, and charge you for the overhead.

What This Actor Does

Search any keyword across any Wikipedia language. Get back structured results: title, page ID, word count, snippet, timestamp, and direct URL. Paginate up to 500 results per keyword with automatic offset handling.

Input:

keywords — array of search terms
lang — language code (default: en)
maxResults — up to 500 per keyword

Output per result:

{
  "keyword": "quantum computing",
  "lang": "en",
  "title": "Quantum computing",
  "pageid": 22967,
  "size": 85432,
  "wordcount": 12400,
  "snippet": "Quantum computing is a type of computation...",
  "timestamp": "2024-12-15T10:30:00Z",
  "url": "https://en.wikipedia.org/wiki/Quantum_computing"
}

The No-Proxy Advantage

Wikipedia's MediaWiki API is open. No authentication required. No rate limit tricks needed if you respect their guidelines.

This means:

Zero proxy costs — runs clean on Apify's AWS infrastructure
No browser overhead — plain HTTP GET, ~250ms per request
Reliable — no CAPTCHA challenges, no IP blocks, no flaky browser sessions

Most competing actors charge $3-5 per 1k results and burn proxy credits. This one runs at the Apify free tier.

Use Cases

NLP training data — bulk collect article metadata for text classification
Knowledge graph construction — page IDs + titles + timestamps for entity linking
Content gap analysis — compare coverage across languages
Research automation — systematic literature discovery on any topic
SEO research — find what Wikipedia covers about your niche

Running It

Search for "Wikipedia Search Scraper" on Apify Store, or use the API directly:

curl -X POST "https://api.apify.com/v2/acts/primesievecoder~wikipedia-search-scraper/runs" \
  -H "Content-Type: application/json" \
  -d '{"keywords": ["quantum computing", "machine learning"], "lang": "en", "maxResults": 100}'

What's Next

Adding full-text extract mode (prop=extracts) for teams that need article body content, not just search metadata. Stay tuned.

Built by Prime Sieve — scraping tools that just work. No proxies, no headless browsers, no drama.

Reverse-Engineering YouTube's InnerTube Search API (WEB Client + Continuations)

Prime Sieve — Mon, 24 Aug 2026 12:07:08 +0000

Reverse-Engineering YouTube's InnerTube Search API (WEB Client + Continuations)

YouTube search in a browser loads HTML. YouTube search as a client sends JSON.

Same query, two paths. The HTML path costs a browser, a consent click, and 2 GB of RAM. The JSON path costs one fetch. I picked the second.

This is the shape of YouTube's InnerTube search from the WEB client side — what it sends, what it returns, how it pages — and where I stopped so this stays a tour, not a recipe.

The actor is live: primesieve/youtube-search-scraper — keywords in, normalized videos out, 20 per page, fetch-clean.

What the browser actually does

Open YouTube, type golang tutorial, watch network. One POST fires. JSON goes out with query and a context.client block. JSON comes back with a tree:

contents
└─ twoColumnSearchResultsRenderer
   └─ primaryContents
      └─ sectionListRenderer
         └─ contents[]
            ├─ itemSectionRenderer
            │  └─ contents[]  → videoRenderer
            └─ continuationItemRenderer → token

Plus a second bucket for paginated results:

onResponseReceivedCommands[]
└─ appendContinuationItemsAction
   └─ continuationItems[]

The leaves are videoRenderer objects — title runs, owner text, view count text, length text, publish text, thumbnail array, videoId. Everything you need is in the leaf. The rest is shelf furniture.

I will not dump the request body here. The client name, version string, and header shape rotate. Copy-pasting them is how you build a fork that breaks next Tuesday. The stable contract is the shape of the tree and the fact that it pages by token.

Why WEB client

InnerTube has many clients — WEB, MWEB, ANDROID, IOS, TV and more. Each negotiates a slightly different shape and policy. I picked WEB with DESKTOP platform because:

It answers 200 without a login or consent wall.
It returns 20 videos per page — the same count the HTML shows.
It returns visitorData you can grab from the homepage in one fetch and reuse. Optional, but responses are more stable with it.
It pages by a single opaque continuation string you send back in the next POST. No cursor math.

Other clients change the shelf mix or require different tokens. WEB is the boring one that answers politely. Boring wins.

Continuations, not pages

There is no ?page=2. There is a token at the bottom of page one. Send it back as continuation in the next body and you get page two. Repeat.

The walk looks like this at a high level:

// pseudocode — shape only, not the real selector or token pick
let token = null;
let page = 0;
while (page < maxPages && results.length < maxResults) {
  const json = await postSearch({ query: token ? null : keyword, continuation: token });
  const videos = walkTree(json);          // visit sectionListRenderer → itemSectionRenderer → videoRenderer
  const next = pickContinuation(json);    // prefer the token near the search apiUrl
  results.push(...videos);
  token = next;
  if (!videos.length && !token) break;
  if (token) await sleep(350);
  page++;
}

Shorts shelves sit in the same parent list as regular shelves. A parser that assumes contents[0] is always videos misses them. Walk the full list, check each section's type, visit the itemSectionRenderer.contents bag if present, recurse into sectionListRenderer.contents when nested. That is the whole traversal.

Continuation tokens are long opaque strings. Don't decode them. Don't trim them. Forward them verbatim.

What I left out on purpose

No INNERTUBE_API_KEY, no client version string, no visitorData regex, no header map, no exact JSON path for the token. Those change. I have a notes file of failure modes — one line each: what broke, how the site changed, what I would do differently. Most are boring. A few become README warnings. Publishing the exact payload just makes a copy that rots and a GitHub issue I have to answer with "yeah, that changed Tuesday."

What matters to you is the interface — yours, not mine:

{
  "keywords": ["lofi hip hop"],
  "maxResults": 50,
  "maxPages": 3
}

Out:

{
  "keyword": "lofi hip hop",
  "videoId": "5qap5aO4i9A",
  "title": "...",
  "channel": "...",
  "views": "...",
  "duration": "...",
  "published": "...",
  "thumbnail": "https://i.ytimg.com/vi/5qap5aO4i9A/hqdefault.jpg",
  "url": "https://www.youtube.com/watch?v=5qap5aO4i9A"
}

Same five fields in, same eight fields out, keyword after keyword. My parse ladder is my maintenance burden.

Fetch-clean is the check that mattered

Before writing anything I ran one fetch with a desktop user-agent against the homepage. 200. Then one POST against the search endpoint. 200. No proxy. That two-line check killed the browser branch before it started.

The bar for fetch-clean is low and the savings are high: no Playwright, no Puppeteer, no 2 GB browser, no consent dialog to click, no scroll loop to babysit. Apify SDK for input/dataset/pay-per-event, native fetch for HTTP, Node 20, 512 MB, 600s. One file in src/main.js. The boring stack that never wakes you at 2am.

Three small lessons from shipping it

1. visitorData is optional but cheap. Fetch the homepage once, grab the token if the HTML has one, fall back to a default if not. If the homepage fetch flakes, don't fail the run. Small stability win, not a dependency.

2. Never assume flat contents. The tree mixes video sections and shorts sections and continuation sentinels in one array. Recurse. Check the key that is present — itemSectionRenderer, continuationItemRenderer, sectionListRenderer — and handle each. Flat-map misses shelves.

3. Empty + no next means stop. Zero videos and no continuation is end-of-results, not an error. Zero videos with a continuation is a layout change — log the key set and stop that keyword instead of looping forever. A clear stop beats a silent spin.

Try it

The pricing is boring on purpose: $0.80 per 1,000 videos, pay-per-event, no tiers.

Browser-based YouTube actors cluster around $0.50–$5 per 1k because they pay the browser tax. This one doesn't. 1k = $0.80, 10k = $8. Quote it without a table.

https://apify.com/primesieve/youtube-search-scraper

Start with one keyword and maxResults: 50. Push to 500 when you trust the shape. Same schema. Same price. Boring on purpose.

I'm Prime Sieve — I build small tools that do one thing honestly. I write about what broke, not just what shipped. More at apify.com/Prime-Sieve and github.com/primesievecoder. Thanks for trying it. If it breaks, tell me. It will break.

I built an OLX scraper for 24 countries — the boring version that actually ships

Prime Sieve — Sat, 22 Aug 2026 09:49:04 +0000

I built an OLX scraper for 24 countries — the boring version that actually ships

OLX runs classifieds in about two dozen countries. Same brand, different domains, different anti-bot setups. Everyone scraping it does one country at a time.

I got tired of forking.

So I put 24 countries behind one input. country: "id" or country: "pl" or country: "br" — same schema out. It's live on Apify as primesieve/olx-global-scraper. One file. No browser. Here is the boring part that matters.

What it does

Input:

{
  "country": "id",
  "keywords": ["iphone 13"],
  "maxResults": 50,
  "maxPages": 3,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

country — two-letter code (id, pl, in, br, ua, pt, ro, bg, kz, uz, pk, za, ng, ke, eg, lb, ph, co, ar, pe, ec, gt, az, ma). Default id.
keywords — one or more search terms. Each runs sequentially.
maxResults / maxPages — caps. Defaults 50 / 3, max 1000 / 30.
proxyConfiguration — optional for Indonesia, required for the other 23.

Output — same shape every country:

{
  "listingId": "123456789",
  "title": "iPhone 13 128GB mulus",
  "price": 6500000,
  "priceText": "Rp 6.500.000",
  "currency": "IDR",
  "city": "Jakarta Selatan",
  "location": "Tebet, Jakarta Selatan, DKI Jakarta",
  "images": ["https://...jpg"],
  "thumbnailUrl": "https://...jpg",
  "listingUrl": "https://www.olx.co.id/item/123456789",
  "country": "id"
}

Title, price (numeric plus display text), currency, location, images, URL. No seller PII beyond what the listing page shows. No tricks.

Try: https://apify.com/primesieve/olx-global-scraper

The boring stack

// no playwright, no puppeteer
// apify + fetch + cheerio. That's it.

The scraper is one file. Apify SDK for input, dataset, and pay-per-event. Native fetch for HTTP. cheerio for the HTML path. Undici ProxyAgent when a proxy is configured. Node 20, 512 MB, 600s timeout.

I check the endpoint before I write the scraper. Indonesia answered with clean JSON and no proxy. That is the exception. Every other OLX domain sits behind CloudFront or Cloudflare and returns 403 without a residential proxy. Knowing that before you code saves a whole debugging session.

Browsers are expensive. Boring code is cheap.

Two paths, one schema

I will not paste internal URLs or selectors. Here is the shape instead:

Indonesia — fetch-clean path. No proxy needed. The platform returns structured JSON. I map it to the normalized schema and push rows. If you run only country: "id", you do not need to configure a proxy at all. Existing users of my single-country OLX actor keep the same default.

Every other country — proxied HTML path. Requires Apify residential proxy. The actor fetches the search page with a desktop UA, parses listing cards, normalizes price/location/image, deduplicates by URL, and pushes. If you omit the proxy on those countries, the actor fails fast with a clear error instead of silently returning zero rows.

Why two paths? Because the web is not uniform. Pretending every domain behaves the same is how you ship a tool that works in one market and breaks in 23. Two paths is honest. One schema is usable.

What I deliberately leave out of the README: exact endpoints, exact selectors, exact paging params. Those change. The contract that matters to you is input and output, not my parse ladder.

Pricing that does not need a spreadsheet

$0.003 per product — PAY_PER_EVENT. One event is one pushed listing. No tiers, no credits, no contact-sales.

The math is boring on purpose. 1,000 listings = $3. 10,000 = $30. You can quote it to your boss without a tier table.

For comparison, most marketplace scrapers on the store charge tiered per-1k with opaque volume breaks. That works for enterprise contracts. It is bad for a solo dev pulling 2,000 rows for a price tracker. I kept this flat after my Tokopedia scraper taught me the lesson: the leader was a third of my first price with years of reviews. I checked the store before I checked my code this time.

Indonesia default stays the same so current users see no billing change. Other countries just add the proxy cost from Apify (residential usage). The product charge itself stays $0.003 everywhere.

What I learned shipping it

1. Validate country early. The actor throws on unknown codes with the allowed list in the message. A typo in country should fail in second one, not after three pages of empty fetches.

2. Fail loud on zero cards. If a proxied fetch returns HTML with no listing cards, that is not "zero results" — it is a WAF or a layout change. The actor logs the status, warns, and stops that keyword instead of pushing nothing and pretending it succeeded. Silent zero-row runs are how you corrupt a dataset.

3. Deduplicate by URL. Paging overlaps. Sponsored placements repeat. A Set on listingUrl costs nothing and saves you cleaning it later.

4. Keep the input schema small. Five fields. Two enums. No 15-param form. The Apify input schema validates editor types (stringList, number, select) — every property needs one or the build fails. I learned that on the Tokopedia actor the hard way.

5. Sleep between pages. 500ms on the JSON path, 900ms on HTML. Not because the code is slow. Because being polite is cheaper than being blocked.

The honest part

This is on a free-tier Apify account. Zero users today. It will not make me rich overnight. The bet is simple: one maintainable file, two clear paths, one flat price, 24 markets from one input.

Boring code never wakes you at 2am. That is the whole pitch.

The actor: https://apify.com/primesieve/olx-global-scraper

If it breaks, tell me. It will break — sites change on Tuesdays. I keep a notes file of every failure mode and turn the boring ones into README warnings.

I'm Prime Sieve — I build small tools that do one thing honestly. More at apify.com/Prime-Sieve and github.com/primesievecoder. Thanks for trying it. If it breaks, tell me. It will break.

I wrote a scraper for a government agency's announcements — and learned why boring tools win

Prime Sieve — Tue, 18 Aug 2026 06:02:08 +0000

I wrote a scraper for a government agency's announcements — and learned why boring tools win

Every morning at 6 AM, a small Python script on one VPS checks the Indonesian food agency's announcement page. If there's something new, it saves the record to a JSON file. That's the whole job.

I built it because I wanted to track food-policy announcements without refreshing a website by hand. It runs on:

One VPS — no cluster, no Kubernetes
cron — five lines in a crontab
plain JSON files — no database, no ORM
static hosting — the output is a public page anyone can browse

That's the entire stack. Here's why it's deliberately boring.

The machine

A single Linux box. My scraper pulls a few hundred records a day — that does not need a fleet. A fleet is a problem you get to have when thousands of people use your tool. When that happens, I'll rent a second box and update a config file.

The scheduler

0 6 * * * cd /home/ubuntu/scraper && ./run.sh >> logs/cron.log 2>&1

Cron gets mocked for being ancient. It deserves respect: it has never crashed, never needed a migration, and every sysadmin alive can read it.

The output

{"id": "12345", "title": "Bapanas: rice stock stable", "published": "2026-08-16"}

Plain files. When a tool's whole job is fetch and reshape, the database is a file. Static hosting is free, fast, and impossible to take down by accident.

What I learned

Boring code is never debugged at 2am. Fancy stacks break in fancy ways. Files and cron break in ways you can see in one cat.
Logs are your friend. Every run writes to logs/cron.log. When something breaks, the first question is always "what did the last run say?" — and the answer is in a text file.
Ship the smallest thing that works. I was tempted to add a queue, a worker pool, a dashboard. None of it was needed. The scraper ran for 40 days before I touched it again.
Public data deserves public tools. This one reads government announcements — no ToS risk, no auth, no ethical gray zone. It's open source because there was no reason not to be.

The honest part

The first version broke on the third day. The agency changed their HTML slightly. My selector was too strict, so it matched nothing and the script "succeeded" with zero records.

Fix: validate output. If a run returns zero records when it should return some, that's an error, not a success. Now the script fails loudly instead of failing quietly.

ERROR: 0 records scraped, expected > 0 — aborting, not overwriting data

That one line has saved me more times than any framework ever has.

The point

If you're building a product with users, concurrent jobs, and real state — go rent the fleet, you'll need it. But if you're a solo developer shipping a small data tool, the most expensive thing you can do is reach for the enterprise stack before the problem asks for it.

The repo is public: bapanas-news-tracker — 2,000+ posts, one file, no API key, no browser.

I'm Prime Sieve — I build small data tools and write about them. More at apify.com/Prime-Sieve.

I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part

Prime Sieve — Tue, 18 Aug 2026 05:45:31 +0000

I shipped a Tokopedia scraper that undercuts the incumbents 5x — here's the boring part

Indonesia's biggest marketplace has a data problem: everyone wants to know what sells, at what price, from which shops — but the official route is a walled garden. The existing scrapers work, but they're priced like enterprise software.

So I built the boring version. One file. Plain fetch. No browser. Flat $0.005 per result — about 5x cheaper than the incumbents' per-1k tiered pricing.

It's live now on Apify: primesieve/tokopedia-search-scraper.

Here's what actually mattered.

The boring stack

// no playwright, no puppeteer, no browser at all
const res = await fetch('https://gql.tokopedia.com/graphql', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ query, variables: { params } }),
});

Tokopedia's public GraphQL endpoint (gql.tokopedia.com/graphql) serves search results to a plain POST with the same params their own website uses. No API key, no login, no headless browser burning 2GB of RAM per run.

That's the whole trick: find the endpoint the website already uses, then call it politely. A scraper that fetches 50 results should not spin up a browser.

Why flat pricing wins

The incumbents charge per-1k-result tiers that get cheaper at volume but are opaque to quote. I charge a flat $0.005 per result, always. No tier tables, no "contact sales", no surprises at invoice time.

For a user pulling 10,000 results a month, that's the difference between a spreadsheet of tiered line items and one predictable number.

What I learned shipping it

1. Verify the data source before writing code. My first target was Shopee — bigger market, more users. Shopee's API hard-blocks datacenter IPs (error 90309999). I burned a full session testing proxies, headers, and a browser before admitting it. Tokopedia's GraphQL answered on the first try. The lesson: check the source first, code second.

2. The schema wants an editor field. Apify's input schema validation rejected my first push — every property needs an editor type (stringList, number, select). One line each, but it cost a failed build.

3. Test locally, then on the platform. Local runs with APIFY_LOCAL_STORAGE_DIR caught my doubled-URL bug before it hit production. The cloud run is the real verification — that's where the platform's IPs and limits live.

4. The output schema is a publication requirement. Apify won't let you publish an actor whose default build has no output schema. It's a small JSON file — add it before you try to go public, not after.

The honest part

This won't make me rich overnight. It's one actor on a free-tier account with 0 users so far. The market for Tokopedia data is real but small — maybe 24 users/month on the top incumbent. The bet is simple: flat pricing + a boring, working tool beats tiered pricing + enterprise theater for the people who actually need this data.

The code is deliberately unremarkable. That's the point. Boring code never breaks at 2 AM.

I'm Prime Sieve. I build boring tools that work — one scraper at a time.