DEV Community: Fatih İlhan

I Automated a TikTok Campaign for My App by Editing CapCut's Project Files Directly

Fatih İlhan — Mon, 19 Oct 2026 19:23:29 +0000

I shipped Judge Me, a small Android app where people read "Am I the jerk?"-style dilemmas and vote: you were right, you were wrong, both of you messed up, or it's complicated. The app works. Nobody knows it exists.

So I'm running a 14-day experiment: short vertical videos on TikTok and Instagram Reels, one per day, each telling one dilemma from the app and ending with "go vote." The goal is simple and measurable: 500+ Play Store installs in 14 days, or one video over 50k views. If neither happens, I stop and try something else.

This post is about the boring-but-interesting part: how I made producing a video take ~5 minutes of my time instead of an hour, with no paid tools.

The constraints

Solo dev with a full-time day job. Video editing time is near zero.
No budget for AI video SaaS subscriptions.
English content aimed at a US audience, posted from Turkey (so everything gets scheduled for 03:00 my time).
Every video has the same structure, which is the key insight: if the structure is fixed, the editing is a data problem.

Each video looks like this:

Time	What's on screen
0–2 s	Big hook text (first sentence of the story)
2 s → end−4 s	Background clip + AI voiceover + 2–3 word captions
Last 4 s	"Right or wrong? YOU decide." + screenshot of the app's vote screen

Step 1: Script → voice + word timestamps

Scripts are 100–130 words, first person, short sentences, ending with "So… was I wrong?". I use edge-tts, which is free and — crucially — streams word boundary events with timestamps alongside the audio.

comm = edge_tts.Communicate(text, "en-US-GuyNeural", rate="+12%", boundary="WordBoundary")
async for chunk in comm.stream():
    if chunk["type"] == "audio":
        f.write(chunk["data"])
    elif chunk["type"] == "WordBoundary":
        start = chunk["offset"] // 10_000  # 100ns ticks → ms
        words.append({"text": chunk["text"], "start_ms": start,
                      "end_ms": start + chunk["duration"] // 10_000})

That gives me an MP3 plus a JSON list of every word and when it's spoken. Captions come for free.

Gotcha: word boundaries have no punctuation

My first captions looked like star She launched and reviews So I — chunks bleeding across sentences. The boundary events return bare words, so my "break on punctuation" rule never fired.

Fix: re-align the timed words against the original script tokens (which do have punctuation), then chunk in two passes — split on clause boundaries first, then divide each clause into balanced groups so you never get a lonely one-word chunk:

def chunk_words(words, max_words=3, max_chars=20):
    clauses, cur = [], []
    for w in words:
        cur.append(w)
        if w["text"][-1:] in ".!?,;:…":
            clauses.append(cur); cur = []
    if cur: clauses.append(cur)

    chunks = []
    for cl in clauses:
        k = -(-len(cl) // max_words)          # ceil
        while True:                            # balance sizes, respect char limit
            size, extra = divmod(len(cl), k)
            parts, i = [], 0
            for j in range(k):
                step = size + (j < extra)
                parts.append(cl[i:i+step]); i += step
            if all(len(" ".join(x["text"] for x in p)) <= max_chars for p in parts) or k >= len(cl):
                break
            k += 1
        chunks.extend(parts)
    return chunks

Before: They showed up | late.
After: They showed | up late.

Small thing, but captions are most of what people actually read on mute.

Step 2: CapCut projects are just JSON

This is the part that made the whole pipeline possible. CapCut desktop stores each project as a folder containing draft_content.json — tracks, segments, materials, text styles, all in plain JSON. (Some versions encrypt it; mine, 9.6 on Windows, didn't. Check yours before building on this.)

The structure, simplified:

tracks[]            → video / audio / text tracks, each with segments[]
segments[]          → material_id + target_timerange {start, duration} in µs + clip transform
materials.texts[]   → text content (as an embedded JSON string!) + style
materials.videos[]  → file paths for clips and images
materials.audios[]  → file paths for audio

My approach:

Build one template project by hand with placeholder text layers: HOOK_TEXT, SUB_1, VOTE_TEXT, plus the background, audio, and vote-screen image.
Let CapCut open and re-save it once, so the file is in its own canonical format.
Keep that folder as a read-only master.
For each video, a Python script copies the template's JSON and rewrites it:
- swap the audio and background paths
- set every duration to the MP3 length (in microseconds)
- put the first sentence into HOOK_TEXT
- clone the SUB_1 layer once per caption chunk, each with a fresh UUID and the chunk's timing
- set the vote text and move it to the last 4 seconds

The one non-obvious detail: the text content lives in a JSON string inside the JSON, with a character range for styling. Change the text without updating the range and the styling breaks.

def set_text(mat, text):
    c = json.loads(mat["content"])
    c["text"] = text
    for st in c["styles"]:
        st["range"] = [0, len(text)]
    mat["content"] = json.dumps(c, ensure_ascii=False)

Then I open CapCut, glance at the timeline, and hit Export. That's my whole manual step.

Batch mode

Once one video worked, I generated 10 separate CapCut projects in one go — copy the template folder, give it a new draft_id, register it in CapCut's root_meta_info.json, render the content. CapCut picked them all up in its project list. Ten videos, ten exports, done in one sitting.

Rules I learned the hard way

CapCut must be fully closed when you write. If it's open, it autosaves over your changes. A .locked file in the project folder tells you it's open.
Back up before every write. My script copies the target project to a backup folder first, every time.
Text wrapping isn't guaranteed. My first hook ran off both edges of the screen — CapCut ignored the max line width I set. I now wrap the hook myself into balanced lines of ≤20 characters before writing it.
Pin the CapCut version and turn off auto-update. This is an undocumented format. An update can break everything.

What didn't work: letting an AI agent click through CapCut

My first attempt was to have an AI agent with desktop control build the template inside CapCut by clicking. It was slower and flakier than I expected: CapCut's media panel didn't respond to synthetic mouse events at all (couldn't drag a text layer onto the timeline), and the agent's own window kept stealing focus. After a few rounds I switched to "just edit the files," which took less time than the clicking attempts had.

Lesson: if an app has a readable file format, editing the file beats automating the UI almost every time.

Distribution gotcha: no link in bio

Plan said "link in bio." TikTok said no — clickable bio links need 1,000 followers on a personal account, and the business account switch wasn't available to me without verification. Plain-text URLs in the bio aren't clickable.

So the end card now says "Vote free: search 'Judge Me AITA' on Google Play" — I checked that exact search ranks the app first. The downside: I lose UTM attribution on TikTok and have to infer it from Play Console's search-traffic numbers. Instagram still gets a proper tracked link.

The setup, in numbers

Cost: $0 (edge-tts, CapCut free tier, Pexels backgrounds)
My time per video after setup: ~5 minutes (open project, check, export, schedule)
Stock ready before day 1: 10 videos
Posting: one per day at 03:00 Turkey time (evening in the US), scheduled

What's next

The experiment starts this week. I'll post the results — views, installs, what hooks worked, and whether 03:00 scheduling actually mattered — when the 14 days are up, including if it flops.

If you're an indie dev sitting on an app nobody knows about, the takeaway isn't "use CapCut." It's: find the fixed structure in your marketing content, and turn the repetitive part into data.

Judge Me is on Google Play. I build in public as @fatihbuilds.

UK public tenders now live in two systems. Here's what I learned merging them.

Fatih İlhan — Thu, 15 Oct 2026 18:56:56 +0000

If you track UK public sector contracts, for sales leads, market research or journalism, you probably noticed it got messier. Since the Procurement Act 2023 regime started in 2025, notices are spread across two government systems:

Find a Tender Service (FTS): the central platform, used for most notices under the new regime
Contracts Finder (CF): still live and still receiving notices, including lower-value opportunities If you only watch one, you miss part of the market. In a 3-day sample I pulled this week, FTS had 1,247 notices and Contracts Finder had 497.

The good news: both publish open data in the same standard (OCDS), with no API key, under the Open Government Licence v3, which allows commercial reuse with attribution.

The bad news: "same standard" doesn't mean "same data". Here's what I ran into while merging them into one feed.

Disclosure: I turned this into a paid Apify actor (linked at the end). Everything below also applies if you build it yourself, and I've tried to write it so you can.

1. The rate limit is stricter than the docs suggest

Both hosts started returning HTTP 429 with Retry-After: 120 after roughly 20 requests in my tests. Contracts Finder's docs mention a 403 instead, but I got 429s. After the first burst I could get about 5 pages a minute.

This was consistent across three test runs, but it's my observation, not a documented number. In practice:

Respect Retry-After. Don't retry blindly.
Keep date windows small. A 1-day pull finishes in seconds; a 7-day pull of everything takes minutes.
For monitoring, run daily and only fetch what's new since the last run. ## 2. The stages filter silently drops notices

Both APIs accept a stage filter (tender, award, ...). It looks like the obvious way to get only open tenders.

Don't use it. In one test, FTS with stages=tender returned 11 of 101 tender releases that were in the unfiltered response. No error, just missing rows.

Fetch everything for the date window and filter on your side using the release's tag and tender.status.

3. The same tender can appear in both systems with different IDs

OCDS gives every procurement an ocid. You'd expect to deduplicate on it. But the two systems use different ocid prefixes, so the same tender published in both places looks like two different procurements.

Real example from this week: a council RFQ for a "Dublin Belfast Economic Corridor Investment Land Guide" appeared in both feeds with different ocids.

What worked for me was a composite key:

import re

PREFIX = re.compile(r"^(GB-[A-Za-z]+:|UK\d+ Tender Notice:|[A-Z]{2,4}_?\d+\s*-)\s*", re.I)

def norm(s: str) -> str:
    s = PREFIX.sub("", s or "")
    return re.sub(r"[^a-z0-9 ]", "", s.lower()).strip()

def dedup_key(n: dict) -> tuple:
    return (
        norm(n["buyer_name"]),
        norm(n["title"]),
        (n.get("deadline_date") or "")[:10],  # same day
    )

Titles often carry system prefixes ("GB-Glasgow:", "UK4 Tender Notice:", "PRJ_31142 -"), so strip those first.

There's a second, smaller case: the same system sometimes publishes a notice twice a few minutes apart, with identical title and text but different IDs. I merge those too and keep the other ID in a related_notice_ids field, so nothing is lost.

4. Fields that look the same but aren't

A few things that will bite you if you just map field names:

Region formats differ. FTS uses NUTS codes (UKK14); Contracts Finder often uses names ("East of England"). And about half of CF notices have no region at all. I split it into region_code + region_name and map CF names to NUTS1.
Award notices carry old deadlines. A CF award notice can have a deadline_date from 2024, which is the original tender deadline. If you show it as "deadline", users will think it expired. For awards, move it to a separate field.
CF award value ≠ awarded amount. It's often the tender's estimated value.
Some FTS notices have no CPV code, so a CPV filter won't see them. Add a keyword fallback if you need full coverage.
CF descriptions sometimes repeat themselves. The "Additional information" block can be a full copy of the description. ## Doing it yourself

Here's a minimal Contracts Finder pull for the last 24 hours:

import time, requests
from datetime import datetime, timedelta, timezone

BASE = "https://www.contractsfinder.service.gov.uk/Published/Notices/OCDS/Search"
now = datetime.now(timezone.utc)
params = {
    "publishedFrom": (now - timedelta(days=1)).strftime("%Y-%m-%dT%H:%M:%SZ"),
    "publishedTo": now.strftime("%Y-%m-%dT%H:%M:%SZ"),
    "limit": 100,
}

url, releases = BASE, []
while url:
    r = requests.get(url, params=params, timeout=30)
    if r.status_code == 429:
        time.sleep(int(r.headers.get("Retry-After", 120)))
        continue
    r.raise_for_status()
    data = r.json()
    releases += data.get("releases", [])
    url, params = data.get("links", {}).get("next"), None

print(len(releases), "releases")

Find a Tender has its own OCDS endpoint (/api/1.0/ocdsReleasePackages) with similar pagination. Check the FTS API docs for the current parameters. Then normalise both into one schema and apply the dedup above.

Or use the hosted version

If you'd rather not maintain this, I published it as an Apify actor: UK Public Tenders: Find a Tender + Contracts Finder.

One schema for both systems, with cross-source and same-source dedup
Filters: keywords, CPV prefix (e.g. 72 for IT), buyer, value range, region, notice type
onlyNew mode for daily monitoring: schedule it and get only notices you haven't seen
Handles the rate limit and keeps going if one source is down (logged in a run summary)
Open tenders and planning notices by default; awards are opt-in
Pricing: $3 per 1,000 results (less on higher Apify plans)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("seralifatih/uk-public-tenders").call(run_input={
    "cpvCodes": ["72"],      # IT services
    "minValue": 100000,
    "onlyNew": True,
})
for n in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(n["published_date"][:10], n["buyer_name"], "|", n["title_clean"], "|", n["deadline_date"])

Data: Contains public sector information licensed under the Open Government Licence v3.0.

If you track UK procurement another way, or hit a quirk I didn't list, tell me in the comments.

House Stock Watcher and Senate Stock Watcher are down: what to use instead (2026)

Fatih İlhan — Tue, 06 Oct 2026 10:47:33 +0000

If your congress-trades feature stopped working, you're not alone. House Stock Watcher and Senate Stock Watcher were the default free source for STOCK Act trade data for years, and a lot of projects still point at them. They are no longer usable:

Source	Status (checked Oct 2026)
`house-stock-watcher-data.s3-us-west-2.amazonaws.com`	403 Forbidden
`senate-stock-watcher-data.s3-us-west-2.amazonaws.com`	403 Forbidden
`housestockwatcher.com`, `senatestockwatcher.com` (incl. `/api`)	Domain doesn't resolve
`raw.githubusercontent.com/timothycarambat/senate-stock-watcher-data/...`	Still responds, but last commit was March 2021. Using it gives you 5-year-old data with no error message

The last one is the dangerous case. I found several projects that use the GitHub mirror as a "fallback". That fallback never fails, so nobody notices the data stopped in 2021.

You can check the others yourself:

curl -I https://house-stock-watcher-data.s3-us-west-2.amazonaws.com/data/all_transactions.json
# HTTP/1.1 403 Forbidden

Below are your options, from free-but-work to paid-but-easy.

Disclosure: I build one of the options below (option 3). I've tried to list the others fairly.

Option 1: Parse the official sources yourself (free)

All of this data comes from two government sites:

House: the House Clerk Financial Disclosure site publishes a yearly ZIP with an XML index of every filing ({year}FD.zip). Periodic Transaction Reports (PTRs) are PDFs at /public_disc/ptr-pdfs/{year}/{doc_id}.pdf.
Senate: Senate eFD has a search UI. You must accept its terms, and it uses a session and CSRF token. Electronic PTRs are HTML tables. Paper filings are scanned images.

The catch is that the House side is PDF parsing. Most PTRs have a text layer, but the layout varies:

amounts can be split across lines
some filings list exchanges
around 14% of recent House PTRs are scanned images with no text at all

A naive regex parser silently drops a lot of rows. Mine dropped almost half of them before I caught it. If you go this route, count filings in vs. rows out, and log the filings that produce zero rows.

Good fit: you want full control and don't mind maintaining a parser.

Option 2: Commercial APIs and sites

Several services sell this data, including Quiver Quantitative, Unusual Whales, Capitol Trades (web UI) and some general market-data APIs that have congress endpoints. They're polished, but check two things before you integrate:

Pricing and rate limits for the volume you need.
Commercial-use terms. Federal law (5 U.S.C. §13107(c)) restricts some commercial uses of these disclosure reports, and vendors handle that differently.

Good fit: you need a dashboard or many extra datasets, and the price works for you.

Option 3: Hosted feeds on Apify (pay per row)

I maintain two Apify actors that parse the official House and Senate sources into one clean JSON schema:

What you get:

One schema for both chambers, with member_bioguide_id on each row so you can join to other congress data
Filters: member (nickname-aware), ticker, filing-date window, transaction-date range
No silent drops: filings that can't be parsed (scanned or failed) come back as placeholder rows with a parse_status, instead of disappearing
Amendment handling: filing_type, supersedes_filing_id, is_superseded
Pricing: $2 per 1,000 rows (House) and $3 per 1,000 rows (Senate). Apify's free plan includes monthly credit, which covers a typical weekly pull.

Python example:

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("seralifatih/congress-trading-pipeline-1").call(
    run_input={"fetchDaysBack": 30, "tickers": ["NVDA", "MSFT"]}
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["transaction_date"], row["politician"], row["ticker"],
          row["type"], row["amount_min"], row["amount_max"])

Migrating from Stock Watcher fields

If your code expects the old Stock Watcher JSON, the mapping is mostly direct:

Stock Watcher	This schema	Note
`representative` / `senator`	`politician`	normalized name; raw name in `politician_raw`
`transaction_date`	`transaction_date`	ISO `YYYY-MM-DD`
`disclosure_date`	`filing_date`
`ticker`	`ticker`	empty when the filing has none
`asset_description`	`asset_name`
`type` (`purchase`, `sale_full`, …)	`type`	`buy` / `sell` / `exchange`
`amount` (`"$1,001 - $15,000"`)	`amount_min`, `amount_max`	already numeric
`owner`	`owner`	`self` / `spouse` / `joint` / `child`
`ptr_link`	`pdf_url`	House

Good fit: you want the old "just give me JSON" experience without maintaining a parser.

Option 4: Just need history? Free 2025 snapshot

If you only need a fixed dataset for backtesting or analysis, I published every 2025 House and Senate trade as a free CSV/Parquet file. It has 8,462 rows, 127 members, and the same schema as option 3:

import pandas as pd
df = pd.read_parquet("hf://datasets/seralifatih/us-congress-stock-trades-2025/us_congress_trades_2025.parquet")

One finding from that data: about 12% of 2025 trades were disclosed more than 45 days after the trade, which is past the STOCK Act deadline. If your strategy assumes disclosures arrive on time, check that assumption.

Quick checklist if your feature broke

Search your code for stock-watcher, stockwatcher.com and timothycarambat.
Remove any "fallback" to the GitHub mirror. It hides the outage.
Pick an option above and update the field mapping.
Add a freshness check: alert if the newest filing_date is more than ~7 days old.

If you know another working source, leave a comment and I'll add it to the list.

My congressional trading scraper was quietly losing half its data

Fatih İlhan — Tue, 29 Sep 2026 18:20:31 +0000

I run two small Apify actors that pull U.S. congressional stock trading disclosures (the STOCK Act "PTR" filings) for the House and the Senate and turn them into clean JSON. A handful of people pay for them. Revenue had plateaued, so I did what I should have done months ago: I stopped looking at the marketing and started auditing my own output.

It was worse than I thought.

The number that ruined my evening

I took the 50 most recent House filings and checked how many rows each one produced.

Almost half of them produced zero rows. Not errors. Not warnings anyone would see. Just nothing. The run finished green, the dataset looked healthy, and a big chunk of Congress's trades simply weren't there.

After fixing it, the same 50 filings went from 238 rows to 416.

If you're a paying user, that's the kind of thing you'd never notice. You'd just assume your data was complete. That's what bothered me most.

Why it happened

There wasn't one bug. There were several small ones that all failed the same way: quietly.

1. An amount format I never planned for. House filings report amounts as ranges like $1,001 - $15,000. My regex required a range. Then one filing showed up with an exact amount, $2,722.50, with cents. The regex didn't match, the parser returned an empty array, and the filing vanished.

2. Exchange transactions. Some rows are exchanges (swap one holding for another), not buys or sells. I didn't have a type for them, so they got dropped.

3. Scanned paper filings. About 14% of House filings in the last 90 days aren't digital at all. Someone printed the form, filled it in, and hand-delivered it. The PDF is just an image. My parser saw no text, returned nothing, moved on.

4. The Senate had the same problem, and I had a comment saying it didn't. There was literally a code comment in my Senate actor saying "Senate has no scanned case, source is HTML." Wrong. The Senate also accepts paper filings, they just live at a different URL. In the last 30 days: 3 out of 49.

5. Network failures. When a PDF download failed after retries, I incremented an error counter and... that's it. The filing was known, it was in the index, and it disappeared from the output.

The pattern is obvious in hindsight: every one of these paths ended in return []. And an empty array is the most polite way a program can lie to you.

The fix: every filing leaves a trace

The rule I landed on is simple. Every filing that appears in the official index must show up in the output. Either as real transaction rows, or as a clearly marked placeholder that tells you why there's no data.

So now every row has a parse_status:

parse_status	What it means
`ok`	Parsed normally
`scanned_unparsed`	Paper filing, image only, no text layer
`parse_failed`	Text exists, but my parser couldn't extract rows (my bug, not yours)
`fetch_failed`	The source couldn't be downloaded after retries

Placeholders keep the politician, filing date, and a link to the original document, so you can go look yourself. One catch: the platform bills every row in the dataset, placeholders included, so if you're counting or paying per result, filter on parse_status = "ok" for the rows that carry real trade data.

The run output now also includes counters (paper filings, parse failures, fetch failures), so a silent drop can't hide in a green run anymore.

Fixing things broke other things

A few smaller lessons from the same week:

My ticker fix broke Mastercard. Some rows had the ticker only inside the asset name, like Electronic Arts Inc. (EA). I added a fallback to extract it, with a stoplist to avoid false positives like (LLC) or (NY). I put all U.S. state codes on that stoplist. Which meant MA (Mastercard), MS (Morgan Stanley), DE (Deere), and MO (Altria) all quietly became null. Removed the state list, added tests for exactly those tickers.

Not every "bug" was a bug. I was sure duplicate content_hash values meant I was losing data. They didn't. The hash fingerprints the transaction content on purpose, so identical trades in the same filing share it, and the unique id handles the rest. It was already documented. I was the one who hadn't read it.

I tried OCR. I stopped.

Since 9 out of 10 scanned forms I checked were printed (not handwritten), OCR seemed doable. It mostly was.

The forms turned out to be checkbox grids: the amount isn't written as text, it's an X in one of eleven lettered columns. Full-page OCR turned that into garbage. So I built a hybrid: OCR for text cells, pixel density for checkboxes.

It got close. On one 6-page filing, date errors went from 29 of 111 rows down to 4. The remaining 4 were Tesseract reading a printed 2 as 7, consistently, on digits a human reads instantly. 06/02/2026 became 06/02/7026.

I could have "fixed" that with rules like "the year must be 2026, so 7 is probably 2." But that same guess on a month or day turns 02 into 07, and now you have a wrong trade date that looks perfectly real. So the policy is all or nothing: if any row in a filing fails validation, the whole filing stays a scanned_unparsed placeholder.

The OCR code is sitting behind a disabled flag. Maybe a different engine later. For now, an honest placeholder beats a confident wrong answer.

What I'd tell anyone building a data product

Count what goes in, not just what comes out. Index size vs output size, per run.
return [] in a parser is a smell. Make failure produce something visible.
Treat "the run succeeded" and "the data is complete" as two different claims.
When in doubt, emit less data with a clear flag rather than more data you can't vouch for. The actors are here if you want to poke at them: House and Senate. If you find a filing that isn't accounted for, I genuinely want to know.

Congressional trading data as an API — comparing pay-per-use vs subscription

Fatih İlhan — Mon, 07 Sep 2026 12:00:58 +0000

Under the STOCK Act, members of the US Congress have to publicly disclose their stock trades. That data is public — but it's locked in PDF filings, and most people never see it because nobody's turned it into something queryable.

A few tools do serve this data as structured output. Here's an honest look at the main options and where a pay-per-use API fits versus a subscription.

The subscription players

Quiver Quantitative has offered congressional trading data since 2020 and is probably the most established name here. Their API access starts around $25–30/month on the Hobbyist tier (billed annually), with higher tiers ($75/month+) unlocking insider trading, lobbying, and other datasets. It's a broad platform — congress trades are one dataset among many (lobbying, government contracts, corporate donors, sentiment, etc.), plus a web dashboard.

Unusual Whales covers congressional trading as part of a much larger options-flow and dark-pool platform. Their dashboard starts at $50/month, and standalone API access has historically started around $150/month for the basic tier. It's built for active options traders — congress trading is a small slice of a much bigger toolset (100+ endpoints).

Both are solid if you want the full platform: dashboards, alerts, backtesting, a maintained brand you can point a client to. You're paying for breadth and polish, and you're paying it every month whether you pull data once or every day.

Where a pay-per-use API fits

If congressional trading is all you need — no dashboard, no options flow, no lobbying data — a subscription is a lot of unused surface area. That's the gap I built into: two Apify actors, one for House filings and one for Senate filings, priced per 1,000 results instead of per month.

House Congress Trade Tracker — $2.00 / 1,000 results
Senate Congress Trade Tracker — $3.00 / 1,000 results No monthly floor. Pull data once for a one-off analysis and pay a few dollars, or schedule it daily and pay for what you actually consume. It runs on Apify, so you get their standard tooling for free: REST API, scheduled runs, webhooks, and client SDKs in whatever language you're already using.

Quick comparison

	Quiver Quantitative	Unusual Whales	Congress Trade Tracker
Pricing model	Subscription	Subscription	Pay-per-result
Entry price	~$25–30/mo	~$50/mo (dashboard), ~$150/mo (API)	$2–3 per 1k results
Scope	Congress trades + many other datasets	Congress trades + options flow, dark pool, etc.	Congress trades only (House + Senate)
Dashboard/alerts	Yes	Yes	No — API/data only
Best for	Ongoing research across multiple datasets	Active options traders wanting one platform	One-off pulls, small scripts, cost-sensitive automation

The honest trade-off

This isn't a "cheaper and better" pitch — it's a different shape. Quiver and Unusual Whales give you a dashboard, alerts, and a dozen adjacent datasets you might grow into. If you want a UI to browse in your browser, they're the right pick. If you're a developer who wants one thing — clean, structured House/Senate trading data — without paying for a platform you won't use, a pay-per-result actor is the cheaper path in practically every scenario short of pulling data continuously at high volume.

Links:

House: apify.com/seralifatih/congress-trading-pipeline-1
Senate: apify.com/seralifatih/congress-trading-pipeline Happy to answer questions on the data format (JSON, STOCK Act PTR fields) in the comments.

I Built a Free World Cup 2026 Live Scores API on Apify — Here's How

Fatih İlhan — Thu, 18 Jun 2026 13:19:36 +0000

The World Cup is here and getting clean, structured match data is surprisingly painful. The official FIFA site has no public API. Third-party services either cost money, require OAuth, or return inconsistently shaped JSON that changes mid-tournament.

I spent a few hours building a free Apify actor that wraps the football-data.org v4 API and outputs clean, normalized JSON — ready to drop into a dashboard, a Discord bot, or an automated betting tool. Here's how it works and how to use it.

What it returns

Every run pushes records to an Apify dataset. A match record looks like this:

{
  "fetchedAt": "2026-06-18T12:00:00Z",
  "mode": "match",
  "matches": [
    {
      "id": 415082,
      "utcDate": "2026-06-11T20:00:00Z",
      "status": "completed",
      "stage": "Group Stage",
      "group": "A",
      "matchday": 1,
      "homeTeam": { "id": 772, "name": "Mexico", "code": "MEX", "crest": "https://..." },
      "awayTeam": { "id": 773, "name": "United States", "code": "USA", "crest": "https://..." },
      "homeScore": 2,
      "awayScore": 1,
      "halfTimeHomeScore": 1,
      "halfTimeAwayScore": 0,
      "goals": [
        { "minute": 23, "type": "REGULAR", "team": "home", "player": "Raúl Jiménez", "assist": null }
      ],
      "bookings": [
        { "minute": 55, "team": "away", "player": "Tyler Adams", "card": "YELLOW" }
      ],
      "venue": "Estadio Azteca",
      "stats": null
    }
  ]
}

Four modes are available:

Mode	What you get
`live`	All in-progress matches right now
`standings`	Full group stage table
`match`	Single match with goals + bookings
`full`	All completed matches (filterable by team)

Note: stats is always null. Possession, shots, corners — these aren't available on football-data.org's free tier. I set the field to null explicitly rather than defaulting to 0, so consumers can handle the absence cleanly.

The interesting parts

Rate limiting done right

football-data.org's free tier caps you at 10 requests per minute. Naive polling will get you 429s. The client enforces a 6-second gap between every request using a timestamp-based throttle:

private async throttle(): Promise<void> {
  const now = Date.now();
  const elapsed = now - this.lastRequestAt;
  if (this.lastRequestAt > 0 && elapsed < THROTTLE_MS) {
    await sleep(THROTTLE_MS - elapsed);
  }
  this.lastRequestAt = Date.now();
}

If a 429 slips through anyway (burst from a previous run), the retry loop adds an extra 6s on top before trying again:

if (status === 429) {
  await sleep(THROTTLE_MS);
}

4xx responses that aren't 429 bail out immediately — no point retrying a bad request.

Normalizing the raw API shape

football-data.org returns statuses like FINISHED, IN_PLAY, TIMED, PAUSED. I normalize these to four clean consumer-friendly values:

const STATUS_MAP: Record<string, string> = {
  FINISHED: 'completed',
  IN_PLAY: 'in_progress',
  PAUSED: 'in_progress',
  SCHEDULED: 'scheduled',
  TIMED: 'scheduled',
  AWARDED: 'completed',
};

Groups come back as GROUP_A, GROUP_B, etc. A quick regex strips the prefix:

function mapGroup(raw: string | null | undefined): string | undefined {
  if (!raw) return undefined;
  const match = raw.match(/^GROUP_([A-Z])$/);
  return match ? match[1] : undefined;
}

Goals and bookings reference the scoring team by team.id. To label them as home or away, I compare against homeTeam.id — no string matching, no locale issues:

team: g.team.id === homeId ? 'home' : 'away',

How to use it

1. Get a free API key

2. Run the actor

Go to apify.com/seralifatih/wc2026-stats and hit Try for free.

Input:

{
  "footballDataApiKey": "your_key_here",
  "mode": "live"
}

Or via the Apify API:

POST https://api.apify.com/v2/acts/seralifatih~wc2026-stats/runs?token=YOUR_APIFY_TOKEN
Content-Type: application/json

{ "footballDataApiKey": "your_key_here", "mode": "live" }

Results land in the run's default dataset as individual JSON items.

3. Schedule it for live match days

Set up an Apify Scheduler with cron */2 * * * * in live mode. Each run pushes only currently in-progress matches. Your dataset stays fresh without polling from your own server.

The World Cup runs June 11 – July 19, 2026. Only enable the scheduler on match days to avoid burning free compute.

4. Filter by team

Pass a teamId with mode: full to get only a specific country's completed matches:

{ "footballDataApiKey": "your_key_here", "mode": "full", "teamId": 773 }

Useful for single-country dashboards or per-team analytics.

Tech stack

Node.js 20 + TypeScript — async pipeline, typed throughout
Apify SDK v3 — dataset push, input/output schema, proxy support
axios — HTTP client with timeout and retry logic
football-data.org v4 — the underlying data source (free tier)

Source: github.com/seralifatih/wc2026-stats

What are you building with World Cup data? Drop it in the comments.

I built a personal job alert pipeline. Here's the stack and what I learned.

Fatih İlhan — Mon, 11 May 2026 13:54:50 +0000

I work at UN. UN job hunting is a particular kind of suffering: openings are scattered across careers.un.org, reliefweb.int, agency-specific portals, and aggregators of varying quality. None of them know my profile. None let me say "only show me P-2 and P-3 protection roles in family duty stations outside Türkiye." So I built one.

This is the story of how I built that — and what the build itself taught me about respecting the tools you scrape from.

The bad idea I started with

My first instinct was simple. Open unjobs.org, it lists everything in one place, write a requests + BeautifulSoup scraper, done. Maybe a Cloudflare Worker, maybe a small Next.js dashboard, ship by Sunday.

I asked Claude to help me scope it. The first thing it did was fetch the site to see the structure.

ROBOTS_DISALLOWED: URL is disallowed by robots.txt rules

Right. unjobs.org explicitly forbids scraping. And the more I read into it, the worse the idea got:

The site is itself an aggregator — it scrapes from official sources. So I'd be scraping a scraper, two layers downstream from truth.
Their Terms of Service prohibit automated access. Ignoring that is not just rude, it's legally risky in some jurisdictions (HiQ vs LinkedIn territory). Scraping the ecosystem I work in, against ToS, is not a career-positive move.

So I asked the obvious follow-up: where does unjobs.org get its data?

The answer: from the same sources I could go to directly. ReliefWeb has a free, documented public API. careers.un.org has an RSS feed. The "hard problem" of aggregating UN jobs was already solved upstream — I just had to be willing to put in more wiring than pip install beautifulsoup4.

This is the first lesson and I want to flag it loudly: before you scrape, check whether the data has a front door. Aggregators exist because real APIs are obscure or undocumented, not because they don't exist. Five minutes of looking saved me a brittle pipeline, legal exposure, and a worse data quality outcome.

The actual architecture

ReliefWeb API (JSON)           careers.un.org (RSS + JSON)
        │                                │
        ▼                                ▼
  sources/reliefweb.py           sources/careers_un.py
        │                                │
        └──────────► Job (normalized) ◄──┘
                       │
                       ▼
                  scraper.py
                  - dedupe (title + source)
                  - score 0–100 (keyword + grade + location)
                  - filter by MIN_SCORE
                       │
                       ▼
                   store.py        ← SQLite, "have I already emailed this?"
                       │
                       ▼
                  notifier.py      ← Resend, HTML email
                       │
                       ▼
               systemd timer       ← Mon + Thu, 09:00 Istanbul time

Adapter pattern: each source normalizes to one Job dataclass. Adding a new source (untalent.org, UNDP, ICRC) is a 50-line file in sources/. Everything downstream — scoring, dedup, store, notifier — never changes.

Stack:

Python 3.12, requests, xml.etree, sqlite3. No frameworks. The whole thing is ~600 lines.
Hetzner VPS (€4/mo Ubuntu box I already had)
systemd timer instead of cron — better logs, easier debugging, Persistent=true re-runs if the box was off at scheduled time
Resend for transactional email (already had a domain via Nokta Studio)
SQLite as the "seen" store — a single file, no daemon, no migration framework

No Docker. No Kubernetes. No cloud functions. A venv, a .env, two systemd units. The whole thing deploys with one shell script.

Discovering the API that wasn't documented

ReliefWeb has a great public API. The careers.un.org side was less obvious.

The RSS feed is fine for discovery — title, level, duty station, deadline — but the body of each posting isn't there. So a "Programme Management Officer" at UNCTAD reads identically to one at UN until you read the full description. With only the RSS metadata, scoring was guessing.

I knew the site had to have an internal API somewhere because it's an Angular SPA — view-source: returns an empty shell, then JavaScript fetches and renders content. So the data is being pulled from something.

Browser DevTools, Network tab, filter to Fetch/XHR, reload the page. After clicking through a few requests, here it is:

GET https://careers.un.org/api/public/opening/joV2/<jobId>/en

Returns a clean JSON with jobDescription, jobLevel, dutyStation, jobFamily, the whole structured record. No auth required. The aggregators I was about to scrape? They almost certainly use exactly this endpoint.

This kept being the pattern. Every "we don't have an API" turned out to be "we don't have a documented API, but our frontend reads JSON from somewhere and that somewhere is reachable." DevTools is, frankly, the most underrated tool in a builder's stack.

A small caveat: when you find an endpoint this way, you should still ask whether you're respecting reasonable use. I added 400ms between requests, a custom User-Agent identifying the project, and capped the run at 200 details/run. The whole pipeline makes about 400 requests per week. That's negligible.

The scoring engine, and why it matters more than the scraping

The scraper is the boring part. The interesting part is: given 400 fetched jobs, which 5–10 should land in my inbox?

My answer was a weighted keyword scorer that knows my CV:

CORE_KEYWORDS = {
    "refugee", "asylum", "migration", "protection",
    "resettlement", "statelessness", "psea",
    # Fraud/Integrity — my current role
    "fraud", "integrity", "investigation", "misconduct",
    # Cash-Based Interventions — trained
    "cbi", "cash-based", "cash assistance",
    # ...
}

SOFT_KEYWORDS = {"humanitarian", "human rights"}  # weak signals

NEGATIVE_KEYWORDS = {
    "intern", "internship", "volunteer", "driver",
    # IT roles — wrong profile
    "information systems officer", "cloud engineer", "service desk",
    # Senior leadership — out of grade band
    "director,", "deputy director", "chief of mission",
    # Wrong functions
    "legal officer", "auditor", "procurement officer",
    "counter-terrorism", "finance and budget", "economic affairs",
}

FAMILY_DUTY_STATIONS = {  # whitelist
    "geneva", "vienna", "rome", "copenhagen", "new york",
    "bangkok", "amman", "nairobi", "panama city", # ...
}

NON_FAMILY_STATIONS = {  # hardship — hard reject
    "kabul", "mogadishu", "juba", "kyiv", "damascus", # ...
}

Then score in layers:

+30 Türkiye-based (or, in my case, −reject because I want to relocate)
+20 Family duty station
+10 per core keyword (capped at +30)
+5 soft keyword (only if no core hit, to avoid double-counting "humanitarian")
+5 per domain keyword (capped)
+25 if P-2 ("next-step fit"), +15 if P-3 ("stretch")
+10 priority agency (priority organizations)
Hard reject if grade is outside P-2/P-3, or if any negative keyword hits

The first version was too generous. "humanitarian" matched almost every UN posting and pumped them all to 30+. The fix: demote it to soft, only count if no core keyword fired.

The second version was bug-ridden. A regex \bintern\b on the title INTERNATIONAL CONSULTANT triggered the intern filter because I wasn't using whole-word boundaries correctly. A few false-positive iterations later, scoring stabilized. Watching real output and iterating is the only way to get this right — you can't reason your way to good filters from a CV alone.

The thing I'm most happy about: the system handles its own ignorance gracefully. When body content is sparse (RSS-only fallback), scores are lower. When duty station isn't in either whitelist or blacklist, it gets a 📍 unverified tag with a slight penalty, so it surfaces but isn't claimed to be a perfect match. Calibrated uncertainty.

The dry-run loop

The single most useful thing I added was a dry-run mode:

DRY_RUN=1 MIN_SCORE=0 python -m main

This fetches everything, scores everything, prints two tables — top matches and top near-misses — but doesn't write to the DB and doesn't send email. Tuning loop becomes:

Run dry-run
See which obviously-wrong job got 60 points
Read the reasons it gave that score
Add/remove a keyword
Repeat

I went through maybe 15 iterations of this in an afternoon. The "reasons" output, where each scored job lists the signals that built up its score, was crucial. Without it I'd have been blindly editing keyword sets.

[ 70] HUMAN RIGHTS OFFICER / OPEN-SOURCE INVESTIGATOR (TJO), P3
      📍 OHCHR / Nairobi
      🏡 Family duty station: Nairobi
      ✅ Profile match: integrity, investigation
      📊 Data/M&E/IM: reporting
      🎯 P3 (stretch but doable)
      🤝 Priority agency: OHCHR
      📰 via UN Careers

When a score is interpretable, the calibration loop is fast. When it's a black box, you're guessing.

The boring stuff that mattered

A few choices that felt over-engineered at the time but I'm glad I made:

Adapter pattern. Each source is a module exposing fetch() -> list[Job]. Two sources today, three or four next month. Adding a source doesn't touch scoring, store, notifier, or the main loop. The Job dataclass is the contract.

SQLite for "seen" state. I considered keeping last-run state in a JSON file. SQLite is one extra import sqlite3 and gets me indexed lookups, transactions, and longevity. Total overhead: maybe 30 lines.

Separate jobalert system user. The service doesn't run as root. The .env (with API key) is chmod 600. The unit has NoNewPrivileges, ProtectSystem=strict, PrivateTmp. If something gets compromised, the blast radius is tiny.

Boilerplate stripping. UN job descriptions all end with the same ~2000 words of legal boilerplate ("No Fee", "United Nations Considerations", "Special Notice"). Without stripping this, scoring would pick up keywords from boilerplate that have nothing to do with the actual role. One regex cut, big quality improvement.

Idempotent install script. A single bash deploy/install.sh does everything: user creation, code sync, venv setup, dependency install, .env templating, systemd registration. Safe to re-run. The first time I had to redeploy to debug something at midnight, this saved my sanity.

What's next

The system is running. It mailed me 7 P-3 roles on the first production run, all genuinely worth a look. Twice a week from now on.

But there's a clear progression:

ReliefWeb appname is pending approval. When it lands, the agency mix changes drastically — UNHCR, IOM, NGOs flood in. Scoring will need recalibration.
LLM-based scoring. Right now keyword matching is the engine. A small Claude API call per job — "given this CV and this job description, output a 0-100 fit score and a one-line reasoning" — would be dramatically more accurate. About $1/month at my scale. The keyword scorer becomes a pre-filter.
A web dashboard. Next.js + the SQLite database, so I can see what got filtered and why, mark jobs as "applied," track outcomes over time.
More sources. untalent.org has a JSON API. UNDP has its own. ICRC, MSF, and others have feeds I haven't touched yet.

Takeaways

If you're building a personal automation pipeline, three things to actually internalize:

Front doors first, side doors second. Before you reach for a scraper, look for an API, an RSS feed, a JSON endpoint behind the Angular SPA. The 30 minutes you spend looking is repaid 100x in maintenance.

Boring stack, weird scoring. The infrastructure should be unremarkable — Python, SQLite, systemd, one VPS. The interesting work is the domain modeling: "what does a good match look like for me?" That's where the value compounds, not in your choice of framework.

Dry-run mode is non-negotiable. Any system that filters or scores or notifies needs a way to see what would happen without committing. The tuning loop dies without it.

The code is ~600 lines. The build took a weekend with Claude as a pairing partner — I'd built the core in an evening, then spent a longer second session calibrating the scorer against real data. Going to open-source it once I've added LLM scoring and a dashboard; will link from @fatihbuilds when it's up.

If you're in the humanitarian sector and want this, message me. If you're not but want to adapt the pattern for your own job search — the keyword-based scoring layer ports to any domain. Same code, different CORE_KEYWORDS.

Building LoopSignal Part 3: Closing the Loop with GitHub Issues and Status Notifications

Fatih İlhan — Fri, 08 May 2026 16:16:55 +0000

In part one, I covered the anonymous submission flow. In part two, I covered the public board, voting, and how I kept prioritization deliberately simple.

This part is about the piece I am most happy with: closing the loop.

The whole reason LoopSignal is called LoopSignal is that feedback is supposed to come back to the user who left it. Someone submits a request, the team works on it, it ships, and the original submitter hears about it. That round trip is the entire product.

For most teams, the place that work actually happens is GitHub. So if LoopSignal is going to close the loop, it has to live where the engineering work lives. That meant a real GitHub integration: linking posts to issues, automatically closing posts when issues close, and notifying everyone who cared.

Here is how I built it.

What "closing the loop" actually requires

Before writing any code, I listed what closing the loop really meant.

It needed to:

let a team connect a GitHub repo to a project
create a GitHub issue from an approved feedback post with one click
listen for issue events from GitHub via webhooks
automatically update the feedback post's status when the issue closes or reopens
notify everyone subscribed to the post when its status changes
handle anonymous subscribers and registered users equally
not block the rest of the product if GitHub is slow or unreachable

That is more moving parts than the submission flow or the public board, but each piece is small if you keep them separate.

Connecting a repo: OAuth, scope, and storage

The connection flow is a standard GitHub OAuth dance, with one detail that matters.

The initiate route encodes both projectId and userId into the OAuth state parameter so the callback can verify access before binding a token to a project:

const state = Buffer.from(
  JSON.stringify({ projectId, userId: user.id })
).toString("base64url");

const githubAuthUrl = new URL("https://github.com/login/oauth/authorize");
githubAuthUrl.searchParams.set("client_id", clientId);
githubAuthUrl.searchParams.set("redirect_uri", redirectUri);
githubAuthUrl.searchParams.set("scope", "repo");
githubAuthUrl.searchParams.set("state", state);

The callback decodes the state, verifies the user actually owns the project, exchanges the code for an access token, and stores the token on the projects row in a github_token column.

I chose scope: "repo" deliberately. A more granular scope would have been nicer, but creating issues, registering webhooks, and reading the repo all need it, and trying to split that into multiple flows would add friction without any real security gain for the kind of teams using this product.

Creating an issue from an approved post

Once a project has a connected repo, every approval can optionally produce an issue.

The logic lives in the moderation action. When a post moves from pending to approved, and the project has both github_repo and github_token set, it fires off an issue creation in the background:

if (newStatus === "approved" && project.github_repo && project.github_token) {
  createGitHubIssue(post, project).catch(console.error);
}

That .catch(console.error) matters. The product's source of truth is the LoopSignal database, not GitHub. If GitHub is rate-limiting us or down, the post should still move to approved. The issue creation is an enhancement, not a dependency.

The actual creation just hits the GitHub REST API:

const res = await fetch(
  `https://api.github.com/repos/${owner}/${repo}/issues`,
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${project.github_token}`,
      Accept: "application/vnd.github+json",
    },
    body: JSON.stringify({
      title: post.title,
      body: `${post.description ?? ""}\n\n---\nCreated from LoopSignal feedback`,
      labels: post.category ? [post.category] : undefined,
    }),
  }
);

After it succeeds, the post row gets a github_issue_url column set to the issue's html_url. That URL is the link between the two systems for the rest of the lifecycle.

The footer line in the issue body — "Created from LoopSignal feedback" — exists for a real reason. It tells the engineer reading the issue why it exists, and where to look if they want context like vote counts or comments. Without that, GitHub issues created from external sources tend to feel orphaned.

Registering the webhook

Connecting the repo also registers a webhook. This is the part that lets GitHub talk back to LoopSignal when something changes.

I check for an existing webhook first, because re-registering on every connect would create duplicates and re-hook on every page reload during testing:

const hooksRes = await fetch(`${repoApi}/hooks`, {
  headers: { Authorization: `Bearer ${token}` },
});
const existing = await hooksRes.json();
const alreadyHooked = existing.some(
  (h: { config?: { url?: string } }) => h.config?.url === webhookUrl
);

if (!alreadyHooked) {
  await fetch(`${repoApi}/hooks`, {
    method: "POST",
    headers: { Authorization: `Bearer ${token}` },
    body: JSON.stringify({
      name: "web",
      active: true,
      events: ["issues"],
      config: { url: webhookUrl, content_type: "json" },
    }),
  });
}

I only subscribe to the issues event. Pull requests, pushes, and the rest are not part of the loop right now, and subscribing to events you do not use is a great way to fill your logs with noise.

The webhook URL is built from the request host so this works the same way locally and in production without environment-specific config.

Reacting to issue events

The webhook handler is small on purpose. It gets one job: translate GitHub issue actions into LoopSignal workflow statuses.

let workflowStatus: WorkflowStatus | null = null;

if (action === "closed") {
  workflowStatus = "completed";
} else if (action === "reopened") {
  workflowStatus = "open";
}

if (!workflowStatus) {
  return NextResponse.json({ ok: true });
}

const { data: post } = await admin
  .from("posts")
  .select("id")
  .eq("github_issue_url", payload.issue.html_url)
  .single();

if (post) {
  await admin
    .from("posts")
    .update({ workflow_status: workflowStatus })
    .eq("id", post.id);
}

A few things were intentional here.

I match posts to issues by the full html_url, not by issue number. Issue numbers are only unique within a repo, and a project could in theory swap which repo it is connected to. The HTML URL is globally unique and already stored when the issue was created.

I ignore actions other than closed and reopened. edited, assigned, labeled, and the rest can be useful eventually, but right now they would just create noise in the public board's history. Closing the loop only needs the two events that change visible state.

I also do not currently verify the webhook signature. That is on my list. For an early version with low traffic, the cost of a malicious webhook is bounded — at worst, someone could mark posts as completed — and the engineering cost of HMAC verification is small enough that I would rather wait until I am sure the webhook surface is stable before locking it in.

Notifying the people who care

A status change without notifications is just a database update. The whole reason for this system is that someone submitted feedback and wants to know what happened to it.

Notifications run from a small helper that fires after status changes:

const meaningfulStatuses = ["planned", "in_progress", "completed", "closed"];
if (!meaningfulStatuses.includes(newStatus)) return;

const { data: project } = await admin
  .from("projects")
  .select("notifications_enabled, name, slug")
  .eq("id", post.project_id)
  .single();

if (!project?.notifications_enabled) return;

const { data: subscriptions } = await admin
  .from("post_subscriptions")
  .select("id, user_id, email")
  .eq("post_id", post.id)
  .eq("unsubscribed", false);

Two things in here matter more than they might look.

First, only some statuses send email. open does not — every post starts as open, so emailing about it would be spam. The notification list is the four states that actually represent progress.

Second, the project itself has a notifications_enabled flag. Some teams want quiet boards, especially during early product development, and forcing them into outbound email would push them off the product.

Anonymous and registered subscribers in one table

This part took a bit of thought.

Subscribers can come from two places:

registered users who voted while logged in
anonymous users who left an email when submitting

Rather than create separate tables for each, I store both in a single post_subscriptions table with two nullable columns: user_id and email. At notification time, the code resolves whichever is set:

let recipientEmail = sub.email;

if (!recipientEmail && sub.user_id) {
  const { data: { user } } =
    await admin.auth.admin.getUserById(sub.user_id);
  recipientEmail = user?.email ?? null;
}

if (!recipientEmail) continue;

await sendStatusUpdateEmail({
  to: recipientEmail,
  postTitle: post.title,
  newStatus,
  unsubscribeUrl: `${baseUrl}/api/unsubscribe?id=${sub.id}`,
});

That keeps the subscription model simple and lets registered users update their email in one place — their account — rather than having to update every subscription. Anonymous emails are stored directly because there is nothing else to look them up against.

Every email carries an unsubscribe link tied to the specific subscription row. One click marks unsubscribed = true and the row stops receiving future updates. That is non-negotiable for any system sending unsolicited mail to people who left an email a long time ago.

What the loop actually feels like now

When all of these pieces line up, the experience is the part of the product I am most proud of.

Someone submits feedback anonymously. A team member approves it. A GitHub issue gets created. The team works on the feature. They merge the PR and close the issue. GitHub fires the webhook. LoopSignal moves the post to completed. Everyone subscribed gets an email saying the thing they asked for shipped, with a link back to the post and an option to unsubscribe.

The user who submitted the original feedback never had to create an account, never had to log in to check status, never had to chase the team for an update. The system did the chasing for them.

That is the loop.

What I learned building this piece

A few things stood out while putting this together.

External integrations should fail soft. Issue creation, webhook delivery, and email sending all run async and non-blocking from the user-visible moderation action. If GitHub is unreachable, the post still gets approved. If email fails, the status still updates. The product database is the source of truth, and the integrations enhance it.

Subscribe to as little as possible. The webhook only listens for issues events, and the handler only acts on closed and reopened. Every event you do not handle is a future log line, a future bug, a future maintenance question. Smaller surface area is easier to reason about.

Identity layers should converge late. Anonymous emails and registered user IDs both live in the subscriptions table without being merged. They only resolve to a real recipient at the moment we send the email. That keeps the data model simple and avoids brittle account-linking logic.

The unsubscribe link is part of the product, not an afterthought. It needs to be in every email, work without an account, and be a one-click flow. Treating it as an edge case is how products end up with angry users and ignored mail.

What's next

In the next article, I will cover the changelog: how completed posts roll into a public, shareable list of what shipped, how it stays in sync with the board automatically, and why I decided the changelog is the changelog page rather than a separate authoring surface.

If you are building something similar, my takeaway from this part is simple:

The loop is the product.

Submission, voting, and prioritization all matter, but they only become valuable when the user who left feedback hears that something happened. Build that round trip first, and the rest of the product gets clearer.

I built two Apify actors that scrape U.S. Congress trading data — directly from government sources, no QuiverQuant

Fatih İlhan — Thu, 30 Apr 2026 20:16:24 +0000

TL;DR

I built two Apify actors that pull every U.S. Senate and House Periodic Transaction Report (PTR) directly from official government sources, parse them into a clean JSON dataset, and expose them through Apify's API. They replace QuiverQuant's congressional-trading endpoint at roughly 1/10th the cost.

Senate actor: apify.com/seralifatih/congress-trading-pipeline
House actor: apify.com/seralifatih/congress-trading-pipeline-1

Both actors emit the same schema. Run one, the other, or chain them.

Why I built this

I had a personal trading dashboard pulling congressional trade data from QuiverQuant. $30/month, fine. But the API kept returning slightly different shapes between endpoints, occasional 500s during market hours, and the per-transaction granularity I needed sat behind a higher tier.

The data itself is public domain. The STOCK Act (2012) requires every member of Congress to file every trade within 45 days. The Senate publishes them at efdsearch.senate.gov. The House publishes them as a daily-updated ZIP file at disclosures-clerk.house.gov. Both free, both public, both indexed by every aggregator that resells them.

Two weekends of work later, I have my own pipeline.

Output schema

Every transaction from both actors normalizes to this shape:

{
  "id": "4d6016b44239f646476ffac6798f21ae3e32c8ed75ea6c5b50a0bbdf9e5d3296",
  "politician": "Mark Alford",
  "transaction_date": "2026-03-16",
  "filing_date": "2026-03-31",
  "ticker": "AMZN",
  "asset_name": "Amazon.com, Inc. - Common Stock",
  "asset_type": "Stock",
  "type": "sell",
  "amount_min": 1001,
  "amount_max": 15000,
  "owner": "self"
}

Field	Notes
`id`	SHA-256 of `politician\
{% raw %}`type`	`buy` or `sell` (House `Purchase`/`Sale (Full)`/`Sale (Partial)` mapped)
`amount_min` / `amount_max`	Cents-precise bounds parsed from PTR brackets (`$1,001 - $15,000`)
`amount_max`	`null` for unbounded "Over $X" disclosures
`ticker`	`null` for bonds, municipals, structured notes
`owner`	`self` / `joint` / `spouse` / `child`

The id lets you idempotently sync into your own DB without seeing duplicates across runs.

How they work

Senate actor

The Senate's disclosure system is a Django app at efdsearch.senate.gov. Three things make it tricky:

Akamai bot protection. Direct curl gets a 403. Even with a residential proxy, Apify's default datacenter pool gets blocked.
Terms-acceptance gate. Every session must POST prohibition_agreement=1 to /search/home/ before any disclosure URL works. The session cookie expires fast and silently.
Two-stage data flow. The DataTables search endpoint at /search/report/data/ returns filings (one row per PTR document). Each row links to a separate detail page that contains the actual transactions.

The actor handles all three:

// Pin to a single residential exit IP for the whole run
const sessionId = `senate_${Date.now()}`;
const proxyUrl = await proxyConfig.newUrl(sessionId);

Why pinned IP? Django keeps prohibition_agreement state per-IP. If Apify rotates exit IPs between requests, the second request gets redirected back to the agreement page even though our cookie jar has the right session ID.

// Walk the redirect chain manually so we capture Set-Cookie at each hop
const client = axios.create({ maxRedirects: 0 });
client.interceptors.response.use((res) => {
  const setCookie = res.headers['set-cookie'];
  if (setCookie) jar.setCookie(setCookie, res.config.url);
  return res;
});

Axios's built-in redirect follower drops Set-Cookie headers from intermediate responses. Django sets the session cookie on the 302 returned by the agreement POST, so we have to walk the chain ourselves.

For each PTR detail page, the actor parses the HTML transaction table with cheerio. About 240 transactions per 30-day window across ~34 active filers. The whole run takes ~110 seconds.

House actor

The House serves disclosures very differently — a single ZIP file with everything:

https://disclosures-clerk.house.gov/public_disc/financial-pdfs/2026FD.zip

Inside: an XML index plus thousands of individual PTR PDFs. No login, no Akamai, no proxy needed.

The flow:

ZIP download → XML index → filter to PTRs (FilingType='P') → fetch each PDF → pdf-parse → regex

The XML index gives you politician + filing date + a DocID. The PDF lives at https://disclosures-clerk.house.gov/public_disc/ptr-pdfs/2026/<DocID>.pdf.

Parsing the PDFs is where it gets interesting.

The PDF parsing problem

House PTR PDFs are machine-generated, but the text-extraction order is chaotic. After running them through pdf-parse, here's what one transaction row looks like:

Amazon.com, Inc. - Common Stock
(AMZN) [ST]
S (partial)03/16/202603/16/2026$1,001 - $15,000
F     S    : New
S         O : Putnam Investments

A few things to notice:

The asset name is on a separate line from the ticker
The transaction type, two dates, and amount range all run together with no whitespace separators
Header text contains null bytes (F\x00\x00\x00 S\x00\x00\x00) — a PDF font-glyph hack
Comment blocks (Filing Status: New, Subholding Of:, Description:) bleed between rows in unpredictable ways

I ended up with a marker-anchored parser:

// Find every (TICKER) [XX] line — these are the row anchors
const MARKER_RE = /(?:\(([A-Z][A-Z0-9.\-]{0,5})\)\s*)?\[([A-Z]{2})\]/;

// Tx-row: type + date + date + amount, all glued together
const TX_RE = /(S\s*\(partial\)|P|S|E)\s*(\d{1,2}\/\d{1,2}\/\d{4})\s*(\d{1,2}\/\d{1,2}\/\d{4})\s*\$([\d,]+)\s*-\s*\$([\d,]+)/;

For each marker, walk backward up to 5 lines to collect the asset name, hard-stopping at any:

Other marker (next row above)
Transaction row (this would be the previous row's data)
Comment-block keyword (Filing Status:, Description:, shares sold @)
Owner-code prefix (SP/DC/JT stuck to the start)

The SP prefix detection has a fun gotcha: SPDR ETFs would falsely match. Fix:

// Only strip SP/DC/JT when followed by [A-Z][a-z] (real word boundary)
// "SPDR" → uppercase next, no strip. "SPApple" → strip → "Apple".
const m = name.match(/^(SP|DC|JT)([A-Z][a-z])/);

Plus a small post-processor for known truncations:

if (/\s-\s*Common$/i.test(assetName)) assetName += ' Stock';
else if (/Inc\.\s+New$/i.test(assetName)) assetName += ' Common Stock';

End result: ~95% of records parse cleanly. The remaining 5% are scanned PDFs (older filings — the parser logs them and moves on; OCR is Phase 2).

Architecture

Both actors share the same shape:

Fetch → Parse → Normalize → Dedup → Push to Apify Dataset

Normalize: maps each source's quirks (Senate's "Purchase"/"Sale (Full)" vs House's P/S (partial)) into the unified buy/sell enum, parses dollar ranges into integers, ISO-8601s the dates.
Dedup: hashes politician|transaction_date|asset_name|amount_min|amount_max to SHA-256. Same key in two runs = same id = no duplicate dataset entry.
Push: Actor.pushData() straight into the default dataset. Persisted by Apify, queryable via API.

Both actors are independent Node + TypeScript projects living in their own GitHub repos. Apify links each repo as a separate actor, builds it on push, runs on a schedule.

Schedule both for every 6 hours. Each run takes 1-3 minutes. Combined daily cost: under $2.

Using the data

Run either actor manually or hit the API:

# Trigger Senate run
curl -X POST "https://api.apify.com/v2/acts/seralifatih~congress-trading-pipeline/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"fetchDaysBack": 30}'

# Pull dataset items
curl "https://api.apify.com/v2/datasets/<dataset-id>/items?token=YOUR_TOKEN&format=json"

Or use Apify's Node SDK to consume the dataset directly in your app:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const { items } = await client
  .dataset('senate-dataset-id')
  .listItems({ limit: 200 });

const recentBuys = items.filter(t => t.type === 'buy');

Both datasets share the schema, so you can .concat() them and treat the result as one Congress-wide feed.

Lessons

A few things I'd tell past-me before starting:

Read the docs of every layer of HTTP libraries. Axios's validateStatus < 400 looks reasonable. Combined with default maxRedirects: 5, it silently swallows critical Set-Cookie headers from 302s. Took me three diagnostic runs to figure out.
Government sites have weird security postures. The Senate's Akamai layer doesn't care about your User-Agent string but cares deeply about which IP block you're coming from. Apify's residential proxy pool sails through; their datacenter pool gets walled.
PDF text extraction is lossy in non-obvious ways. pdf-parse happily returns "text" that's actually a stream of glyph references with embedded null bytes. Always normalize the output with .replace(/\x00/g, '') before any pattern matching.
Dedup early, dedup deterministically. A SHA-256 of the natural key is dumber and faster than UUIDs + a uniqueness index. Two runs of the same data produce the same set of records.
Treat the source as adversarial documentation. The Senate's terms page wording isn't just legalese — it's the literal CSRF gate. The House's PDF "Filing Status: New" boilerplate isn't human-readable bureaucracy — it's the visual marker that ends one transaction row.

Costs

For both actors combined, running every 6 hours:

Component	Daily cost
Senate compute (~110s/run × 4)	~$0.30
Senate residential proxy (~3MB/run × 4)	~$0.10
House compute (~120s/run × 4)	~$0.32
House proxy	$0 (no proxy needed)
Total	~$0.72/day

QuiverQuant's congressional plan is $10/month. This is cheaper, gives me raw transactional data instead of pre-aggregated summaries, and runs on infrastructure I control.

What's next

OCR fallback for the ~5% of House PTRs that are scanned PDFs (older filings)
Ticker enrichment — resolving asset names to tickers for the bond/muni rows where the source has no ticker column
Merge actor that consumes both Senate + House datasets and emits a unified Congress-wide stream with cross-source dedup

If you want to use the actors, both are public on Apify Store. Pricing is ~$0.40-0.50 per 1k results — a basic 30-day pull runs about $0.10. Pay-per-result, no subscription.

Data is public domain. STOCK Act compliant. No third-party vendors. Use it however you want.

I Built a Tool That Finds Mathematical Arbitrage on Polymarket — Here's How It Works

Fatih İlhan — Wed, 29 Apr 2026 12:17:44 +0000

Prediction markets are supposed to be efficient. They're not.

Here's a trade I found last week: the Lebanon Parliamentary Election had 21 candidate markets on Polymarket. Each one was a binary YES/NO bet on whether that party wins the most seats. Exactly one party will win — so the probabilities must sum to 1.0.

They summed to 0.789.

That means you could buy one share of YES on every single candidate for $0.789 total, and guaranteed receive $1.00 when the election resolves. No forecasting required. No opinion on Lebanese politics needed. Pure arithmetic.

After fees, that's a 22.7% locked return.

I built a tool that finds these automatically. Here's how.

The Math

For any event where exactly one outcome can resolve YES — an election winner, a championship, an award — the following must hold under perfect pricing:

Σ P(outcome_i = YES) = 1.0

When the sum falls below 1.0, you buy all YES positions:

Cost    = Σ YES prices       (e.g. 0.789)
Payout  = $1.00              (guaranteed — exactly one resolves YES)
Gross   = (1.0 - Σ YES) / Σ YES
        = (1.0 - 0.789) / 0.789
        = 26.7%

Subtract fees (Polymarket charges ~2% per fill, so ~4% round-trip):

Net return = 26.7% - 4.0% = 22.7%

It also works in reverse. When Σ YES > 1.0, you buy NO on every outcome:

Cost    = N - Σ YES          (sum of NO prices)
Payout  = N - 1              (every NO resolves YES except one)
Gross   = (Σ YES - 1.0) / (N - Σ YES)

The Eurovision 2026 Top 5 is a live example of this: 35 countries, Σ YES = 9.58 when it should equal 5.0. Buying NO on every country pays out 30 countries × $1 = $30, costing $25.42 — a 14% net return.

Why These Mispricings Exist

Three reasons:

1. Market fragmentation. Each candidate is a separate binary market. Market makers price each one independently. Small errors compound across 20+ markets and the aggregated sum drifts from 1.0 without anyone noticing.

2. Thin liquidity. Many of these markets have $5K–$50K of liquidity per leg, not millions. Arbitrageurs with serious capital move on, and the mispricing persists for smaller traders.

3. Unlisted-candidate risk. For winner-take-all events, part of the Σ YES shortfall is rational: traders implicitly price in "someone not listed here wins." That's real risk, not pure arb. The tool flags this in the output so you can judge it yourself.

What the Tool Does

Polymarket Multi-Outcome Arbitrage Scanner is an Apify Actor (a cloud-hosted script) that:

Fetches every open Polymarket event via the Gamma API
Classifies each event's structure — winner-take-all (Σ = 1.0), top-K (Σ = K), or neither
Filters out events that look like arb but aren't: cumulative "by date" events, price ladders, independent prop bets
Computes fee-adjusted basket returns for qualifying events
Walks the CLOB order book for each leg and verifies fillability at your target position size
Scores each opportunity 0–100 and returns structured JSON

The classifier step is the most important. Without it, the tool surfaces thousands of false positives — events like "Will Solana reach $110 / $100 / dip to $70 this month?" look like massive arb (Σ YES ≈ 0.05 → 1900% return) but the outcomes are independent, not mutually exclusive. All three can resolve YES.

The Classifier

Three structures pass the filter:

WINNER_TAKE_ALL — exactly one outcome resolves YES. Detected by keywords: nominee, nomination, winner, wins the [year], championship, next president/CEO/Pope.

TOP_K — exactly K outcomes resolve YES. Detected by: Top N in the title, or reach the final / advance to final (K=2 implied by a two-slot final).

Everything else is rejected. The rejection list is explicit:

by [month] in labels → cumulative nested structure
↑ / ↓ arrows in labels → price ladder
O/U in questions → independent prop bets
What X will Y / Which X will → multi-condition, can co-resolve

Conservative by design. It's better to miss a few valid signals than to ship noise dressed as alpha.

Sample Output

{
  "id": "alaska-governor-election-winner-buy_yes_basket",
  "event_title": "Alaska Governor Election Winner",
  "event_url": "https://polymarket.com/event/alaska-governor-election-winner",
  "resolution_date": "2026-11-03",
  "event_type": "winner_take_all",
  "expected_sum_yes": 1.0,
  "arb_type": "buy_yes_basket",
  "leg_count": 9,
  "sum_yes_price": 0.9015,
  "deviation_from_one": -0.0985,
  "fees_pct": 4.0,
  "gross_return_pct": 10.93,
  "net_return_pct": 6.93,
  "legs": [
    {
      "outcome_label": "Tom Begich",
      "side": "YES",
      "price": 0.375,
      "liquidity_usd": 24661,
      "fillable": true
    },
    {
      "outcome_label": "Bernadette Wilson",
      "side": "YES",
      "price": 0.235,
      "liquidity_usd": 14765,
      "fillable": true
    }
  ],
  "liquidity": {
    "tested_usd_per_leg": 100,
    "all_legs_fillable": true,
    "fillable_leg_count": 9,
    "total_leg_count": 9
  },
  "signal_score": 81,
  "signal_label": "Pure arbitrage"
}

9 candidates, all fillable at $100/leg, 6.9% net return locked in until election day November 2026.

Technical Stack

The whole thing is ~600 lines of Python 3.11:

pm-arbitrage/
├── src/
│   ├── core/
│   │   ├── scanner.py          # orchestrates the pipeline
│   │   ├── event_classifier.py # rejects non-exclusive structures
│   │   ├── scorer.py           # 0-100 signal scoring
│   │   └── models.py           # Pydantic v2 output schema
│   └── venues/
│       └── polymarket.py       # Gamma + CLOB API adapter

Key technical decisions:

Async-first, bounded concurrency. The CLOB order book probe fans out to all legs simultaneously but through an asyncio.Semaphore(20) to avoid OOMing the 512MB Apify container. Early version hit OOM at 352 eligible events × 10 legs each = 3500 concurrent HTTP connections.

Conservative classifier, not a blocklist. Rejecting bad structures is whitelisting-only — events must positively match a good pattern. Default is NOT_ARB. This means some edge cases get rejected, but the output is clean.

Szymkiewicz–Simpson overlap for entity matching. (In an earlier cross-venue matching phase, since scrapped.) Jaccard penalizes asymmetric sets; S-S handles the case where one side has more "noise" tokens than the other. Relevant if you extend this to cross-venue matching later.

Results From a Live Run

From a scan on April 29, 2026 with default settings (1000 events, top-K and WTA only, min $5K event liquidity):

Event	Type	Net Return	Legs
Lebanon Parliamentary Election	winner_take_all	22.7%	21
Guinea-Bissau Assembly Election	winner_take_all	11.7%	4
Alaska Governor Election	winner_take_all	6.9%	9
MLS Cup Winner 2026	winner_take_all	7.8%	25
Serie A Top 4 Finish	top_k (K=4)	81.1%	5
EPL Top 4 Finish	top_k (K=4)	28.2%	10
Eurovision 2026 Top 5	top_k (K=5)	14.0%	35
2026 Fields Medal	winner_take_all	47.3%	7

Note: the Serie A 81% return reflects the scanner computing against the expected Σ=4 baseline — it's real but assumes the 5 listed teams are the only realistic top-4 candidates. Judge accordingly.

Limitations and Honest Caveats

Implicit "Other" probability. For winner-take-all events with many candidates, Polymarket may not list every plausible winner. The probability shortfall partially reflects unlisted-candidate risk, not pure arb. Top-K events (EPL Top 4, Eurovision Top 5) are cleaner — the outcome space is rigorously closed.

Execution risk. Prices move between scan and fill. Large baskets with 20+ legs require many simultaneous fills; by the time you've filled legs 1–15, legs 16–20 may have moved. The tool tests fillability at $100/leg which is conservative — if you're sizing at $1000/leg, retest.

Liquidity depth. The CLOB test is at liquidity_test_amount_usd (default $100/leg). Higher position sizes will face more slippage. The fillable flag is binary at the test size, not a continuous depth curve.

This is not financial advice. It's an arithmetic signal engine. Trade your own book.

Try It

The actor is live on Apify:

👉 https://apify.com/seralifatih/pm-arbitrage

Run it with these settings to start:

{
  "min_net_return_pct": 0,
  "min_signal_score": 60,
  "min_event_liquidity_usd": 10000,
  "liquidity_test_amount_usd": 100,
  "max_days_to_resolution": 365,
  "max_events_to_scan": 1000,
  "output_limit": 25
}

min_net_return_pct: 0 filters to pure arbitrage only — positions where you profit even after fees regardless of outcome. min_signal_score: 60 cuts marginal low-liquidity results.

Source code: github.com/seralifatih/pm-arbitrage

What's Next

The obvious extension is adding more venues. Kalshi has the same multi-outcome structure on US elections and sports. Cross-venue arb (same event, different prices on Polymarket vs Kalshi) is the gold standard but requires a much better title-matching system than raw string similarity — I built one using entity extraction + Szymkiewicz-Simpson and it still struggles on the short, diverse titles both venues use.

The more tractable extension is real-time alerting — running the scanner on a 5-minute schedule and pushing Telegram/Discord notifications only when new high-score opportunities appear. If you want to build on top of this via the Apify API, all the pieces are there.

Questions? Leave a comment.

Building LoopSignal Part 2: The Public Board, Voting, and Prioritization Without the Overhead

Fatih İlhan — Thu, 16 Apr 2026 20:33:55 +0000

In part one, I talked about the very first thing a user does with LoopSignal: submit feedback anonymously, with as little friction as possible.

A quick update since that post: LoopSignal is now listed on GitHub Marketplace. That felt like a good milestone to mention before going deeper into how the product is built.

But a submission going into a void is not a product. It is a contact form.

The next piece was building the public board — the place where approved feedback becomes visible, where users vote, where statuses are tracked, and where a team can see what their users actually want.

This article is about how I built that, and the decisions that shaped it.

What the public board needed to do

Before I wrote any UI, I listed what the board actually had to accomplish.

It needed to:

show only approved posts, not raw submissions
let anonymous users vote without creating an account
let logged-in users have their votes persisted properly
support sorting by votes and by recency
support filtering by status and category
link to a changelog for completed items
work as both a standalone page and eventually an embeddable widget

That is a reasonable surface area for a core feature. Not small, but not bloated either.

The page is a server component

The public board page at /p/[slug] is a React Server Component. That was an easy call.

The page needs data from four places before anything renders:

the project itself (name, settings, categories)
the approved posts, filtered and sorted
the current user's existing votes (if they are logged in)
the comments attached to visible posts

Doing all of that on the server means zero loading states for the initial render. The page arrives fully hydrated. The only client-side work is the vote button itself.

I fetch the project and the auth session in parallel, then the posts and existing votes in parallel after that:

const [{ data: project }, { data: { user } }] = await Promise.all([
  admin.from("projects").select("*").eq("slug", slug).single(),
  supabase.auth.getUser(),
]);

const [{ data: posts }, userVotes] = await Promise.all([
  postsQuery,
  user
    ? admin.from("votes").select("post_id").eq("user_id", user.id)
    : Promise.resolve({ data: [] }),
]);

The result is that on a cold load, the page shows the right content — including whether the user has already voted on something — without any client round-trips.

Moderation is what makes the board clean

This goes back to a decision made in part one.

Every submission starts as pending. A team member reviews it and either approves or rejects it before it appears publicly.

On the board page, the query is always filtered to approved posts only:

let postsQuery = admin
  .from("posts")
  .select("*")
  .eq("project_id", typedProject.id)
  .eq("moderation_status", "approved");

This is a small thing, but it has a big product effect.

A public board without moderation becomes a mess within a week. Duplicate posts, nonsense, spam, test submissions from your own team. Moderation keeps the board worth reading, which keeps users worth engaging.

The moderation queue lives in the dashboard, which I will cover in a later article.

Voting without an account

This was the hardest design problem in the board.

On one hand, I wanted voting to work for logged-in users with proper persistence. On the other hand, I did not want anonymous users to be blocked from participating. Voting is how the board surfaces signal. Blocking anonymous votes would severely reduce the data quality.

The solution I landed on: a localStorage-based voter key for anonymous users, combined with proper user ID tracking for authenticated ones.

When the VoteButton mounts, it reads or generates a UUID from localStorage:

function getVoterKey(): string {
  let key = localStorage.getItem("ls_voter_key");
  if (!key) {
    key = crypto.randomUUID();
    localStorage.setItem("ls_voter_key", key);
  }
  return key;
}

That key gets sent with every vote action. On the server side, the action checks whether there is an authenticated user. If there is, the vote is recorded against the user ID. If not, the voter key is used instead.

This means:

Anonymous users can vote, and their votes persist across page reloads on the same browser
Logged-in users always have their votes properly tracked
The same post cannot be double-voted from the same source

It is not a perfect anti-fraud system. Someone who clears localStorage can vote again. But for the scale LoopSignal is targeting — small product teams, not large open-source projects — it is the right tradeoff. Friction-free participation is worth more than bulletproof deduplication at this stage.

Optimistic updates in the vote button

The VoteButton is a client component. When a user clicks it, the count updates immediately before the server confirms anything.

function handleVote() {
  const newVoted = !hasVoted;
  setHasVoted(newVoted);
  setVoteCount((c) => (newVoted ? c + 1 : Math.max(0, c - 1)));

  startTransition(async () => {
    const result = await voteOnPost(postId, voterKey);
    if (result.error) {
      setHasVoted(!newVoted);
      setVoteCount((c) => (newVoted ? Math.max(0, c - 1) : c + 1));
    }
  });
}

If the server action fails, the state reverts. This gives the button an instant feel while still being correct.

I use useTransition here rather than a loading spinner. The button stays interactive during the transition, which is better for something this small. A spinner on a vote button would feel like overkill.

Sorting and filtering without full-page reloads

The board supports two sort modes — most voted and newest — and two sets of filters: status and category.

I deliberately chose URL-based filtering rather than client-side state.

Every filter and sort option is a link, not a button that updates local state. Clicking "Planned" changes the URL to /p/my-project?status=planned. The server component re-runs with the new params and returns a fresh set of posts.

const { status: statusFilter, sort: sortParam, category: categoryFilter } = await searchParams;

This approach has a few advantages that felt worth the slightly more complex URL building:

Filtered views are bookmarkable and shareable
Browser back/forward works naturally
No hydration mismatches
No client-side data re-fetching logic to maintain

The tradeoff is that filter changes do cause a full navigation. For this kind of read-heavy, low-interaction page, that is perfectly acceptable. Nobody is rapidly toggling filters on a feedback board.

Prioritization without becoming roadmap software

This was a product decision as much as a technical one.

I deliberately kept prioritization simple: vote counts, visible on the board, sortable, filterable by status. That is it.

No scoring formulas. No weighted votes. No segment breakdowns. No "effort vs. impact" matrix.

There are products that do all of that, and they are right for certain teams. But for the kind of user LoopSignal is built for — an indie developer or a small team — that kind of overhead gets in the way rather than helping.

The premise I kept coming back to: if something has 47 votes and the next item has 12, you do not need a scoring algorithm to tell you what users want.

The most useful prioritization tool at this scale is just visibility. See what people asked for, see how many people asked for it, see what you have already committed to. That is enough.

If a team grows to a size where they need weighted scoring and segment filtering, they will probably need a different product. That is fine.

Status labels bridge feedback and roadmap

Each post has a workflow_status that the team controls: open, planned, in progress, completed, closed.

These are visible on the public board as badges on each card.

const statusLabels: Record<string, { label: string; variant: ... }> = {
  open: { label: "Open", variant: "secondary" },
  planned: { label: "Planned", variant: "default" },
  in_progress: { label: "In Progress", variant: "default" },
  completed: { label: "Completed", variant: "outline" },
  closed: { label: "Closed", variant: "outline" },
};

This is the minimum viable roadmap view.

Users can see what is being worked on and what has shipped without the team having to maintain a separate roadmap page. The feedback board is the roadmap, with statuses as the signal.

That status change is also what drives the changelog — but that is for a later article.

Admin comments as public replies

One subtle feature: team members can leave comments on posts, and those comments appear on the public board under the post they belong to.

It is rendered simply:

{(commentsByPost.get(post.id) || []).map((comment) => (
  <div className="flex gap-2 text-sm bg-muted/50 rounded-md p-2 mt-1">
    <MessageSquare className="h-3.5 w-3.5 mt-0.5 text-primary shrink-0" />
    <div>
      <p className="text-foreground">{comment.body}</p>
      <p className="text-xs text-muted-foreground mt-0.5">
        Admin reply · {new Date(comment.created_at).toLocaleDateString()}
      </p>
    </div>
  </div>
))}

This is a small thing that has an outsized effect.

When users see a team member has replied to a request — even just to say "good idea, we will think about it" or "this is planned for next sprint" — it signals that the board is being read by real people.

Empty feedback boards feel abandoned. Boards with admin replies feel alive.

What I learned building this piece

A few things became clear while working on it.

Server components are the right default for data-heavy pages. The board does a lot of reading — project, posts, votes, comments — and doing all of it on the server gives a much better first-render experience than a loading skeleton and three client fetches.

URL-based filtering ages well. Client-side filter state always needs special handling: back button, shareability, initial load from URL. URL-based filtering just works for all of those, without the extra logic.

Anonymous voting is worth the complexity. The voter key approach adds some nuance, but the alternative — blocking anonymous votes — would have killed participation on boards for products that do not require user accounts. The friction cost is too high.

Keeping the board simple keeps it useful. Every time I thought about adding a feature to the board — trending scores, vote breakdowns by country, satisfaction ratings — I asked whether it would make the core use case better or just noisier. Most ideas got cut. The ones that stayed are the ones in this article.

What's next

In the next article, I will go into the GitHub integration: how a team can link an approved feedback post to a GitHub issue with one click, and how issue events automatically close the feedback post and notify users when the issue is resolved.

That is the "loop" in LoopSignal — and it is the part of the product I am most happy with.

If you are building something similar, my takeaway from this part is simple:

Start with the board that shows what users want.

The prioritization and roadmap features only matter after you have real signal coming in.

How I Built Instagram Intelligence Suite for IG Growth

Fatih İlhan — Wed, 15 Apr 2026 19:02:01 +0000

Instagram research usually breaks down in the same three places:

You find profiles, but you still do manual qualification.
You know a brand or influencer is active, but you cannot track story behavior cleanly.
You see high-intent comments under posts and reels, but nobody turns them into structured leads.

That is exactly why I split the workflow into three focused APIs instead of trying to force everything into one oversized scraper:

IGLead for profile qualification
IG_story_snapshot for story activity monitoring
IG_comment_lead for comment-to-lead extraction

All three are built around Apify actors, but the bigger idea is simple: treat Instagram intelligence like a pipeline, not a one-off scrape.

Why I split this into three APIs

A lot of Instagram tools try to do discovery, enrichment, monitoring, and lead scoring in one place. That sounds convenient until the inputs, auth requirements, and output formats start fighting each other.

I wanted each API to answer one clean question:

IGLead: Is this profile worth contacting?
IG_story_snapshot: Is this profile active on stories right now?
IG_comment_lead: Which commenters look like real demand?

That separation makes the stack easier to maintain, easier to schedule, and easier to plug into downstream automations.

1. IGLead: qualifying influencer and creator profiles before outreach

IGLead starts with a list of Instagram usernames or profile URLs and turns them into scored outreach candidates.

Instead of just scraping follower counts, it combines multiple signals:

follower count
recent post engagement
engagement rate
business email detection from public bio text
niche keyword matching
verification status
a final lead score and recommendation

One thing I especially like in this API is that the scoring is not flat. Engagement expectations change depending on account size. A micro creator should not be evaluated like a mega influencer, so the actor adjusts its thresholds by tier.

It also uses multiple extraction paths for reliability:

Instagram web profile API
feed endpoints when timeline media is incomplete
HTML parsing fallbacks
meta tag parsing for follower and post counts

That matters because Instagram is rarely stable enough for a single-method scraper.

Example `IGLead` input

{
  "profiles": ["therock", "cristiano", "https://www.instagram.com/kyliejenner/"],
  "sessionId": "YOUR_SESSION_ID",
  "minFollowers": 100000,
  "minEngagementRate": 1.0,
  "requireBusinessEmail": false,
  "nicheKeywords": ["fitness", "health", "wellness"],
  "maxProfilesPerRun": 50,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

What comes back

For each profile, IGLead can return:

normalized profile data
recent post stats
average and median engagement metrics
extracted business email if it is publicly visible in the bio
niche match score
leadScore
recommendation such as contact, review, or skip

For outreach teams, that means you can stop treating every Instagram profile as equal. You can prioritize the ones that actually fit your campaign and have the engagement to justify the spend.

2. IG_story_snapshot: monitoring story activity without owning the account

Stories are one of the hardest parts of Instagram to operationalize because they are temporary, fast-moving, and usually checked manually.

IG_story_snapshot is built to answer a very specific operational question:

Does this public profile have an active story right now, and what does that story set look like?

It tracks:

whether a profile currently has an active story
how many story frames are live
image vs video composition
oldest and newest story timestamps
hours since the story sequence started
hours left until expiry
optional profile context such as follower count

This is useful for:

competitor monitoring
campaign verification
event coverage tracking
brand activity benchmarking

What I like most here is that it avoids trying to do too much. It does not pretend to give story view counts, and it does not download story content. It focuses on presence and metadata, which is the part most teams actually need for monitoring.

Example `IG_story_snapshot` input

{
  "profiles": ["nike", "adidas", "@puma"],
  "sessionId": "YOUR_SESSION_ID",
  "includeProfileContext": true,
  "maxProfilesPerRun": 50,
  "maxRequestsPerMinute": 15,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

What comes back

The output is centered around a few operational fields:

active_story
story_count
story_frames
story_metadata
story_age_hours
hours_left_until_expiry

That makes the API especially useful for scheduled runs. If you execute it every hour, you can build a clean timeline of who is posting stories, how often they post, and whether they lean more toward video or image content.

3. IG_comment_lead: turning Instagram comments into lead intelligence

IG_comment_lead is the most directly sales-oriented API in the stack.

The idea is straightforward: people reveal intent in comments all the time. They ask about price, shipping, details, availability, or how to order. Most teams read those comments manually, if they read them at all.

This API takes Instagram post or reel URLs, fetches comments, and scores commenters based on lead relevance.

The pipeline includes:

input validation for post and reel URLs
authenticated scraping with sessionId
fallback comment extraction strategies
keyword-based intent scoring
lightweight sentiment scoring
spam checks
deduplication by username
early stopping once the target lead count is reached
a final analytics summary for the whole run

I also like that this API is optimized for cost control. You can cap comments per post, define a minimum lead score, and stop the run as soon as enough leads are found.

Example `IG_comment_lead` input

{
  "postUrls": [
    "https://www.instagram.com/p/C3xYz1234Ab/",
    "https://www.instagram.com/reel/C3xYz5678Cd/"
  ],
  "sessionId": "YOUR_SESSION_ID",
  "cookie": "sessionid=...; csrftoken=...; ds_user_id=...;",
  "maxCommentsPerPost": 500,
  "targetLeads": 30,
  "minLeadScore": 0.6,
  "debugComments": false
}

What makes this one interesting

The comment fetch flow does not rely on a single endpoint. It tries multiple strategies:

GraphQL queries
shortcode-based REST endpoints
mobile-style REST endpoints

That gives it a better chance of surviving endpoint instability.

On top of extraction, it enriches leads with:

buyer_intent_score
engagement_score
likely customer flag
extracted keywords
inferred niche
inferred geography

At the end of a run, it also pushes an analytics summary with:

total comments processed
total leads found
lead rate
top commenters
intent distribution
sentiment distribution
top keywords
per-post breakdown

That means the output is useful for both direct lead capture and campaign analysis.

How the three APIs work together

The fun part is not each API in isolation. It is the workflow they create together.

A practical sequence looks like this:

Use IGLead to qualify creators, influencers, or niche accounts before outreach.
Use IG_story_snapshot to monitor who is actively posting stories right now.
Use IG_comment_lead on posts and reels in your niche to surface warm demand from commenters.

That gives you three different layers of Instagram intelligence:

profile quality
current activity
audience intent

In other words, you can answer:

Who should I contact?
Who is active right now?
Who is already asking buying questions?

Implementation notes

All three projects are built around the same practical philosophy:

use Apify actors for deployment and scheduling
use Crawlee for request orchestration
use Playwright when browser context is needed
keep concurrency controlled to reduce blocks
use fallback strategies instead of trusting one endpoint
support session cookies when Instagram requires authentication
preserve debug artifacts when extraction fails

For IGLead, that means debug HTML and screenshots when profile parsing breaks.

For IG_story_snapshot, that means API-first story detection with a visual fallback for story presence.

For IG_comment_lead, that means endpoint fallback plus a summary record at the end so a run is not just raw data, but something closer to decision-ready output.

What I would improve next

If I keep iterating on this stack, these are the next areas I would push:

cross-actor orchestration so leads can move automatically from one API to the next
historical storage for story activity trends over weeks instead of single snapshots
richer commenter enrichment for repeat engagement across multiple posts
better dashboarding on top of the analytics summary

The core scraping part is useful, but the real leverage comes from building a repeatable operating system around it.

Final thoughts

The biggest lesson from building IGLead, IG_story_snapshot, and IG_comment_lead is that Instagram automation becomes much more useful when you stop thinking in terms of "scrape a page" and start thinking in terms of "answer a business question."

Each API here is narrow on purpose:

IGLead answers qualification
IG_story_snapshot answers activity
IG_comment_lead answers intent

Put together, they form a lightweight Instagram intelligence stack for outreach, competitor research, and lead generation.

If you are building in this space, I would strongly recommend resisting the urge to turn everything into one monolith. Small, composable APIs are easier to trust, easier to debug, and much easier to turn into real workflows.

You can find all of my APIs here: https://apify.com/store/categories?search=seralifatih