I run two small Apify actors that pull U.S. congressional stock trading disclosures (the STOCK Act "PTR" filings) for the House and the Senate and ...
For further actions, you may consider blocking this person and/or reporting abuse
238 rows to 416 on the same 50 filings. That's a brutal number to find on your own.
We had the mirror image of this in a product data pipeline. Instead of returning nothing, a fallback returned something: when no evidence page was found, it filled in a URL from a targets file, and for 1 product that URL was a source we'd banned. The record looked complete. We added an assertion right before the file is written, and it fired on its very first run. That's how we found the fallback at all.
Your
parse_statuscolumn is the better version, and I'd take a placeholder row over a missing one every time, since it keeps the filing and says why. Do you watch the ratio week to week? A jump inparse_failedseems like the earliest warning you'd get that the House changed its PDF format.That fallback story is scarier than mine, honestly. A missing row at least leaves a hole you might eventually notice. A filled-in row with a banned source looks finished, so nobody goes looking. I like that the assertion fired on its very first run. That's the best argument for adding one I've heard.
To answer your question: not yet, and you're right that I should. Right now every run writes the counts (parse_failed, scanned_unparsed, fetch_failed, paper filings) to its output, but I only look at them when I go digging. So they're a snapshot, not a trend.
Your point about parse_failed as an early warning is spot on. Scanned filings run at a pretty steady ~14% on the House side, so they won't tell me much. But parse_failed should sit near zero. If it jumps, either the Clerk changed the PDF layout or I broke something, and I'd want to know within a day, not a month later when someone complains.
Next step is probably a simple threshold alert: if parse_failed or fetch_failed goes above a small percentage in a run, it pings me. Thanks for the nudge.
A steady ~14% for scanned filings is a great baseline to have, because it means a jump in that one is also news. If paper filings suddenly drop to 2%, the Clerk probably changed how scans are posted, not Congress.
On the alert: pick the number first. Before you see a bad run. Once a run has gone wrong it's tempting to set the threshold just above whatever that run did. Where do you think parse_failed should trip, 1 percent or 5?
Fair point on the baseline. If scanned filings suddenly dropped to 2%, that would tell me something changed on the Clerk's side, so I'll alert on that too, in both directions.
And I'll commit to a number now. For parse_failed it's one filing, not a percentage. The first bug I found in this audit was a single filing out of 133 over 90 days, about 0.75%. A 1% threshold would have missed it, and 5% wouldn't have been close. On real data the baseline is zero, so even one is worth looking at. It usually means a new format variant, and those are cheap to fix when you catch them early.
fetch_failed is different because network blips are normal. I'll trip that at 3 or more filings in a single run, or if it shows up in two runs in a row.
I'm writing both numbers down before the next run, so I can't move them after the fact.
One filing out of 133 is a great argument for "one" over any percentage. At 0.75% a 1% rule stays quiet forever, and that bug would still be there.
Writing the numbers down first is the whole trick. Good call.
The two-runs-in-a-row rule for fetch_failed is nice too, because a single network blip and a source that has actually gone away look identical inside one run and look nothing alike across two. Will the alert go to you directly, or to the people paying for the data as well?
Me, directly. I can't reach the people paying for the data. Apify doesn't give me contact details or per-user visibility, so a proactive "heads up" to users isn't possible.
What they do get is the signal inside the data. Every row carries parse_status, so a user's own pipeline can check for non-ok rows, and each run's summary has the counts. If you consume this data, that's the thing to alert on yourself. I'd rather say that plainly than promise a notification I can't send.
Your point about two runs is why I picked it. One blip and a dead source look identical within a run, and completely different across two.
The "every filing leaves a trace" rule is the right call, and the 14% scanned baseline is a nice detail. On the billing catch: if you move to pay-per-event, it might be worth checking whether you can push the placeholder rows without attaching the charge event, so the trace stays in the dataset but only real rows hit the invoice. Curious whether you've looked at that route.
Good catch, and yes, that's exactly the gap right now. The actors use Apify's synthetic
apify-default-dataset-itemevent, which bills every row in the default dataset automatically, placeholders included (I had to correct my own README on that).The fix is the route you describe: switch to a custom event (e.g.
transaction-row) and callActor.charge()only for parsed rows, so placeholders stay in the dataset for free. In practice they're a small share of rows (one per unparseable filing, not per trade), so the cost difference is tiny, but it's the right principle: users shouldn't pay for "we couldn't read this."It's on my list. Thanks for pushing on it.
The nastiest part of this class is that all five failures look identical from outside: zero rows, green run. Nothing to alert on.
The defense I would add is a per-input row count where zero is only legal if the source truly has nothing, plus a periodic total checked against the source's own published tally. That catches "the parser ate it" before you conclude "the source is incomplete."
I hit the smaller cousin. I compared a convenience API's current weather against the airport's own report and nearly filed it as "the app lies." The gap was mostly the comparison, not the source: wind in km/h vs knots, RH derived from dewpoint, cloud as percent vs octas, readings 3 to 69 minutes apart. 3 of 10 wind pairs flipped verdict once units matched. Teardown: dev.to/maxcarnage/wttrin-says-28-k...
And yeah, the code comment was the bug. Comments age into claims nobody re-tests.