My Actor worked for humans and failed for agents: a bug postmortem in four acts

On August 12, 2026, I looked at the public stats for my SAM.gov Contract Opportunities Scraper (an Apify Actor that searches US federal contract notices) and saw 22 failed runs in the last 30 days. My own runs? 7 lifetime, all green. My local test harness? Passing for weeks.

The Actor had been quietly broken since late July. The platform's health checks had been failing daily since roughly July 21, and I hadn't looked. Meanwhile my Florida license Actor, the only Actor in my portfolio earning money, had failed 9 of its 48 runs in 30 days while its daily health check kept passing. Sit with that ratio for a second: the check that ran every day was green, and one run in five was not.

A note if you click that listing today: its 30-day stats still show the failures from this incident. All 12 of them are from a single day, August 6, on the build this article is about. Every run since the August 12 fix has succeeded, including one from a stranger this morning. The number rolls back on its own as the window moves, which is its own small lesson about what public quality metrics actually measure.

Both Actors "worked" in every test I ran. Both were failing the callers that mattered. And increasingly, those callers are not humans clicking around Apify Console. They're AI agents hitting the Actor through an API, whether via the Model Context Protocol (MCP), Apify's MCP server, or my own us-govdata-mcp wrapper. Agents don't retry creatively. They don't eyeball suspicious output. They trust your schema, your pricing, and your exit code.

Here is what broke, what I changed, and what I'd do differently. Every build number, run ID, and dollar amount below is real.

Act one: the dataset schema that lied

My SAM.gov Actor declares a dataset schema, the platform-validated contract for every record it outputs. placeOfPerformance.city is a string or null. primaryContact.email is a string or null. And so on.

SAM.gov's notice-detail API had other plans. For some notices it returns nested objects where strings should be: a city as {}, or as {"code": "0"} instead of a name. My mapping code passed those objects straight through, and I never questioned it, because nothing I ran ever complained.

On the Apify platform, pushing a record that violates the dataset schema doesn't just drop that record. It aborts the whole pushData batch with "Schema validation failed." Runs died mid-flight. The platform's automated health checks started failing daily, and the Actor got auto-flagged as broken.

Here's the part that stung: local runs never validate the dataset schema. My local harness ran the same code against the same API and passed every time. The only environment where the bug existed was the one with paying users in it.

The fix, shipped in build 0.1.4 on August 12, is a one-function idea: no value reaches a string-typed output field without being coerced. From src/main.js:

/**
 * Coerce an API value to a trimmed string or null - never an object/array.
 * The SAM.gov detail API sometimes returns nested objects where a string is
 * expected (e.g. placeOfPerformance.city as {} or {"code":"0"} instead of a
 * name). Objects leaking into string fields fail the platform's dataset
 * schema validation, which aborts the whole pushData batch ("Schema
 * validation failed") - the bug that got this actor auto-flagged as broken
 * (local runs never validate the dataset schema, so they passed).
 */
const asText = (value) => {
    if (typeof value === 'string') return clean(value);
    if (typeof value === 'number' && Number.isFinite(value)) return String(value);
    return null;
};

(clean(), referenced above, is a two-line helper: trim the string, return null when nothing is left.)

And every mapping line now goes through it, with fallbacks for the API's shape-shifting:

record.placeOfPerformance = {
    street: asText(pop.streetAddress) ?? asText(pop.street),
    city: asText(pop.city?.name) ?? asText(pop.city),
    state: asText(pop.state?.code) ?? asText(pop.state),
    zip: asText(pop.zip?.code) ?? asText(pop.zip),
    country: asText(pop.country?.code) ?? asText(pop.country),
};

Boring code. That's the point. The dataset schema is a promise, and for an agent consumer it's the only promise. An agent building a pipeline on my output fields has no human in the loop to notice a city that's suddenly an object. Coerce at the boundary, always, and treat upstream APIs as adversarial about types even when they're government-official.

The lesson, in one line: your dataset schema is validated by the platform, not by your laptop. If your output can be shaped by an upstream API, run schema validation yourself in CI, or ship the coercion layer from day one.

Act two: charging 700 for 500

While reproducing the schema bug I found something worse in the billing data. My Actor uses pay-per-event (PPE) pricing: one contract-opportunity event, at $0.004, per record delivered.

I ran two reproduction runs on build 0.1.3 with maxResults: 500. Both delivered exactly 500 records. Both charged 700 events. So I did the arithmetic a customer would do: 700 events at $0.004 is $2.80 billed for $2.00 of delivered data, an extra $0.80 (40%) per run, hiding in the interaction between charging, the aborted pushData batches from act one, and the Actor's CSV fallback path. And per the failure mode above, a run could charge events and then die, leaving a user paying for a failed run.

It's worth walking the incentive structure here, because it's what makes this bug a different animal from a crash. On PPE pricing, every event my code posts is me billing a stranger's account. Apify takes its platform cut and I keep the rest. The user's only protection is that my accounting is honest. Mine wasn't. Not by intent, but the user's card can't tell the difference.

The chargedEventCounts on my own repro runs made the signature undeniable:

Run	Build	Delivered items	Charged events
vaIzPUYUmZbzzuM4I	0.1.3	500	700
1W2kINLpaI2miHz1m	0.1.3	500	700
Z7b0J8yEFuT2XCeSq	0.1.4	500	500
cnr4Y0rP0kUIlXTPN	0.1.4	44	44

The fix in 0.1.4 centralizes the invariant in one function that everything routes through. Charge first, then push only what was actually charged. Delivered items can then never exceed charged events, and the pushed count is whatever the platform confirms it charged, regardless of what my own bookkeeping expected:

/**
 * Push dataset items, charging one event per item when pay-per-event is active.
 * Never pushes more items than were successfully charged.
 */
async function pushCharged(items, eventName) {
    if (items.length === 0) return { pushed: 0, limitReached: false };
    if (!isPayPerEvent()) {
        await Actor.pushData(items);
        return { pushed: items.length, limitReached: false };
    }
    const { chargedCount, eventChargeLimitReached } = await Actor.charge({ eventName, count: items.length });
    if (chargedCount > 0) await Actor.pushData(items.slice(0, chargedCount));
    return { pushed: chargedCount, limitReached: eventChargeLimitReached };
}

(isPayPerEvent() is a one-liner around the SDK's charging manager; it returns false whenever the run isn't on PPE pricing, so the same code works on free and rental runs.)

A footnote I owe you, because this article is about billing honesty. This 0.1.4 version later turned out to have its own edge case. At the run's charge ceiling the SDK grants a partial charge, and pushData can then silently drop the batch, billing records that never land: the same harm, arriving by a different door. Build 0.1.8 replaced the pair with the SDK's atomic Actor.pushData(items, eventName), which charges exactly what it delivers. If you copy a pattern out of this article, copy that one.

Then came the uncomfortable part: making affected users whole. I learned that the developer API cannot enumerate other users' runs. GET /v2/acts/{actorId}/runs returns only runs started by my own account. 7 runs, all mine, while the public stats showed 41. The affected run IDs exist only on Apify's side.

So I wrote support a charge signature instead of a run list: any run of my Actor in the window, by any user but me, that either failed with billed PPE events, or succeeded with more billed contract-opportunity events than clean dataset items. I asked for the refunds to be debited from my payout. I sized the exposure at $1–$7, with an absolute ceiling around $45. Trivial money. But a pay-per-result model is the trust, and being prompt and boring about making people whole is part of the product, not a gesture on top of it.

Support answered the next day, and the answer is the reason I asked instead of guessed. Most of those failed runs were Apify's own test system, which runs every Store Actor daily on its default input. The failures I had read as burned customers were largely the platform telling me my Actor was broken, in the one channel I wasn't reading. The real user exposure was a handful of runs from a single free account: $0.17, which support credited directly. Two lessons, and I'd rank them in this order. My blast radius was small because I caught this while I had almost no users, not because I was careful. And the numbers a developer can see (22 failed runs, a 19% failure rate) do not distinguish a robot's daily probe from a paying customer's ruined afternoon. I sized my exposure at up to $45 and was off by two orders of magnitude, in the lucky direction.

The lesson: on PPE pricing, your charge accounting is part of your public interface. Test it with the same rigor as your output. chargedEventCounts on your own runs is the ground truth; diff it against your dataset item count in CI if you can.

Act three: the configuration that lives only in Console

Two smaller incidents, same root cause: some Actor properties don't live in your repo.

The swapped billing events. Earlier in the summer, my NPI and FMCSA Actors (National Provider Identifier registry, and the Federal Motor Carrier Safety Administration's carrier census) had their PPE events swapped in the Console monetization config (NPI charging FMCSA's event and vice versa). The code was correct. The config, which lives only in Console and can't be pushed from the repo, wasn't. The fix was manual, followed by a paid smoke-test run of all the Actors, watching the logs for 'unknown event' warnings. Event names in Actor.charge() must match the Console config character for character, and nothing in your repo will tell you they don't.

The title that wouldn't ship. In an SEO audit I'd rewritten all my Actor titles around what buyers actually search ("lookup", "verification", "no API key"). I updated every .actor/actor.json, pushed, verified the builds succeeded. The live store titles didn't change. For a published Actor, apify push updates code and versions but does not propagate title or description to the store listing; those are Publication-tab metadata, edited in Console. I verified this the hard way, via the public store API after pushing: modifiedAt updated, title unchanged.

And when the titles finally went in through Console, they had to shrink. My approved SEO title for the SAM.gov Actor was 77 characters; what's live is 57. The Publication title field caps at 63 characters, and every title in my portfolio now sits at or under that line. The 63-character reality annoyed me and then improved me: agents discovering Actors as tools see the title and description first, and a title that states the capability in 10 words beats one that lists every keyword.

The lesson: know which parts of your Actor are code and which are Console state. Monetization events, published titles and descriptions, and SEO fields won't version, diff, or deploy with your repo. Keep a checklist for them, because your repo's green build says nothing about them.

Act four: the proxy that cuts the stream at 60 seconds

My Florida license Actor streams the official license extracts of Florida's Department of Business and Professional Regulation (DBPR), filters rows, and charges per record returned. The construction file alone runs 46.3 MB. These are public-records files the state publishes for this very purpose, and the Actor fetches each one once per run at normal download rates; the proxy business below is about reachability, not volume. In August the Actor kept passing its daily platform health check while 9 of its 48 runs failed. Call it one in five. Whose runs those were, I can't tell you. The developer API shows me my own runs and aggregate counts, never whose run died. It's the same blind spot that had me overestimating the SAM.gov exposure by two orders of magnitude in act two.

I reproduced it with a whale-shaped input: Miami-Dade county, certified general contractors (license type CGC), maxResults: 1000, which needs a multi-megabyte scan of the file. DBPR's content delivery network (CDN) blocks datacenter IPs, so the Actor escalates to Apify Proxy. And in every one of my repro runs, the proxied connection died at roughly 60 seconds mid-stream. My code treated the truncated stream as end-of-file. The CSV parser (csv-parse) hit the truncation mid-quoted-field and crashed the run: Quote Not Closed ... line 15263 (repro run kI1VkPKBOJcYs3JqB, failed at 1m05s). Worse: 400 of the 1,000 records had already been pushed and charged. A paying user, paying for a failed run. The worst outcome available on this pricing model.

Why did the health checks pass? They fetch 25 records, match early, and close the stream well before the 60-second cut. A small-input health check is the one kind of run guaranteed never to see a large-input failure. And there was a nastier latent case: if the cut landed on a row boundary, the run would succeed with silently truncated data. A human might notice a suspiciously short list. An agent would report it as complete.

The fix (build 0.1.5, pushed the same night) was to make the download resumable. The DBPR CDN honors HTTP Range requests (I verified the 206 responses and Content-Range headers), so on any mid-stream cut the Actor resumes from the exact byte offset on a fresh connection, feeding one continuous stream to the parser:

function requestFromOffset(url, proxyUrl, offset) {
    return new Promise((resolve, reject) => {
        // Uncompressed body so byte offsets line up with what we already consumed.
        const headers = { Accept: 'text/csv,*/*', 'Accept-Encoding': 'identity' };
        if (offset > 0) headers.Range = `bytes=${offset}-`;
        const stream = gotScraping.stream({
            url,
            proxyUrl,
            timeout: { response: RESPONSE_TIMEOUT_MS },
            headers,
            https: { rejectUnauthorized: true },
        });
        stream.on('response', (res) => {
            if (res.statusCode === 206) {
                const match = /^bytes (\d+)-\d+\/(\d+)$/.exec(res.headers['content-range'] ?? '');
                if (!match || Number(match[1]) !== offset) {
                    stream.destroy();
                    reject(new Error(`Server returned an unexpected Content-Range "${res.headers['content-range']}" for offset ${offset}`));
                    return;
                }
                resolve({ stream, totalBytes: Number(match[2]), skipBytes: 0 });
                return;
            }
            if (res.statusCode === 200) {
                // Server ignored the Range header and replayed the file:
                // skip the bytes we already have instead of double-counting them.
                const totalBytes = Number(res.headers['content-length'] ?? 0) || null;
                resolve({ stream, totalBytes, skipBytes: offset });
                return;
            }
            stream.destroy();
            reject(new Error(`HTTP ${res.statusCode} while downloading ${url}`));
        });
        stream.once('error', reject);
    });
}

(gotScraping is the got-scraping HTTP client; RESPONSE_TIMEOUT_MS is the Actor's response-header timeout constant.)

Details that mattered in practice:

• Accept-Encoding: identity matters. With compression on, your byte offsets and the server's don't line up, and Range resumption corrupts silently.
• Check that the 206's Content-Range starts at your offset. Some servers answer 206 with the wrong window.
• Handle the server ignoring Range entirely (a 200 replay) by skipping the bytes you already consumed.
• The stream is only ended once all bytes have arrived, which kills both the crash and the silent-truncation case with one invariant.
• An idle watchdog destroys a stalled socket after 60 seconds of no data. Node's pipe() does not forward source errors; without the watchdog, a dead connection hangs until the run timeout.
• Progress (rowsProcessed, pushed) checkpoints to the run's key-value store at every flush. After a platform migration the Actor fast-forwards and never double-pushes or double-charges. I verified this locally with a seeded checkpoint: it resumed at 500 pushed and delivered exactly the remaining 500.

Verification on build 0.1.5, same night:

Run	Input shape	Result	Time	Platform cost
du5uozEtuf72Tycjx	Miami-Dade + CGC, 1,000 records (the shape that failed)	1,000 records, 1 cut resumed	1m 37s	$0.011
nq26dTSyyVsGEBRPh	Full 46.3 MB scan to end-of-file (worst case)	241 records, matches the local count, 4 cuts resumed	4m 16s	$0.018
6bgjK0P7MBhzIrNjO	Health-check input, 25 records	25 records	6s	$0.001

Four cuts on a single full-file scan. For any serious input this was the everyday case, and every small test was blind to it.

The lesson: small health checks are structurally blind to large-input failures. My 6-second check could never see a cut that happens at 60.

The input schema, or: leaving the agent nothing to guess

The incidents above are about output, billing, and transport. The remaining surface is input, and here the work is preventative. My rule after this month is blunt: an agent should be able to construct a correct call from the schema alone, with nothing to guess.

Concretely, from the Florida Actor's input schema:

• Closed inputs are enums with labels. profession is an enum of 8 licensing boards with enumTitles like "Construction contractors (CILB: general, building, residential, roofing, plumbing, HVAC, pool...)". An agent cannot invent a board I don't support.
• Formats are machine-checkable. Dates carry "pattern": "^\\d{4}-\\d{2}-\\d{2}$". A malformed date fails validation with a message, instead of silently matching nothing.
• Costs are stated where the number is set. The maxResults description says "You are only charged for records actually returned." That sentence is doing billing-anxiety work for humans and budget arithmetic for agents.
• Semantics that could surprise are spelled out. The monitoring mode's licenseNumbers watch list deliberately overrides the status filters, because a license that gets suspended must not fall out of its own watch list. The description spells that out, and says why. An agent that reads it configures compliance alerts correctly on the first try.

What I'd do differently

If I were starting the portfolio again:

1. Validate output against the dataset schema in CI. The platform validates; my laptop didn't. That asymmetry cost me 3 weeks of silent failures. It's the single highest-leverage fix on this list.
2. Treat charge counts as testable output. Assert chargedEventCounts equals delivered items on a real platform run before every release of a PPE Actor.
3. Test at the scale users buy, not the scale that's convenient. My health checks were 25-record runs; my customer's were ~1,000. Everything interesting happened past the 60-second mark that small runs never reach.
4. Keep a Console-state checklist. Event names, published title, description, SEO fields, all reviewed by hand, because no diff will catch them.
5. Assume every long-lived connection will be cut, and design downloads to resume rather than restart. Range plus identity encoding plus an idle watchdog is a reusable pattern; I've since carried it into my Texas and California license Actors.

The framing I keep coming back to: humans forgive an Actor its quirks because they can see them. Agents can't see anything you didn't put in the schema, the pricing events, or the error message. Making an Actor "AI-friendly" turned out to mean making it precise. And every user, human or not, got a better Actor out of it.

If you're monetizing Actors, pull your own chargedEventCounts tonight. I thought mine were fine too.