After an Actor reboot, the Apify SDK forgets what it has already charged

I build pay-per-event Actors for the Apify Store. This week I tested what happens to charging when an Actor reboots in the middle of a run, because Actors can be rebooted or migrated during long runs. I tested a forced reboot; I have not tested a platform migration. The result surprised me, so here it is, with the fix I use.

What I tested

I wrote a small private Actor that charges one event per dataset row, then forces a reboot part-way through with Actor.reboot(). I ran it with a low "maximum cost per run" so the limit would matter, and checked the run metadata afterwards. I used Apify SDK for Python 4.0.2.

What I found

After the reboot the SDK reported zero events charged, even though a few had already been charged before the reboot. As far as I can tell, the SDK takes its charged counts from environment variables set when the container starts, so a restarted container starts from the old numbers instead of the current ones.

In one test, the run metadata ended up showing more charged events than the maximum I had set. Apify's documentation says the platform enforces the limit and that charges above the charged count are not billed, so I can't say a customer would have been billed for the extra events. What I can say is that the run's own record disagreed with the limit I had set, and my Actor had no way to notice. I'd rather not rely on the platform to quietly absorb that.

I haven't seen an existing SDK issue describing this exact behaviour, so it may be worth a report.

The fix I use

Three small pieces, all inside the Actor:

1. Keep your own record of charging. In my Actors, a CHARGE_STATE record in the run's key-value store is written as "pending" before each charge and updated to "charged" afterwards. After a restart you can read it and know what really happened, even if the SDK's counts say zero.
2. Dedupe against what you've already stored. Before pushing a row, check whether it is already in the dataset. A restart then can't charge for the same row twice.
3. Keep your own spend limit. Work out how many events you may still charge as: the user's maximum, minus the larger of your own record of earlier spend and the SDK's total, minus what you've charged since this start. Use Decimal for the money and round down. Never use floats for prices: (0.011 - 0.005) / 0.002 gives 2.9999999999999996 in floats, so rounding down would lose an event.

With those in place my test Actor stopped at exactly the limit I set ($0.011, three rows) in a run that rebooted, and a control run with a higher limit produced no duplicate rows.

How to check your own Actor

Use a throwaway copy, a low per-run cost limit, and a forced Actor.reboot() part-way through. Then read the run metadata and compare the charged counts with the maximum you set, and with the number of distinct rows you expect to have been charged. If they disagree, you have the same gap.

I'm happy to answer questions in the comments.

Comments

[0xGollum]

Useful test, and thanks for including the control run. On the dedupe step: do you read the dataset back to check, or keep a set of seen keys in the key-value store next to CHARGE_STATE? On big runs, reading the dataset back after every restart gets slow, so I'm curious which way you went.

[Russell Yapp]

Thanks. I read the dataset back, once, at the start of a restarted run, and keep just the row keys in an in-memory set. The same pass counts the items, which is what the earlier starts charged. After that each row is checked against the set before it is stored and charged, so there is no re-read per row. I have only run it on small runs (a few hundred rows), so I can't say how it behaves on a big one. A key set kept in the key-value store would be cheaper there, but it has to stay in step with the dataset: write it before the push and a crash can leave a row uncharged, write it after and a crash can double-charge it. Reading the dataset keeps one source of truth, which is why I chose it.