DEV Community

Discussion on: Designing Apify Actors that AI agents actually pick (and can afford)

Collapse
 
kevin_hall profile image
Kevin Hall •

Reading this from the calling side (I build GTM lists with agents, I do not publish Actors): the two things that made me trust an Actor enough to put it in a prompt last month are exactly the ones you describe. A free count or preview mode, so the agent can size a segment and report the price before it spends anything, and a README that states the limit of the data (how fresh the dates are, which months are thin) instead of hiding it. The charge cap per run is the third thing I check. With those three, "count first, then buy" becomes a one-line instruction and the agent stops being scary to point at a metered API.

The checklist at the end is good. I would add one item from the user's chair: an example input in the README that an agent can copy verbatim, because that is what it does.

Collapse
 
locaihost_data profile image
locaihost data •

Thanks Kevin, this is the view we can't get from the publishing side.

Agreed on the copy-verbatim example input: an agent uses whatever example it finds, so we're moving one runnable example to the top of each README instead of leaving it at the bottom.

On the charge cap, we build our Actors to stop cleanly at the run's max-charge limit, so a capped agent gets a partial result rather than a failed run.

Count-first is the gap. For a Maps-style list an honest count costs almost as much as the list, because the only way to count is to page through the results. So a question back, since you size segments for real: what's the smallest preview your agent would act on? "How many businesses match" (fairly cheap), or "how many of them have a dead, broken or missing website" (what our Google Maps leads Actor sells; it goes public this week, and that count is much more expensive)?

Collapse
 
kevin_hall profile image
Kevin Hall •

Good question, and the honest answer is that my agent acts on the cheap count, as long as it comes with one more number: what the next step costs at that count. "X businesses match" plus "the website check is Y per row" is enough for a go/no-go on the whole list.

For the expensive count (how many have a dead or missing website) I would not pay for a full count. What I would act on is a small priced sample, say 50 to 100 rows, with the hit rate reported, and then extrapolate. The thing to avoid is a preview that silently pages through everything; the agent should be able to say "this preview costs Z" before it runs.

So three steps, each with the price stated before it runs: count, sample hit rate, full run.

Thread Thread
 
locaihost_data profile image
locaihost data •

That's a clear spec, thank you. Count, then a priced sample with the hit rate, then the full run, and each step states its price before it runs.

We'll build the Google Maps leads Actor that way. The count step returns "X businesses match" together with the per-row price of the website check. A sample step checks a random 50 to 100 of them and reports the hit rate and the extrapolated cost of the full list. No preview that pages through everything without saying what it will cost first.

We'll reply here when it's live.

Thread Thread
 
kevin_hall profile image
Kevin Hall •

Looking forward to it. One thing I learned since, running the same three steps on a people search this morning, may save you a wrong extrapolation: the sample has to be random. The first 100 rows of a sorted result all came from one country and showed a 49% email hit rate; a seeded random 100 from the same filter showed 78%. If the sample step takes the first N businesses from Maps, the hit rate and the extrapolated cost will be off in the same way. A seed parameter on the sample step fixes it and makes the sample reproducible.

Thread Thread
 
locaihost_data profile image
locaihost data • • Edited

Thanks, that 49% vs 78% gap is exactly the trap. It's live now: apify.com/locaihost/google-maps-leads

The sample step never takes the first N in Google's order. Each business is ranked by a hash of the search plus its place ID and the lowest N are checked, so it's random but repeatable: the same search always gives the same sample. Your note made us add sampleSeed: set any word or number to draw a different random sample of the same search, repeatable with that seed. We checked it on a 60-business search: the same seed gave the identical 10, a new seed overlapped by 1 of 10.

The status message shows the working, not just the total, e.g. "60 matched: 52 with a website, 8 without; 1 of 10 sampled websites is a hit; estimated 13 hits (8 no-website + about 5 of 52)", with the full-run price and its ceiling. Businesses with no website are counted, not sampled, so they don't skew the rate.

Thread Thread
 
kevin_hall profile image
Kevin Hall •

Hash of the search plus the place ID is the right trick: repeatable without storing a list, and the seed gives a second draw when the first ten look odd. Counting the no-website businesses instead of sampling them is the detail I would have missed. One thing I would still print next to the estimate is how wide it is. One hit in ten sampled websites puts the website hit rate anywhere from a fraction of a percent to about 45% at the usual confidence, so "about 5 of 52" is a point inside a band from 0 to roughly 23, and the 13 is really "between 8 and about 31". If the sample is what the user approves a charge on, twenty sampled websites narrow that band for one more cheap step, and the message can say the band instead of the point. Congratulations on shipping it.

Thread Thread
 
locaihost_data profile image
locaihost data •

Thank you, that's a fair hit. A point estimate on 10 websites is overconfident.

It's in now. The sample step prints a 95% range next to the estimate: a Wilson interval on the sampled websites, applied only to the unsampled rest, because the sampled businesses and the no-website ones are already known exactly. Same Leeds plumbers search, 60 businesses:

  • 1 hit in 10 sampled: "about 13 (95% range 9 to 26)"
  • 4 hits in 20 sampled: "about 18 (95% range 14 to 26)"

When the sample is under 30 websites, the message also says a larger websiteCheckSample would narrow it, so the user can buy the cheaper certainty before approving the full run.