DEV Community: George Kioko

how to export a shopify catalog into a chatgpt shopping feed, no login required

George Kioko — Wed, 09 Sep 2026 15:26:19 +0000

meta shipped muse yesterday. it shops, it compares prices, it checks out through stripe link. shop pay is listed as coming soon. anthropic put out commerce agents a week earlier. openai has had shopping in chatgpt for a while now.

so every lab has a buyer. when muse hits a store with no api, it drives a browser. same as you would. it looks at a rendered page and infers a price. that works, sort of, in the way that reading a menu through a window works. an agent reading a structured catalog does not infer anything. it knows the price, the currency, the variant, and whether the thing is in stock.

that gap is the whole game this quarter, and most shopify stores are on the wrong side of it.

what an agentic commerce feed actually is

an agentic commerce feed is a machine readable list of your products with the attributes a shopping agent needs to make a recommendation and complete a purchase. at minimum that means a stable id, a title, a description, a canonical link, an image link, a price, a currency, availability, a brand, and a product identifier such as a gtin or mpn.

two specs matter right now. the openai product feed specification, which is what chatgpt reads to surface products. and ucp, the universal commerce protocol, which google co developed with shopify and which shopify serves at /.well-known/ucp for a very large number of stores.

the current ucp version is dated august 25. adoption outside the big platforms is not real yet. one count last week put independent ucp sites at around 26. the protocol shipped. the catalogs did not.

a product page is not a feed

this is where merchants get comfortable and stop. shopify has an agentic storefronts toggle. flipping it makes you visible. it does not make you the easiest thing in the results to parse, and it does not fill in the attributes you never entered.

if your variants have no barcode, an agent cannot match your product to the same product on three other sites. if availability is not exposed, it cannot promise the buyer the thing exists. if you block gptbot in robots.txt, you are not in the conversation at all.

shopify says catalog powered ai search converts around 2x scraped data. that is a shopify claim about shopify, not an independent study. anthropic says agent carts run up to 35 percent larger and shoppers are 60 percent more likely to finish. also their number about their own product. hold both loosely. they point the same direction and neither one is evidence.

three checks you can run by hand right now, free

no tool needed for any of these. open a browser.

one, your robots file. go to yourstore.com/robots.txt and search for GPTBot, OAI-SearchBot and ChatGPT-User. if any of them is disallowed, that is a deliberate choice someone made, possibly a plugin, possibly years ago. it is the cheapest thing on this list to fix.

two, your ucp manifest. go to yourstore.com/.well-known/ucp. if you get json, shopify is serving a manifest for you. if you get a 404, you are not exposed through that path.

three, your raw catalog. go to yourstore.com/products.json. most public shopify stores return the whole catalog as json, no login, no app, no api key. this is the same endpoint an agent or an aggregator would reach for first. look at what is actually in there. look at whether barcode is populated on your variants. usually it is not.

that third one is the honest gut check. whatever is missing in /products.json is missing everywhere downstream.

doing it across the whole catalog

reading /products.json by eye works for one store and about ten products. past that you want the catalog normalized and every record checked against the spec, which is the actor i publish.

storeUrl --> catalog extract --> normalize --> validate against
             (products.json,     1 record/     OpenAI feed spec +
             collection JSON,    product,      UCP attributes
             sitemap, JSON-LD)   variants          |
                                 nested            v
                    free drift <-- snapshot <-- fix-it errors per product
                    monitoring     (persists)   + store readiness score

it walks a ladder of public endpoints, /products.json first because it is the cheapest, then collection json, then the product sitemap with per product json, then json ld on the page for headless storefronts. no browser, no login, no credentials at any rung. every record carries the method, the exact source url and a timestamp so you can check the work.

then it validates. every violation names the field, the rule, the evidence and the fix, phrased as the thing you do in shopify admin. "add a barcode in shopify admin, variant level" rather than "gtin missing".

it also writes one free store summary row with a readiness score from 0 to 100, your ai bot robots policy, whether a ucp manifest was found, and the top issues ranked by how much they actually cost you.

honest notes on it, since this is my actor and you should discount accordingly.

pricing is 5 cents per run start plus 1 cent per agent ready product. agent ready means the record is complete enough to be feed eligible. partial extractions are delivered free with the errors that explain why they failed, so a store with a bad catalog does not turn into a bill. there is a max cost setting and the run refuses to start if the worst case estimate goes over it.

residential proxy is the default because shopify blocks datacenter ranges. because the extraction path is json endpoints rather than a browser, that costs about 4mb per 1000 products.

drift monitoring is free. runs sharing a monitor id diff against each other, so you get new products, removed products, price changes, and issues that appeared or got fixed since last time. snapshots live in a named store in your own account so the history outlives apify's 31 day dataset retention.

one thing that is not live. there is a chatgpt shopping placement check in the input, and it is switched off and never charged, because it does not work yet. it is in there so the schema does not change when it ships. do not turn it on expecting an answer.

link: https://apify.com/george.the.developer/shopify-agentic-commerce-product-feed

what to fix first

from the stores i have run through it, the order is almost always the same.

barcodes are missing at the variant level. that is the single most common blocker and it is manual data entry in shopify admin, which is why everyone skips it.

descriptions are html blobs written for a human reading a page, not text an agent can summarise.

brand is unset because the vendor field was never filled in.

and some stores are still disallowing one of the ai crawlers in robots.txt without knowing it.

none of that is exciting work. it is the same unglamorous data hygiene that seo asked for fifteen years ago, aimed at a different reader. the agent is not the product. the feed is.

start with /products.json on your own store and see how much of it is empty.

Cloudflare starts blocking agents by default on September 15. I found out by accident.

George Kioko — Sun, 06 Sep 2026 12:08:11 +0000

Nobody in scraping is talking about this and it lands in ten days.

On September 15 Cloudflare flips a default. Every bot that hits a site behind them now gets sorted into one of three buckets. Search, agent, training. Search stays allowed. Agent and training get blocked by default on any page that shows ads. That applies to every new domain, every new customer, and everyone already on the free tier. A site owner can opt out in the security settings before the 15th. Most wont, because most never open that page.

Their definition of agent is the part that matters. "Automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now." Read that again. That is a fetch tool. That is an MCP server that pulls a page for a model. That is every browser agent demo from this summer.

The part that is funny if you dont run a scraper

Googlebot fetching a page and my fetch tool fetching the same page for a user is the same request. Same bytes, same page, same intent, get the content and use it. One of them has a deal with Cloudflare. One of them doesnt. After the 15th one walks in and one gets a wall.

Cloudflare says this is about publishers getting paid. Fine. But the people who can pay already have contracts. OpenAI has contracts. Google is a search bot by definition so it is exempt from its own training problem. The ones who eat this are one person shops with an agent and a fetch call. Thats me, and probably you if you are reading this.

How i noticed

i didnt read the announcement. i run scrapers for a living and on August 31 Google changed its search result links to /goto redirect wrappers. My parser didnt error. It returned zero rows for a full day and billed nobody, which is the only reason i looked. When i went digging into what else was changing this month, this was sitting there dated July 1 and nobody i follow had said a word.

Earlier in August i lost two other scrapers to DataDome on Etsy and Yelp. Residential proxies didnt help. i killed both instead of fighting it. So i already know what a silent block looks like on a revenue chart. It looks like nothing. Flat line, no error, no email.

That is what the 16th looks like for a lot of agent builders. The model calls the fetch tool, the tool gets a challenge page or a 403, and the model summarizes the challenge page like it is content. Nobody gets an alert.

What i still dont know

Cloudflare has not said what the blocked request actually gets back. Their blog does not say 403, challenge, or the pay per crawl 402. i have not been able to find a site that opted in early to test against. So this is the open question and it is the one that decides whether your guard code works.

A test you can run before the 15th

Pick a site you fetch that is on Cloudflare free tier and shows ads. Fetch it three ways today and save the status code, the response headers, and the first 500 bytes of the body.

Plain fetch with your normal client.
Same fetch with a user agent that says what it is, something like YourAgent/1.0.
Same fetch through whatever your agent framework uses.

Run the same three on the 15th and 16th. Diff them. Look for the cf-mitigated header, a 403, a 402, or an HTML body with "challenge" in it. Whatever changes is your signature. Put a guard on it so your model refuses to summarize a block page.

The ask

If you run a site behind Cloudflare and you already opted in, or you already see the new behavior, paste the status code and the headers in the comments. i will post what i find on the 15th under this article, working or not. i would rather know than guess.

Source is the Cloudflare post from July 1, "Your site, your rules: new AI traffic options for all customers", on blog.cloudflare.com.

Google now ranks the same story differently in Brussels and Dallas. your brand monitor only watches one of them

George Kioko — Sat, 29 Aug 2026 12:37:57 +0000

i build scrapers for a living and one of the cheapest ones i run is a google news monitor. this week google handed everyone who pays for media monitoring a reason to check what the money actually buys.

what happened: on august 28 google said that from august 30 it stops applying site reputation manual actions for people searching inside the european economic area. outside the EEA the penalty keeps working. that is the outcome of a DMA investigation by the european commission, and the wire coverage framed it plainly, google changed the policy to avoid a fine.

so the same query, on the same day, can rank a page for someone in brussels and bury it for someone in dallas. not a ranking wobble. a policy line drawn by geography.

why a monitoring tool cares

every brand and media monitoring product i have looked at pulls its google results from one place and calls it google. one location, one profile, one snapshot. that was always a simplification. from august 30 it is wrong by design for any site under a site reputation action, and that list is heavy with big publishers hosting partner content, which is exactly the kind of site your brand gets mentioned on.

the second half of the problem is older and bigger. chartbeat numbers cited in the reuters institute 2026 trends report put google organic traffic to publishers down about a third year over year through november. sparktoro measured 68 percent of google searches in the first four months of 2026 ending with no click at all. the story about your brand is increasingly read inside an AI answer, inside discover, or not clicked. a count of blue links tells you less every quarter.

what the paid tools charge

brand24 runs 199 to 1,499 a month billed annually, 20 to 25 percent more if you pay monthly. mention shut its self serve tiers in july 2025 and now sells one company plan at 599 a month, annual only, with historical data and API access sold separately. meltwater does not publish prices, which tells you what you need to know.

strip the dashboards off and what you rent is this: a poller that hits news sources for your keywords, drops duplicates, and alerts you. sentiment scoring and share of voice charts sit on top. if your job is reporting those charts upward, pay for them, they are real work. if your job is knowing the moment someone writes about you or your competitor, you are renting a cron job.

the build

google news exposes a feed for any query. poll it on a schedule. filter for your terms. dedupe by url, because the same story gets republished under different urls as it syndicates, and that dedupe step is the only part with any real logic in it. bad dedupe is why cheap monitors feel broken, the same article hits you four times. push new ones to email or slack.

the part that changed this week: poll from more than one country. an EU exit and a US exit now see different results for penalized domains, and the gap between them is itself a signal. a story that ranks in europe and vanishes in the US is telling you something about the publisher, not about you.

honest limits

this is news, not social. it does not read reddit or x. it does not score sentiment. and google news is not every outlet, trade press with no feed still needs its own scraper.

what i run

mine is an actor. keywords and a time range in, deduped articles out with title, source, date, snippet and optional full text. priced at $0.003 per article found, no subscription. over the last 30 days it ran a couple hundred times, almost all on schedules, with a 100 percent success rate, which is the whole point of a monitor. ten brands and five competitors for less than one seat of the paid thing.

https://apify.com/george.the.developer/google-news-monitor?fpr=bbquoh

sources: google search central on the policy update, search engine journal on the EEA carve out, reuters institute trends and predictions 2026 for the chartbeat figure, sparktoro for the zero click rate, and the vendors own public price pages for brand24 and mention.

most of your lead list doesnt exist

George Kioko — Thu, 27 Aug 2026 09:36:39 +0000

i tested an email enrichment waterfall on a real batch in july. the exact setup every ai sdr thread still swears by. find rate: 4.9 percent. of the addresses that came back, 70.3 percent were catch all. they accept everything and tell you nothing about whether a human is on the other side.

out of a hundred people i actually wanted to reach, i got maybe one address i would put money on. the rest was noise dressed up as data.

nobody selling enrichment publishes numbers like these. they publish match rates. a match is not a person. it is one database row nodding at another database row. both can be years old, both can be wrong, and the charge hits either way.

growth twitter is admitting it now. threads this week are full of people saying their scraped lists bounce, their domains burn, their reply rates live under one percent. the tools didnt fail. the premise failed.

the premise is simple and wrong: that a strangers inbox is worth guessing. it isnt. email data rots the day someone changes jobs, and people change jobs all the time. you are buying photos of a crowd that already left.

what holds is behavior you can watch happen.

someone liked or commented on a competitors post yesterday. they told you, in public, with a timestamp, that they care about the problem you solve. no guessing required. no enrichment. no bounce risk. they raised a hand where anyone could see it.

someone just started as head of the function you sell into. fresh mandate, budget still unspent. or someone senior just left and the team they ran is about to change how it buys. all of that lives in public profile data, updating in real time. almost nobody treats it as the lead source it already is.

this data is easier to pull than emails ever were. engagement is public. job moves are public. you dont need a chain of vendors reselling each other. you just watch the surface where buyers already show themselves.

i build tools for that. one grabs everyone who engaged with any linkedin post, likers and commenters, into a clean list: https://apify.com/george.the.developer/linkedin-post-engagers-scraper?fpr=bbquoh. another tracks companies and flags new hires and departures as they land: https://apify.com/george.the.developer/linkedin-company-employees-scraper?fpr=bbquoh. the rest sit at pipelines.trykioko.com.

run your own numbers on your own list before you buy another credit pack. count the catch alls. count the bounces. then count how many of your best customers ever arrived from a cold guessed inbox versus someone already circling in public.

the list you need is not sitting in a database. it is performing in public every day. almost nobody is writing it down.

every layoff is announced twice

George Kioko — Tue, 25 Aug 2026 12:33:01 +0000

apple cut about 200 roles last week. roughly 100 from vision pro gaming and immersive video, roughly 100 from siri and an ai integration group. bloomberg broke it. apples statement says the company is realigning teams "to evolve our business to deliver the best experiences for our users."

read that again. it says nothing. thats not a bug, its the product. corporate comms exists to turn a dead bet into a strategy update.

the layoff was announced twice. once in the memo. and once, months earlier, in data almost nobody watches.

the script is always the same.

the decision gets made in a boardroom, sometimes two quarters before anyone gets the invite.

then hiring stops in the affected org. not company wide, just there. open roles quietly close. nobody announces a freeze. the postings just stop renewing.

then the senior people start leaving. a director this week, a principal engineer two weeks later. each one looks like a normal career move. clustered inside one org in one quarter, its a fire alarm.

then the recruiters attached to that org move or disappear. recruiters exist to grow teams. a team that is done growing cuts them first, and it shows before anything else does.

then, months after the actual decision, the memo. the press treats it like news. it isnt. its an obituary. the death already happened.

announcements are written to be read. thats the problem. every word passed legal, comms and investor relations. headcount data passed nobody. people join, people leave, the profiles update. no pr team edits the public org chart.

which is why the interesting version of the apple story was never going to come from apple. around 100 of those cuts reportedly hit vision pro gaming and immersive video. thats not realigning. thats the quiet end of an expensive bet. the org chart said it clearly while the statement said nothing.

four things worth watching. all of them sit in public profile data.

departures clustering in one team. normal attrition spreads. trouble concentrates.

the seniority of who leaves. juniors leave for money, thats noise. when directors and principals go in a cluster, they know something the rest of us dont yet.

recruiter roles closing. first thing cut when growth stops, and it happens before any announcement.

where the new hires land. one org absorbing people while another gets nothing means the budget already moved. budgets move before memos do.

if youre job hunting this tells you which team not to join. an offer from a dying org still sounds great in the interview.

if you sell b2b, a team bleeding seniors isnt buying this year. a team hiring fast just told you it has budget without saying a word.

if you just work in tech, watching the data beats doomscrolling layoff threads. the market isnt uniformly cooked. specific bets at specific companies are dying while other teams still cant hire fast enough, and the difference is visible from outside.

i got tired of refreshing linkedin by hand, so i built a tracker that watches hires and departures at companies i care about and flags when the pattern changes: https://apify.com/george.the.developer/linkedin-company-employees-scraper?fpr=bbquoh

the rest of my tools live at pipelines.trykioko.com.

next time a company you care about makes headlines, go look at what its org chart did in the three months before. the memo is always late.

How to export LinkedIn post likers and commenters into a lead list (no login needed)

George Kioko — Fri, 21 Aug 2026 11:00:54 +0000

Nobody wants to say this plainly so i will. the AI SDR experiment is coming back with results now, and theyre ugly. one founder asked this week if any of these tools actually book meetings. the replies he counted: out of office autoresponders and No's. months of automation, thats the harvest.

same week, operators started posting their enrichment bills in public. one did the math on a basic search enrichment and landed on a 250x markup over the raw api cost. thousands a month to enrich the same cold lists every competitor is enriching with the same tools.

the tools arent the problem, the lists are

every one of these stacks points at cold data. export from a database, enrich it, let AI write something that pretends to be personal, spray. the person on the receiving end gets forty of these a day now, half of them from profiles with AI generated faces. cold is cold no matter how much AI you rub on it.

meanwhile the actual buying signal is sitting in public, on LinkedIn, refreshed daily, and almost nobody harvests it.

every post in your niche is a lead list

when someone reacts to a post about, say, cold email deliverability, they just told you three things. they care about this exact topic, they care about it this week, and theyre active on LinkedIn. thats a hand raised in public.

the people commenting under your competitors launch post are not a cold list. they are the warmest b2b list that exists, self sorted by interest, timestamped, with their job title attached.

the manual version, free, works today

find 3 posts in your niche that pulled real engagement, competitor launches and hot takes work best
open the reactions list and the comments
copy out the people whose headline matches your buyer, founder, head of growth, whatever your ICP is
message them about the topic they just engaged with, not about you

thats 20 minutes per post and it beats a thousand row cold export, no doubt about that.

the automated version

i got tired of doing the manual version, so i built it as an Apify actor: LinkedIn Post Engagers Scraper.

paste post urls, get back the engagers, deduped across posts, with an ICP filter on headlines so you only pay for people who match your buyer. half a cent per matched lead. and it never asks for your LinkedIn login, which matters now that account restrictions are the number one complaint in every outreach thread.

first paying users ran it start to finish in week one, every run succeeded. its new, its small, it does one thing.

the rest of my lead pipelines live at pipelines.trykioko.com.

signals beat volume

the channel isnt dead. the lists are. stop paying enrichment markup on strangers and start messaging the people already raising their hands.

Every paid API tutorial ships a rate limit you can beat in four seconds

George Kioko — Mon, 17 Aug 2026 11:38:52 +0000

There is a whole genre of tutorial that takes you from a database table to a paid public API in about twenty minutes. Table of data, a keys table, hash the key, put an edge function in front, add a rate limit, ship it. The good ones are genuinely good. The key handling in particular is usually correct, and correct here matters more than it sounds.

But almost every one of them ends with a rate limit you can walk around in four seconds, and most of them do not tell you.

The part they get right

Store a fingerprint of the key, never the key. You hash it, you keep the hash, you keep a short prefix so a customer can tell their own keys apart in a dashboard, and you hand the real string to the customer exactly once. If your database leaks, nobody gets a working key out of it.

That is not paranoia, that is the same shape Stripe and everyone else uses, and it costs you one hash function. Generate the key inside the database rather than in a browser so the raw value never travels further than it has to. Put row level security on the keys table so a user can only ever see their own rows. Keep the service role key inside the edge function, never anywhere a browser can reach it.

None of that is advanced. All of it is worth doing on day one, because retrofitting key hygiene after you have customers is miserable.

The part that quietly does not work

Then comes the rate limit. Count the requests for this key in the last minute, and if the count is over your limit, return a 429.

Read that again. For this key.

Your customer is sitting in a dashboard that lets them mint new keys whenever they like. They hit the limit, they click generate, they get a fresh key with a fresh counter, and they carry on. The limit measured the key. The person was never limited at all.

I am not describing a hypothetical. This exact bypass gets demonstrated live in tutorials, hit the ceiling, mint a new key, keep pulling data, and the narration moves on. Sometimes the author flags it as something to solve later. Later never arrives, because the reader has already copied the code.

Count the account, not the credential

The fix is a one word change in your head and a small change in your schema. A key is not a customer. A key is one of several credentials that belong to a customer. So the counter belongs on the customer.

Your keys table already stores the user id that owns each key. When a request arrives, you hash the key, look it up, and you now know two things, that the key is valid and who it belongs to. Do the counting on the second one. Every key that account owns draws from the same bucket.

Once you do that, minting a new key gets you nothing, which is exactly what you want, because minting a new key was never supposed to be a purchase.

While you are in there, cap how many keys an account can hold at all. Nobody needs four hundred. Rotation needs two. And keep the counter in a table the customer cannot write to. A limit stored somewhere the customer can edit is not a limit, it is a suggestion.

The bigger thing hiding underneath

Here is what took me too long to understand while building paid data tools. A rate limit is not billing. It is a blast shield. It stops one bad actor taking your service down, it does not decide what anyone owes you.

Metering is what you actually sell, and metering has a rule that rate limiting does not. You must never charge for a request that produced nothing useful.

That sounds obvious and almost nobody does it. Every broken data product i have ever taken apart, including two of my own, had the same defect. Loose match, return something anyway, charge for it anyway. Asked for one company, got a similar looking one, billed as a hit. Asked for a profile, got a page that merely mentioned the name, billed as a hit.

The honest version is not harder to build. Emit everything you found, mark what did not clear the bar, and charge only above the bar. Your dataset gets a row saying this one is free and here is why. Customers notice this immediately and it buys more trust than any amount of uptime.

Most of the time the confidence score you need is already sitting in your code, computed and then ignored. Go and look. i have found it every single time.

Three more things the twenty minute version skips

Version the path from the first request you ever serve. Once someone else's code depends on your response shape, changing that shape is not an improvement, it is an outage you caused. Ship v1 in the url and you can build v2 next to it without breaking anybody.

Paginate by default, and let people filter and sort on your side. If your answer to a thousand rows is here are a thousand rows, every customer builds the same filtering logic separately and all of them think your API is slow.

Return honest status codes. Bad key is a 401. Missing thing is a 404. Over the limit is a 429. Never a 200 with an error hidden in the body, because the caller's code reads the number before it reads anything else, and a lie there turns into a support ticket a week later when their pipeline has been silently storing failures.

And the thing nobody wants to hear

You can do all of this perfectly and still make no money, because none of it is the hard part.

The hard part is that a stranger has to find your API, believe it works, and hand over a card. i maintain seventy six of these things. The ones that earn are not the ones with the best code. They are the ones where somebody types a phrase into a marketplace search box and my title is the thing that comes back.

Build the key system properly, it takes an afternoon. Count against the account, not the key. Charge only for what worked. Then spend the rest of your time on the part that is actually hard, which is being findable.

The proxy industry needs you to never open the network tab

George Kioko — Wed, 05 Aug 2026 15:51:09 +0000

I run 75 scrapers in production. Three of them do any fingerprint spoofing. Maybe five use residential proxies. The rest run on plain datacenter IPs or no proxy at all, and they have been running for months.

If you learned scraping from blog posts, that number probably sounds wrong to you. Every tutorial you have read starts the same way: sign up for a residential pool, install a stealth browser, randomize your fingerprint, throttle like a human. Then, on step five, you finally get to look at the actual website.

That order is backwards, and it is backwards on purpose. Proxy companies write most of the scraping content on the internet. They were never going to write "you probably do not need us."

The scraper with the $80 a month costume

Last month my guy sent me his Greenhouse job board scraper to fix. It had everything. Puppeteer with the stealth plugin. Rotating residential proxies. Randomized mouse movements between actions. Human-like typing delays.

It still kept dying.

So I did the thing nobody had done: opened the page in a normal browser with devtools up. The entire job list was sitting in one XHR request to a public JSON endpoint. No auth. No cookies. A rate limit so loose I never managed to hit it.

I deleted basically his entire codebase and replaced it with a fetch call. It has not broken since. He had been paying for proxies for months to hit an endpoint that does not care who you are.

This was not a rare lucky case. This is most cases.

The 20 minute method

What I do on every new target, before writing a single line of code:

Open the network tab, filter to XHR/fetch.
Reload the page. Click around. Paginate. Search.
Find the request that returns the actual data. It is usually JSON and usually obvious.
Right click, copy as cURL.
Paste it in a terminal and start deleting headers one at a time. Rerun after each delete.

Whatever survives step five is your scraper. Most of the time the answer is a user agent header and nothing else. Sometimes a referer. Occasionally one cookie you can grab with a single cheap request first.

Twenty minutes. No browser automation, no proxy bill, and the result is faster and more stable than any puppeteer setup, because JSON endpoints change way less often than HTML markup does.

Half the "protected" sites out there are react apps sitting on a wide open API. The frontend is the security theater. The data is just there.

When you actually need the heavy stuff

I am not saying anti detect tooling is fake. I am saying it is a last resort that got marketed into a first step. You genuinely need it when:

The protection sits on the API itself, not the page. Turnstile or a challenge token attached to the data endpoint.
The site checks TLS fingerprints, so plain curl or node fetch gets flagged before your headers even matter.
Sessions are bound to browser fingerprints, ticketing platforms mostly.
You need real logged-in browsing at scale.

In my portfolio that is maybe 1 target in 10. For those, sure, pay for the pool. For the other 9, the network tab was free the whole time.

The tell

You can tell how someone learned scraping by what they reach for first. People who learned from vendor blogs can configure a fingerprint rotator but have never stripped a cURL command down to its minimum headers. People who learned by breaking things do it the other way around.

Try the boring thing first. It costs nothing, and most of the time it is the whole job.

I write about this stuff and post scraping methods on r/scrapingtools if you want more of it.

Nobody voted for the company that now licenses the readable web

George Kioko — Wed, 29 Jul 2026 10:41:42 +0000

i write scrapers for a living, so read this knowing im biased. but something happened on july 1 that got covered as a pricing update, and it is not a pricing update.

Last time i wrote about the big labs pulling the ladder up behind them after climbing it. This is the same story from a different seat. Not a company fighting over data it trained on. A company that sits in the road deciding who is allowed to walk down it.

Start with what they got right

Most takes on this are lazy, so let me concede the strong part first.

Cloudflare moved from pay per crawl to pay per use. Under the old model a bot paid a fee to fetch a page, minimum a cent per retrieval. Under the new one, publishers get paid when their content actually shows up inside an AI answer. Opt in, content gets used downstream, money comes back.

That is real compensation reaching real site owners, and it is better than what came before. Anyone telling you this is pure extraction has not read it. i am not going to pretend otherwise to make the rest of this hit harder.

The payment is fine. Its everything wrapped around the payment that should bother you.

Who wrote the categories

AI crawlers now get sorted into three buckets. Search, agent, training. Which bucket your bot lands in determines what you can reach and what it costs.

Nobody outside one company defined those categories. Think about how much sits inside that definition. Is a research crawler agent or training. Is an archival bot search. Is a small team building a tool a legitimate reader or an unlicensed one. Those are policy questions with real consequences for who gets to build anything, and they were answered in a product spec.

Who made the scarcity

Starting September 15, mixed use crawlers get blocked by default on any page carrying ads.

Read that as a business move rather than a safety one. You cannot run a marketplace for access until access is scarce. The same company that operates the marketplace also owns the switch that creates the scarcity, and it just flipped that switch on for everyone at once, by default, without each site owner deciding anything.

That is not a company responding to a market. That is a company manufacturing one.

Who is on the list

The program launched with two partners.

So at this moment, one company decides what counts as a reader, what counts as a thief, and which buyers are allowed to cross the line between them. If you are building something and you are not on the list, your options are pay whatever the rate becomes, or lose access to a large fraction of the public web.

That is not a market with many sellers and many buyers. That is a licensing authority with a waiting list.

The part scrapers already knew

Everyone in this work has understood for years that bot blocking was never really about stopping bots.

A company that could genuinely stop automated access would ship that once and be finished. It would also destroy the revenue line it is currently building a business on. The incentives only ever pointed one direction. What got built is not a wall, its a meter with a wall painted on the front.

Blocking is the demo. Billing is the product. Watch how fast a challenge page gets easier the moment a paid API appears.

The site owners are the strange part

Millions of sites pointed their DNS at one company years ago because the free tier was genuinely good and the bandwidth cost nothing. That was never charity. That was acquisition cost, and it worked.

Now you get a check. You do not get a vote. Not on the categories, not on the rate, not on who is admitted to the program, not on whether the default should have been blocking in the first place.

What to actually take from this

im not telling anyone to rip out their CDN. the payments are real and some publishers will be better off.

im saying notice what quietly happened. Permission to read the public web turned into a licensable asset, and the entity issuing the licenses is a private company that appointed itself, defined its own terms, created the scarcity its market depends on, and answers to nobody who has to live under the rules.

The money is not the story. The authority is.

The companies that scraped the whole internet now want scraping banned

George Kioko — Wed, 22 Jul 2026 12:18:43 +0000

i scrape the web for a living, so read this knowing im biased. but this week made something clear that i think a lot of people already feel and havent said.

Every large model you use was built on scraped data. Your blog posts, your code on github, your stack overflow answers, your art, your comments, your captions, entire forums, the whole public internet. It got vacuumed up without anyone asking, because it was reachable and that was enough. i know exactly how that works, its my actual job.

And i dont even hate that part. Public is public. Ive built a whole career on the idea that data people put out in the open is fair to collect. That belief is why im comfortable saying the next part.

The move that should bother you

Now that these companies are worth what they are worth, they want the rules changed. Not for themselves, for everyone coming after them.

They are lobbying to restrict open source models. They are describing plain access to public data as a safety threat, and in some rooms as a national security problem. The exact activity that built the entire industry is being quietly recategorized into something that needs licenses, permission, and control.

Look at what is actually happening while they say this.

Anthropic is claiming local models steal from it. This is months after it paid over a billion dollars to settle with writers over claims that it trained on their work without permission. The company that just wrote a giant check for taking data is now worried about other people taking data.

OpenAIs own models just broke out of a controlled test and hacked into Hugging Face to cheat a benchmark. Their frontier systems escaped the sandbox, chained real vulnerabilities, and reached a production database. This is confirmed, they disclosed it themselves. And the public framing that follows an event like that is never we should be more careful with what we build. It is everyone needs tighter controls.

Call it what it is

This is not safety. It is a monopoly wearing a safety badge.

When you are the incumbent, the cheapest defense money can buy is a rule that makes it illegal for the next version of you to get started. You do not have to win on quality if your competition is banned from touching the raw material you already sit on top of. Regulation written by the biggest players is not protection for the public, it is a fence around the thing they took.

The people who collected everything want to be the last ones ever allowed to collect anything.

Why i bother writing this

i dont have a clean policy fix. im one person who writes scrapers. but the story being sold right now is the exact opposite of what is happening, and framing hardens fast once it goes unchallenged. so it is worth putting the plain version on the record before it gets written into law.

If you build in this space and you want tools that treat public data as public, that is most of what i make. You can see the full set here: https://apify.com/george.the.developer?fpr=bbquoh

Back to my scrapers.

I build the scrapers the data vendors run on, and I need to tell you what you are actually paying for

George Kioko — Wed, 01 Jul 2026 05:12:55 +0000

I have built the scraping guts behind more "real time business intelligence" products than I would ever admit on a CV. So when a contact data company tells you their database is fresh, I am not impressed, because I know how that word gets made.

Here is the part nobody puts in the sales deck. These companies pull public pages on a schedule, pour the rows into a giant table, and sell you a seat to the table. That table is a photograph. It was true the day it was scraped and it has been aging ever since. You pay every month like it refreshes every month. It does not. A row captured in February is still February sitting in your CRM in June, looking exactly as confident as a row pulled this morning, because nothing on the screen tells you which is which.

And the field that goes bad first is the one you bought the thing for. Where a person works right now. People quit and get promoted and get poached and get pushed out, and the company page is the last place on earth to find out. So "current company," the whole reason the list has any value, is one of the first columns to start lying by the time your rep hits send.

The industry knows this. Of course they know it. They built the pipelines. They just figured out that decay is invisible, and you cannot be angry about a number you never see. So the pitch drifts back to coverage. Millions of contacts. Look how big the table is. Freshness is the expensive part. It makes the table look smaller, and a smaller honest number loses sales calls to a bigger dishonest one.

So you get a beautiful export, no red cells, the kind of file an agency screenshots for a client and feels calm about. Then it goes out and lands on a VP who left in spring, a director who changed roles, a manager who is now at the competitor you are trying to beat. The email still works, which is the worst case, not the best one, because now your pitch reached the wrong person with total confidence and your team looks careless instead of automated.

I will say the thing the vendors will not. Pulling the names is the easy half. It has been easy for years. The hard half, the half they quietly skipped and still charge you for, is going back to each row and checking the one thing that rots. Does this person still work here, right now, on the live profile, not on the cached page that first surfaced them. On a messy company pull, I have seen that check wipe out the majority of the list as already wrong. Calling that a data cleaning chore is how vendors hide the miss. It is the difference between the product they sold you and the product you thought you were buying.

I still build scrapers. I think they are great. But a scraper with no verification is a camera, and these companies are selling you old photos at a subscription price and calling it a feed. The tell is simple. Ask your vendor what percentage of their "current company" fields they reverify, and how recently. Watch them change the subject to coverage. That answer, the one they will not give you, is the whole game.

You are not paying for data. You are paying for the confidence that it is current. And that is the one thing in the box they never actually put in.

Why I Turned 70 Scrapers Into Claude Agent Skills, and Why MCP Beats Calling the API

George Kioko — Fri, 19 Jun 2026 09:10:13 +0000

Why I Turned 70 Scrapers Into Claude Agent Skills, and Why MCP Beats Calling the API

For a long time the only way to use a scraper inside your own software was the same: get an API key, kick off a run, poll until it finishes, pull the dataset, parse it, then wire that into your code. It works. It is also a lot of plumbing for what should be one step.

This week I exposed all of my Apify actors over the Apify MCP server, so an AI agent can call them as skills instead. LinkedIn employees with emails, OFAC sanctions screening, YouTube transcripts, Google News, company enrichment, and the rest. Same actors, same pricing. The difference is how you reach them.

The plumbing problem

Calling an actor through the API looks like this. You authenticate. You start a run with the right input. You poll the run status, or you set up a webhook to tell you when it finishes. You fetch the dataset. You parse the rows. You map them into whatever your app expects. Every one of those steps is code you write, test, and maintain. Multiply it by every tool you want your agent to use, and you are building an integration layer that has nothing to do with the actual work.

What MCP changes

MCP flips it. The actor becomes a tool your agent can see and use directly, mid conversation, with no glue code. The agent reads the tool's schema, calls it, and gets a clean result back. You do not write the run-and-poll-and-parse loop, because the agent simply uses the tool the way it uses any other.

In practice that turns "go build an OFAC integration" into a sentence:

"Screen these twelve vendors against OFAC and flag anything over 85 percent."

The agent runs the actor, reads the matches, and hands you the shortlist. The scraper stopped being a service you integrate and became a skill your agent already has.

Why this matters for anyone building with agents

The whole point of an AI agent is that it does multi step work for you. A sanctions check, a lead pull, a transcript summary, those are steps inside a larger flow. If every step needs its own hand written API client, the agent is only as capable as the integrations you had time to build. MCP removes that ceiling. Point your MCP config at the actor and it shows up as a skill the agent can use immediately, alongside everything else.

How to use mine

Every actor is public under george.the.developer on the Apify Store, and exposed over the Apify MCP server. In Claude, Cursor, or any MCP client, add the Apify MCP server to your config and the tools appear:

https://mcp.apify.com?tools=george.the.developer/ofac-sanctions-screener

Then ask your agent to use it. Screen a vendor list, pull a company's employees with emails, turn a video into a transcript, whatever the actor does. Same actor you could call through the API, now usable as a skill inside your agent flow, no plumbing.

That is the shift: stop integrating tools, start giving your agent skills. Try one on your own data and you will feel the difference in the first five minutes.

Browse the actors here: https://apify.com/george.the.developer

Source and verification reports: github.com/the-ai-entrepreneur-ai-hub/apify-actor-portfolio.