DEV Community: Eliran Yihye

What AI Crawler Logs Showed Us: Bots vs Fetches

Eliran Yihye — Thu, 15 Oct 2026 19:12:23 +0000

Which AI crawler fetches mean a real person asked a question?

User-fetch agents such as ChatGPT-User, Claude-User and Perplexity-User fire when a person asks an assistant a question and it needs a live page. Index crawlers such as OAI-SearchBot build a search index ahead of time. A user fetch means you were considered for a real question, so report it separately.

I am Eliran, founder of Cuanto Labs, a software company in Fort Lauderdale, Florida. We log crawler requests on our own marketing site on the server side, because most of these bots never run JavaScript and so never appear in a browser-based tool. This post is what the first eight days of that log showed.

A warning about sample size, up front. Our log starts on October 7, 2026, and this is a small site. Every number below is a count from that window, not a rate and not a trend. Treat them as direction, not proof.

What did the counts look like?

This is the server-side log for our main domain plus an old alias domain that now redirects to it, from October 7 to October 15, 2026. Total user fetches were 90, of which 75 were ChatGPT-User. We saw zero hits from GPTBot or Applebot, which is a fact about our log and not a claim about those bots elsewhere.

Bot	Type	Requests	Distinct paths
Bingbot	index	683	80
PerplexityBot	index	448	89
OAI-SearchBot	index	345	61
ClaudeBot	index	99	60
Googlebot	index	86	26
ChatGPT-User	user fetch	75	28
Claude-User	user fetch	13	7
Perplexity-User	user fetch	2	1
GPTBot, Applebot	index	0	0

When do index crawlers show up?

They arrive in bursts, near deploys. OAI-SearchBot's 345 hits were not spread evenly: nearly all fell into three short windows, the morning of October 10, midday and evening of October 12, and the morning of October 13. Every new page from our October 12 evening release was fetched within minutes of going live.

Which pages do assistants fetch for users?

Mostly the plain-facts pages. Of the 90 user fetches, brand, home and portfolio pages took 59, pricing, buy and service hubs took 12, city pages took 7, AI automation pages took 6, and research and blog took 6. These are counts of fetches, not conversions.

An assistant that fetches only the home page should be able to answer who you are, where you are and what you do. A few plain sentences matter more than a clever one. Say what you do, where you are and what it costs, in words an assistant can lift.

Does the daily curve follow what you publish?

It did for us, though one peak proves little. Daily user fetches in our window were 12 on October 10, 32 on October 12 (the peak), 9 on October 13 and 14 on October 14. The October 12 peak lines up with the release day. The shape is consistent with new content being looked at when it exists.

Can an index hit come before a live fetch?

Yes, and it can come days earlier. One example, without identifying anyone: a page on automation services was first fetched by OAI-SearchBot on the evening of October 12. On the afternoon of October 14, ChatGPT-User fetched the same page. About an hour later, a business owner emailed us directly asking to talk about AI and lead generation.

We cannot prove the two are linked, and nothing contradicts it. We put it at roughly even odds on timing alone.

What it does show is that this channel can leave no trace in the usual places. There was no chatgpt.com referrer, no campaign tag and no form submission, because the person went straight to email. If you only look at referrals and forms, a conversation like this is invisible. The server-side fetch log is the only place the chain shows up at all.

What should you log to see AI bot traffic?

Log four fields on every request whose user agent matches a known AI bot, then report index crawlers and user fetches in two separate tables. You do not need an enterprise tool, and the log costs very little to start keeping.

Timestamp, converted to the time zone you read reports in.
User agent family, normalized to a short name.
Host and path, and whether the response was a 200, a redirect or a 404.
Whether the request came from a data-center network, which helps separate bot traffic from ordinary visitors.

Two cautions. User agent strings can be spoofed, so where a vendor publishes IP ranges, verify against them before you treat a hit as real. And log server-side only: these agents mostly do not execute your analytics script, so a browser-based tool shows you none of this. Make sure your tracking script does not fire for them either, or your human counts get polluted.

What does this suggest you do?

These follow from the log. They are suggestions, not a record of results. Put the plain facts on the home page and the pricing page, since those were fetched most. Keep your sitemap current so new pages are discoverable on the first crawl.

If you have an old domain, redirect it to the main one. In our log, 14 of the 75 ChatGPT-User fetches went to the old alias domain, which now redirects, so fetches land on one canonical site.
Do not read meaning into a probe path that returns a 404. Fix it or filter it out of your crawl report.

What are we not claiming?

We are not claiming that appearing in this log produces customers at any rate. Our sample is small, our window is eight days, and the one possible conversion above is a single timing coincidence. The narrower claim: the log is the only way to see user-fetch behavior at all.

If you want help with AI automation, agents or the visibility side of it, our AI automation page lists what we build and what it costs. Related: SEO for AI businesses.

Written by Eliran Yihye, founder of Cuanto Labs, on October 15, 2026.

Originally published at cuantolabs.com.

AI Lead Follow-Up on Next.js and a CRM

Eliran Yihye — Thu, 15 Oct 2026 19:07:07 +0000

How should you wire lead capture before adding an AI agent?

Save the lead first, notify second, and make silence queryable. The database row is the durable record, the visitor never waits on a downstream service, and a missing notification must be distinguishable from a missing lead. An AI follow-up agent only works if the lead arrives intact, exactly once.

I am Eliran, founder of Cuanto Labs, a software company in Fort Lauderdale, Florida. Most AI lead follow-up content is about the clever part, the agent that texts a prospect within a minute. This post covers the capture side on our own Next.js site, then where an agent and a CRM plug in. The capture pattern is what we run, simplified. The agent section is design guidance, and I label it as such.

Step	What it does	Failure it prevents
Validate	Rate limit, honeypot, field checks	Bots and junk rows
Insert	Parameterized write, answers 500 only if the write failed	Reporting a saved lead as unsaved
Notify after the response	Runs once the visitor has their reply	A slow mail provider blocking the form
Bound each channel	Hard timeout per channel	One provider holding the function open
Stamp the row	Records that a channel delivered	Silent failures looking like a quiet week

What should the route handler validate first?

Validate everything before the database is touched. Our route handler rate limits to 5 requests per minute per IP, rejects invalid JSON with a 400, checks every field's shape and length, and uses an allow-list for the source value. A hidden honeypot input gets a normal success reply and nothing else.

The in-memory rate limiter is the right size for a marketing site and the wrong size for anything under real abuse pressure. Know which one you are.

The honeypot is an off-screen input with no label and no autofill hint, so real visitors never fill it. The silent success is deliberate: an error would tell the bot which field tripped you.

Why insert the lead before notifying anyone?

Because the insert is the only step that may answer "failed to save". Keep notification code out of the same try block. If a mail call threw inside it, the handler would answer 500 for a lead that was in fact saved, the client would retry, and you would get duplicate rows and still no notification.

Reporting a saved lead as unsaved is worse than a missed notification, so the error handling ends the moment the row exists. Use parameterized queries only: values travel as parameters and are never concatenated into the SQL string.

How do you notify without making the visitor wait?

Use the after() function from next/server, which runs work after the response has flushed. The visitor gets their success reply at once while the notification runs behind it. Wrap the call in a helper that never throws, and pass after in as an argument so it can be tested.

Two traps apply. A floating promise is not a substitute: on serverless, work not registered with after() can die when the function freezes after the response. And after() can throw synchronously outside a request scope, so log and swallow that failure. The row exists, so the response is still a success.

How do you stop one slow channel from breaking the rest?

Give every channel its own hard timeout and race the send against it. Ours is 5 seconds, with an abort signal passed to the provider, so one slow provider cannot hold the other channels or the function open. Email is the only channel today, and others can be added without touching the route.

Two smaller rules have paid for themselves. Every channel degrades to logging when its configuration is missing, so a missing API key is a no-op and not a crash. And provider error strings are a privacy surface, since a raw mail API error can contain the recipient address. Redact before logging.

How do you make a silent failure visible?

Stamp the row. After the notifier runs, check whether any channel reported delivered, and if so set a notified_at timestamp on the lead. The check is stricter than "not failed": a channel in log-only mode reports logged, and counting that would mark every preview-environment lead as handled.

Now a small watchdog can ask the database a plain question: which leads are older than N minutes, 15 in ours, with no notified_at? Run it on a schedule and send yourself the result. Error monitoring tells you about failures after you suspect something. The stamp tells you about the ones you never suspected.

If the stamp update itself fails, swallow it. The email went out, and the worst case is a duplicate ping from the watchdog, which is noisier than silence and far better than the reverse.

We also send a server-side conversion event from after() in production when the row is created, because ad blockers and closed tabs lose browser events. It carries the lead id and source, never the name, email or message.

Where does the AI follow-up agent plug in?

Everything above is the trigger. Once a lead is a row with an id, the agent should subscribe to "lead created" from the database or a queue, not be called inline from the route. The route stays fast and the agent can be retried on its own.

This is how I would wire it and how we scope it for clients.

A state machine, not a chat log. A lead moves through new, contacted, replied, qualified, handed to a person and closed. The agent reads and writes the state, and your existing CRM stays the system of record.
Approved templates. A person approves the sequence templates once and the agent fills them in. Anything outside a template goes to a person.
Stop conditions built first. Any reply pauses automated sends and alerts a human. An opt-out phrase ends sends on that channel and is recorded. Attempts are capped per lead, with quiet hours and a human takeover flag the CRM can set.
Write back to the CRM. Every send and reply becomes an activity on the lead record. Otherwise a salesperson will contact someone the agent already contacted.

Texting rules on consent, quiet hours and opt-outs are real legal constraints. Have counsel review your sequences. This is engineering experience, not legal advice.

What do we claim, and not claim?

We do not quote a response-time improvement, because our own sample is small. The design guarantees three things: a saved lead is never reported as unsaved, the visitor never waits on a mail provider, and a missing notification shows up in a query.

We build follow-up systems for Florida businesses at a fixed price from $5K, typically in 2 to 6 weeks, with the code and accounts transferring to you. Scope depends on how many lead sources it watches, which channels it sends on and which CRM it connects to. See the AI lead follow-up automation page, or read how fast to follow up with a lead.

Written by Eliran Yihye, founder of Cuanto Labs, on October 15, 2026.

Originally published at cuantolabs.com.

How an AI Voice Receptionist Works, and Fails

Eliran Yihye — Thu, 15 Oct 2026 19:06:59 +0000

How does an AI voice receptionist work?

A voice receptionist is a loop with a hard real-time budget. The carrier streams the caller's audio to your server, speech-to-text turns it into words, a language model decides what to say or which tool to call, and text-to-speech streams the answer back before the caller says "hello?" a second time.

I run Cuanto Labs, a software studio in Fort Lauderdale, Florida. A few weeks ago we put our own phone agent on a real number. It answers, screens, takes messages and hands calls to a person. This post covers the architecture and the bugs that cost us the most time, because most voice-agent write-ups stop at the diagram. It describes components, not products.

Component	Job	What to test
Telephony carrier	Carries the call and streams audio both ways	Audio direction, webhook verification, spend caps
Speech-to-text	Turns caller audio into words	Accents, noise, phone numbers spoken aloud
Language model	Decides the reply or the tool call	Time to first token, tool-use accuracy
Text-to-speech	Turns the reply into audio	Naturalness on a phone line, second languages
Agent framework	Runs the frame pipeline between them	Interruptions, voice activity detection

We used an open source agent framework for voice activity detection, interruption handling and partial transcripts, so we did not own that plumbing.

Why use WebSocket media instead of SIP?

Use WebSocket media when your agent runs behind a home or office router. SIP needs public UDP ports for signalling and audio, while a WebSocket stream is ordinary TCP, so it passes through a tunnel with no port forwarding. If your server has a public IP address, revisit SIP, which is a good system.

We evaluated SIP first, but our agent runs behind a residential router, so the carrier's WebSocket media stream was the easier path.

How do you keep a voice agent from doing something dangerous?

Give the model a short, fixed list of tools and keep dangerous arguments out of its reach. Our agent can call exactly three tools: transfer to a human, take a message, and end the call. The transfer destination is fixed in the server environment, so the model never supplies a number.

That is the cheapest safety decision in the build: a prompt injection from the caller cannot redirect a call to a premium-rate number if the model has no argument to put it in.

We also run a small, fast classifier before answering, to catch spam and robocalls, and on each turn, to decide whether to escalate to a person. It has a hard timeout, because a slow check must never stall the call.

Which language model works best on a phone call?

Pick the model by time to first token, not by benchmark rank. On a phone call, quality that arrives late is quality the caller never hears. In our tests a small fast model answered in well under a second, while a larger model took more than a second and a half, and that gap decided it.

A fast model is not free. One otherwise capable model mangled messy phone-number extraction in half of our runs, because callers say numbers like "seven eight six, uh, four one three, zero four one six." Treat number capture as something you verify by reading it back, never as something the model got right.

How should you choose a text-to-speech voice?

Choose by listening, because the measurable metrics tie. We compared six voices on the same script at telephone audio quality. Time to first audio and word error rate on a round-trip transcription test were effectively tied. A blind human-likeness rating was the only measure that separated them.

A second language costs you. In a Hebrew test, replies took several seconds, against well under a second in English. Plan the greeting and filler phrases accordingly.

What bugs make a voice agent go silent?

Three bugs cost us days, and all three were configuration, not code. The first left the agent unable to hear, the second left it silent in the recording, and the third made webhook signature checks fail, even though the logs looked healthy.

The agent that could not hear

The bot greeted the caller, then never responded. We had asked the carrier for both audio directions, which sends the bot's own voice back to us as a media message tagged as outbound. Our framework's stock serializer did not filter by direction, so the bot's own speech fed the speech recognizer and the voice activity detector. The bot kept "listening" to itself.

The fix was a small serializer subclass that drops every frame not tagged as inbound. Five lines of code, two days of confusion.

The agent that was silent in the recording

Calls connected and transcripts looked fine, but the bot's channel in the recording was empty. One carrier option routes the bot's audio to a bridged call leg. Our call had no bridged leg, so the audio went nowhere. Pointing the option at the call itself fixed it.

We caught it by comparing the two channels of the recording, not the logs. A pipeline can log "audio sent" faithfully while the carrier discards it. Listen to a real recorded call after any change to stream parameters.

The webhook that verified against the wrong key

Webhook signatures should be verified against the public key the carrier serves from its own key endpoint. A key we had copied into our secrets file looked right, was a different length, and did not match. If signature verification passes in tests and fails in production, or is quietly disabled, check the key source first.

What else belongs on a voice agent checklist?

Four smaller items belong on every checklist before launch. They are the ones we hit in the first week of running real calls, and each one is cheap to handle once at build time and expensive to find later, in production, with a caller waiting.

Carrier command payloads can require base64 encoding for state fields, and call identifiers can contain characters that need URL-encoding in paths.
A 403 when creating a call can mean the call-control application was set inactive. Read the response body, which says so.
The first call after a restart can time out while models and sockets warm up. Place a warm-up call in your deploy script.
Telephony is not chat. Set an outbound allowlist where empty means deny everything, an hourly call cap, and a daily spend cap on the carrier profile.

How do you test a voice agent without a phone?

Write a fake carrier. Ours is a WebSocket client that speaks the carrier's media protocol and plays synthesized caller audio, so the full pipeline runs in continuous integration with no real phone numbers, no carrier charges and no cost beyond speech processing.

For real-audio checks, one of our numbers answers with a listener persona, and a test asserts that both call logs contain user turns. If either side has zero user turns, you have a one-way audio bug like the two above.

What does Florida law say about recording a voice agent's calls?

Florida generally requires the consent of all parties to record a phone call. Our greeting states that the call is recorded, and the disclosure is spoken in the caller's language. Have an attorney confirm the wording for your business, since this post is engineering experience and not legal advice.

If your agent talks to the public, also decide up front when it admits it is an AI (ours does when asked directly) and keep a human fallback for urgent or sensitive calls. The agent should never triage an emergency. It should tell the caller to dial 911 and stop.

What would we tell a client?

The parts are not exotic. What decides whether it works is a short list: pick the model by latency, keep dangerous arguments out of the model's reach, listen to real recorded calls, and build the spend caps and kill switches before launch, not after.

We build scoped voice agents for Florida businesses at a fixed price from $5K, typically in 2 to 6 weeks, and the code and accounts transfer to you. Details and the questions to ask any vendor are on the AI voice receptionist page. For prices, see what an AI voice receptionist costs in Florida.

Written by Eliran Yihye, founder of Cuanto Labs, on October 15, 2026.

Originally published at cuantolabs.com.