DEV Community: cloudflare

AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

Aulvem — Sat, 22 Aug 2026 03:31:26 +0000

I used to watch AI crawler traffic on my site as a table grouped by User-Agent: so many requests from ChatGPT-User, so many from GPTBot. Numbers going up meant the AI systems were picking the site up.

Then I re-cut eight days of logs by verification result. Requests that actually fetched an article: 468. Requests probing for .env and friends: 991. Of everything calling itself GPTBot, Cloudflare could verify 13% as OpenAI.

Here is how to separate the impersonators on a Cloudflare free plan, and what the numbers looked like.

A User-Agent is a claim, not evidence

Writing GPTBot/1.2 into a header costs nothing, so a table grouped by UA is a list of claims.

Behind Cloudflare there are two things to check those claims against:

verifiedBotCategory — Cloudflare's reverse-DNS verification result. Empty string means unverified
Whether the requested path exists in your sitemap — separates reading an article from crawl bookkeeping such as robots.txt

What survives both filters is "verified bots fetching real pages", and that is the only number worth reporting.

The query (works on a free zone)

Add userAgent and verifiedBotCategory to the dimensions of httpRequestsAdaptiveGroups. clientAsn and botClass require a paid plan; verifiedBotCategory does not.

QUERY_BY_UA_VERIFIED = """
query ($zoneTag: String!, $since: Time!, $until: Time!) {
  viewer {
    zones(filter: { zoneTag: $zoneTag }) {
      httpRequestsAdaptiveGroups(
        limit: 5000
        filter: { datetime_geq: $since, datetime_leq: $until }
        orderBy: [count_DESC]
      ) {
        count
        dimensions { userAgent verifiedBotCategory }
      }
    }
  }
}
"""

Two constraints to plan around:

One request covers at most one day. Chunk the range in the caller
Retention is short (roughly a week at daily granularity). You cannot re-derive the past, so write a snapshot to disk on every run

def fetch_rows(token, query, zone_tag, since, until):
    # split the range into one-day windows and concatenate the rows
    all_rows, cursor = [], since
    one_day = dt.timedelta(days=1)
    while cursor < until:
        win_end = min(cursor + one_day, until)
        variables = {
            "zoneTag": zone_tag,
            "since": cursor.isoformat() + "Z",
            "until": win_end.isoformat() + "Z",
        }
        try:
            all_rows.extend(rows_from(gql(token, query, variables, exit_on_error=False)))
        except RuntimeError as e:
            print(f"    [skipped] {cursor.date()}-{win_end.date()}: {e}")   # past retention
        cursor = win_end
    return all_rows

One UA string, two rows

That single extra dimension is enough to split claim from reality. Eight days:

Claimed UA	Verified	Unverified	Verified share
ChatGPT-User	211 (AI Assistant)	336	39%
Amazonbot	135 (AI Crawler)	489	22%
ClaudeBot	133 (AI Crawler)	126	51%
OAI-SearchBot	61 (Search Engine Crawler)	132	32%
GPTBot	19 (AI Crawler)	123	13%
meta-externalagent	209 (AI Crawler)	0	100%
Applebot	56 (AI Search)	0	100%
PerplexityBot	0	144	0%
Perplexity-User	0	311	0%

Of 547 requests presenting as ChatGPT-User, 211 came from an address that traced back to OpenAI.

Only two agents came through clean — meta-externalagent and Applebot, 100% verified with zero impersonation. Those are the only rows whose claimed totals are usable as-is. All 455 Perplexity-branded requests were unverified.

One of the agents cannot exist

123 requests claimed Google-Extended. Verified share 0%, and 70 of them hit credential-scanning paths.

No inference required. Google's crawler documentation states that Google-Extended has no separate HTTP request user agent string: crawling happens under the existing Google user agents, and the token exists purely to be addressed in robots.txt for AI-training control.

So every request presenting that UA is, by definition, not Google. The same trick works for any operator that publishes IP ranges — OpenAI ships gptbot.json.

Classify the path, not just the client

Verification alone isn't enough: a verified bot fetching robots.txt has read nothing.

SCAN_PATTERNS = (
    "wp-", ".env", ".git", ".aws", ".svn", ".ssh", "secrets", "credentials",
    "config.json", "service_account", "actuator", "api/auth", "phpinfo",
    ".bak", ".yml", ".yaml", ".php", ".sql", "id_rsa", ".npmrc", ".htpasswd",
)
OPS_PREFIXES = ("/robots.txt", "/sitemap", "/llms.txt", "/favicon", "/rss", "/feed", "/.well-known/")
ASSET_PREFIXES = ("/_astro/", "/images/", "/assets/", "/fonts/", "/cdn-cgi/", "/_image")


def classify_path(path, sitemap_paths):
    # content = a real page was consumed / ops = crawl bookkeeping
    # asset = static file / scan = credential probing / other = path does not exist
    if not path:
        return "other"
    low = path.lower()
    if any(k in low for k in SCAN_PATTERNS):
        return "scan"
    if low.startswith(OPS_PREFIXES):
        return "ops"
    if low.startswith(ASSET_PREFIXES):
        return "asset"
    if not sitemap_paths:
        return "unknown"   # cannot assert existence, so cannot call it content
    return "content" if (path.rstrip("/") or "/") in sitemap_paths else "other"

Using the sitemap as the source of truth for existence is the part that holds up. Deciding from the response status looks easier, but redirects and paths that answer 200 without being real pages both leak in. The set of paths you declared public is a cleaner definition of "a page of mine".

Keep the unknown branch too. Fold it into content and your numbers spike on any day the sitemap fetch fails, with nothing in the output to explain it.

The result: 468 content fetches against 991 scans

Claimed UA	content	ops	scan	total	verified
ChatGPT-User	216	0	181	547	39%
meta-externalagent	93	9	0	209	100%
Amazonbot	85	1	284	624	22%
Applebot	27	9	0	56	100%
OAI-SearchBot	17	43	74	193	32%
PerplexityBot	13	8	67	144	0%
GPTBot	7	9	72	142	13%
ClaudeBot	6	129	70	259	51%
Google-Extended	0	0	70	123	0%
Perplexity-User	0	0	173	311	0%

content totals 468, scan totals 991. By status code, 403s came to 957 against 705 served with 200.

The ClaudeBot row is worth pulling apart: of 133 verified requests, 6 fetched articles and 129 fetched robots.txt and similar. "ClaudeBot sent 259 requests" and "six articles were read" are the same data.

Keep the verified share as a column in whatever you output. When a series' share collapses, that is your signal to stop reading it as a metric for that snapshot.

The week-over-week comparison (content 610 → 468 while scan went 357 → 991), why I cannot tell a genuine drop in interest apart from impersonation being reclassified, and the point where WAF blocks overtook served requests are all on Aulvem → Aulvem | AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

I Built a Crypto Market Intelligence App with React Native, Supabase & Cloudflare Workers AI

Alligator Peach — Sat, 22 Aug 2026 00:59:05 +0000

The problem: alert fatigue

Keeping up with crypto meant juggling five different apps — a price tracker, a scanner, news feeds, sentiment charts, and a stack of alert apps. And even then I kept missing the moves that mattered.

The existing tools fell into two camps:

Too noisy — thousands of alerts, most of them irrelevant.
Too shallow — pretty charts without any real analysis behind them.

So I built SignalWatch: a mobile app that turns overwhelming market data into a clear, prioritized signal board. It scans 100+ coins, understands plain-English questions, tracks market sentiment, and only alerts you about assets that actually pass the bar.

The stack

Layer	Technology
Mobile	React Native (Expo SDK 54), TypeScript, React Navigation 7
Auth & Sync	Supabase (Google OAuth + email/password)
Analysis engine	Node/Express on Cloud Run — scheduled every 15 minutes
Data ingestion	Cloudflare Workers (CoinMarketCap + Binance)
AI semantic scanner	Cloudflare Workers AI
Push notifications	FCM via a Supabase Edge Function
Market data	Binance & CoinMarketCap

Architecture

┌─────────────────────────────────────────────┐
│         React Native App (Expo)             │
│  Dashboard · Scanner · Signal · Calendar    │
└───────┬──────────────────┬──────────────────┘
        │ Supabase queries │ POST /api/ai-query (Bearer JWT)
        ▼                  ▼
  ┌────────────┐    ┌──────────────────┐
  │  Supabase  │    │ Cloudflare Worker│
  │  (Postgres)│    │  Workers AI      │
  └─────▲──────┘    └────────▲─────────┘
        │                    │
┌───────┴────────┐   ┌───────┴────────┐
│  Cloud Run     │   │  Cloudflare    │
│  (analysis,    │   │  Worker        │
│   every 15min) │   │  (CMC ingest)  │
└───────▲────────┘   └───────▲────────┘
        │                    │
        └──── Binance + CoinMarketCap ────┘

Two services feed a shared Supabase database:

Cloud Run backend — technical analysis (RSI, MACD, EMA, Bollinger Bands, SuperTrend) across futures & spot markets, plus economic calendar events.
Cloudflare Worker — CoinMarketCap ingestion (top 100, trending, fear & greed, news) and the AI query endpoint.

The app itself is architected with thin screens over a central hook (useDashboard) that acts as the single source of truth, with a clean service layer for all data access.

The feature I'm most proud of: the AI semantic scanner

Users can ask questions in plain English:

"show me coins with RSI < 30 and volume spike > 100%"

The query goes to a Cloudflare Worker, which detects intent, pulls real data from Supabase, and lets Workers AI compose a structured, data-backed answer. No hardcoded results, no fake confidence scores.

One subtle but critical detail: the app authenticates to the worker with the user's Supabase access token (Bearer), and the worker validates it against the Supabase JWKS. No secrets are ever embedded in the app binary.

// src/services/ai.ts — get a valid token, then call the worker
const getValidAccessToken = async (): Promise<string | null> => {
  const { data: userData } = await supabase.auth.getUser();
  if (userData?.user) {
    const { data: { session: refreshed } } = await supabase.auth.getSession();
    return refreshed?.access_token ?? null;
  }
  const { data: refreshData, error: refreshError } =
    await supabase.auth.refreshSession();
  if (!refreshError && refreshData.session?.access_token) {
    return refreshData.session.access_token;
  }
  return null;
};

⚠️ The ES256 vs RS256 gotcha

This project taught me a lesson: not all Supabase JWTs are RS256. This particular project signs tokens with ES256 (EC P-256). My worker's JWT verifier originally only handled RS256, so every valid token was rejected with a confusing 401 — "Please sign in" — even though the user was clearly signed in.

The fix was to support both ES256 and RS256 in the worker's verification path, plus logging exactly why verification failed (much easier to debug than a generic 401).

Lessons learned along the way

1. External sites will block your cloud IPs

ForexFactory (for the economic calendar's news feed) returns HTTP 403 to Cloud Run datacenter IPs. The calendar HTML page works fine, but the news scrape doesn't. My solution: a graceful fallback — if the forex_news table is empty or the request fails, the app falls back to CoinMarketCap news (news_insights), which is populated by the Cloudflare Worker. The section is never empty, and users can't tell the difference.

// src/services/marketCalendar.ts — fallback to CMC news when forex news fails
export const fetchForexNews = async (limit = 10): Promise<ForexNewsRow[]> => {
  const { data, error } = await supabase
    .from('forex_news').select('*')
    .order('published_at', { ascending: false })
    .limit(limit);

  const forexNews = (data || []) as ForexNewsRow[];
  if (!error && forexNews.length > 0) return forexNews;
  // ...fallback to news_insights (CoinMarketCap via worker)
};

2. Data freshness is a feature

A pipeline that ingests every 15 minutes across multiple timeframes (15m / 1h / 4h / 1d) means the "now" is actually now. That matters enormously in fast-moving markets — and it's a real competitive differentiator, not a buzzword.

3. Alert fatigue is the real enemy

Most tools push volume, not signal. SignalWatch bakes in quality filters so users only get notified about assets that actually pass the bar. It sounds obvious, but it's the thing that makes the product feel calm instead of chaotic.

4. Keep the app architecture boring

Thin screens → one central hook → a service layer → Supabase/worker. It kept a solo developer sane, made the codebase easy to grow, and made the light/dark theme + EN/ID localization trivial to add.

The result

🎯 AI scanner you can talk to (Cloudflare Workers AI over live data)
📊 Multi-timeframe technical analysis on 100+ coins
🌍 Global sentiment (Fear & Greed, bullish ratio, market score)
🔔 Quality-filtered push notifications (no spam)
⭐ Synced watchlist, 📅 economic calendar, 📰 impact-labeled news
🌗 Light/dark + EN/ID, fully customizable

Currently available on Google Play with all Pro features free. iOS is next (the codebase is Expo, so the path is clear).

If you've hit similar walls with external APIs blocking cloud IPs, or JWT algorithms that don't match the docs — I'd love to hear how you solved it. Feedback and feature ideas are very welcome. 👇

Links: Google Play

Cloudflare Bot Preference Sync will write your robots.txt from 21 August 2026 - and it ignores your custom rules

Manu Shukla — Fri, 21 Aug 2026 23:42:54 +0000

Cloudflare Bot Preference Sync will write your robots.txt from 21 August 2026 - and it ignores your custom rules

Summary. Cloudflare published "Say it once: introducing Bot Preference SynC" at 23:19 UTC on 21 August 2026. The feature generates or rewrites your robots.txt from the Search, Agent and Training settings you already hold in the zone dashboard, and it is promised to every plan tier from Free to Enterprise "in the coming week". Three details in that post matter more than the headline. It works only on category-wide policy, so it will not read your per-crawler custom rules. It is on by default for all new customers. And as of this writing the documentation page it needs, at /ai-crawl-control/features/managed-robots-txt/, returns a 404, while the page that does exist - the legacy robots.txt setting doc, last updated 3 August 2026 - still describes a different mechanism that hard-writes Disallow: / for 8 named crawlers. If you run a site behind Cloudflare with a hand-maintained robots.txt, the sequencing here decides whether your file says what you think it says on 15 September 2026, when Cloudflare's new AI traffic defaults take effect.

What Cloudflare actually announced

The mechanism is narrow and worth stating exactly. Bot Preference Sync does not introduce new blocking. It takes the AI bot policy you have already configured at zone level - Allow, Block on pages that serve ads, or Block everywhere for Search and Agent, plus Disallow for Training - and reflects that policy as text inside robots.txt.

If your origin already serves a robots.txt, Cloudflare prepends its block ahead of your content rather than replacing it, wrapped in two comment markers:

# BEGIN Cloudflare Bot Preference Sync

User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /

...

# END Cloudflare Bot Preference Sync

That example is Cloudflare's own, shortened and anonymised in the post. The real user-agent list is pulled from BotBase, Cloudflare's bot database launched on 1 July 2026, and Cloudflare says it will "periodically update the list of bots that is added to robots.txt when you choose to Block or Disallow a given category". You do not pin that list. Cloudflare does, and it changes underneath you.

The stated goal is to close the gap between what a site declares and what it enforces. As the announcement puts it: "When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules." That is a fair description of the problem. It is also the reason the custom-rules carve-out below matters.

The part the announcement does not lead with

Three limits sit further down the post, and each one changes an operational decision.

Category-wide only. Cloudflare states plainly that Bot Preference Sync "is designed to tackle policy decisions made category-wide rather than case-by-case, it will not directly read from individual custom rules with more complex logic." If you have negotiated an exception for one named crawler - a licensing deal, a partner index, an internal scraper - that exception lives in a custom rule and will not appear in the generated file. Your robots.txt will then advertise a stricter policy than your edge actually enforces, which is the exact mismatch the feature was built to remove. Cloudflare's answer is to turn the sync off and hand-maintain the file.

On by default for new customers. The post says Bot Preference Sync "will be on by default" for all new customers. Read alongside the next paragraph, that means the sync is on, not that blocks are on: for non-publisher new customers Cloudflare "will not have any blocks or disallows added by default". The two statements are consistent, but they read as contradictory on a first pass, and the practical effect is that a newly onboarded zone gets a Cloudflare-managed robots.txt file whether or not it has any AI policy set.

A new publisher default. At onboarding, a customer can select "I monetize from pages with ads on this domain", which sets Training to Disallow as the default. That is an opt-in checkbox with a policy consequence, changeable at any time.

The docs have not caught up

The AI Crawl Control documentation set does not contain a Bot Preference Sync page. Requesting /ai-crawl-control/features/managed-robots-txt/ returns Cloudflare's 404 template. The navigation lists Analyze AI traffic, Manage AI crawlers, and Directives - the tab formerly called Robots.txt - and nothing else in that area. The Directives page itself carries a "Last updated Apr 23, 2026" stamp and still points its managed-status card at the older bots/additional-configurations/managed-robots-txt/ page.

That older page, stamped 3 August 2026 in its own structured data, documents a materially different file. It writes a fixed content-signal line and a blanket disallow for a named list:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

The full list in that document is 8 crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent. Turn on the legacy toggle today - Security Settings, filter by Bot traffic, "Set your preference to block training in robots.txt" - and that is what lands in your file. It is not the Search/Agent/Training model the 21 August post describes.

Behaviour	Legacy managed robots.txt (doc updated 3 Aug 2026)	Bot Preference Sync (announced 21 Aug 2026)
Source of directives	Fixed list of 8 named crawlers	BotBase categories, periodically refreshed
Policy model	Block training, one setting	Search, Agent, Training set separately
Reads your custom rules	No	No - category-wide only
Placement in your file	Prepended above existing content	Prepended above existing content
Documentation	Live page, all plans	No page; 404 at the expected path

Why the 15 September date is the real deadline

Bot Preference Sync publishes a preference. It does not decide who gets blocked at the edge, and the edge rules change on a fixed date.

In "Your site, your rules: new AI traffic options for all customers", published 1 July 2026, Cloudflare wrote: "On September 15, 2026, we'll be setting new defaults for each of these three classifications." For all new domains onboarding to Cloudflare, Training and Agent are blocked by default on pages that display ads, while Search stays allowed. The same post carries the sentence that costs the most money if you miss it: multi-purpose crawlers will be "allowed/blocked according to all of their behaviors", with the most restrictive applicable rule winning, so "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)."

Read that twice if you turned on "Block AI bots" in 2025 and have not looked since. Cloudflare's opt-out is a setting in the zone's Security settings, available "any time leading up to September 15", which confirms you want no change to Training crawlers that also crawl for Search.

The 21 August post adds a partial escape hatch on the preference side. The Training Disallow option writes a "no training" preference rather than a hard block, and Cloudflare says cooperating mixed-use crawlers that meet four transparency requirements keep search access. Those four requirements, verbatim from the post, are that the bot respects a "no training" preference in robots.txt by any mechanism; gives site owners a way to opt out of AI summaries; provides URL-level visibility into which pages were made available for training plus metrics on search results; and can show publicly that disallowing training does not hurt traditional search results. Operators meeting them are tracked in the AI bot transparency section on Cloudflare Radar. Operators that do not "are still blocked when you disallow training".

So the preference layer and the enforcement layer resolve differently, on different dates, using different lists. That is the whole problem for anyone measuring AI search visibility.

How to tell whether this is you

Four checks, none of which take longer than a few minutes.

Fetch your own robots.txt and look for # BEGIN Cloudflare Managed content or # BEGIN Cloudflare Bot Preference Sync. Either marker means Cloudflare is writing part of your file.
Look for a Content-signal: line. Cloudflare's managed content sets search=yes, ai-train=no, use=reference. The use field is an optional extension Cloudflare describes as under test, with three values: use=immediate (interact, store and reuse nothing), use=reference (index, excerpt, and link back), and use=full (summarize and reproduce).
Open Security Settings and check whether "Block AI bots" or the legacy training-preference toggle is on. If it is, the 15 September multi-purpose rule applies to you.
Check whether any per-crawler exception you rely on lives in a custom rule. If it does, Bot Preference Sync will not represent it.

One more thing worth knowing before anyone panics about search Console noise: Cloudflare's own documentation states that "Google Search Console may occasionally report Syntax not understood for Content Signals and newer directives in the robots.txt standard. However, we have observed no impact on crawling rates or SEO as a result of these reports." That is Cloudflare's observation, not Google's statement, and it is worth treating as a vendor claim rather than a settled fact.

India-specific considerations

Indian publishers and D2C brands sitting on Cloudflare's Free tier are affected by a detail that is easy to miss. Free-plan domains that have no robots.txt of their own and do not use the managed feature are served Cloudflare's Content Signals Policy text when a crawler requests the file. That policy defines the search, ai-input and ai-train categories and explicitly reserves rights under Article 4 of EU Directive 2019/790, but it expresses no preference of its own. A site owner in Gurugram or Bengaluru reading their own robots.txt may reasonably conclude they have opted out of training when they have not.

For teams whose Indian traffic depends on AI-assisted discovery rather than classical blue links, the practical order is: decide the Training position first, confirm the 15 September opt-out state second, and only then decide whether to let Cloudflare own the file. robots.txt compliance remains voluntary in every case - Cloudflare's documentation says so directly: the file "expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level."

What is still unknown

Cloudflare has not published the generated user-agent list for any category, the refresh cadence for that list beyond "periodically", the API or Terraform surface for Bot Preference Sync, or a dated availability commitment beyond "in the coming week". There is no changelog entry in the AI Crawl Control changelog as of this writing. Until the documentation page exists, the only authoritative description of the feature is a blog post, and blog posts are not a contract.

The safe operational position for the next fortnight is unchanged: keep ownership of the file you can diff. If you have a hand-maintained robots.txt with per-crawler exceptions, leave the sync off until the docs land, and re-check your setting on 14 September. The real cost here is usually the audit after the fact, not the toggle.

FAQ

What is Cloudflare Bot Preference Sync?

Bot Preference Sync is a Cloudflare feature announced on 21 August 2026 that generates or updates your robots.txt file to match the Search, Agent and Training policies you set in the zone dashboard. Cloudflare says it will be available on every plan tier, from Free to Enterprise, in the week following the announcement.

Does Bot Preference Sync delete my existing robots.txt?

No. Cloudflare states that its generated content is prepended to your existing material, so any Disallow directives already in your file are maintained. The added block sits between the markers BEGIN Cloudflare Bot Preference Sync and END Cloudflare Bot Preference Sync, above whatever your origin already serves.

Will it reflect my custom crawler rules?

No. Cloudflare says the feature handles policy decisions made category-wide rather than case-by-case, and that it will not directly read from individual custom rules with more complex logic. If you rely on a per-crawler exception, turn the sync off and maintain the file yourself, or your published preference will not match your enforcement.

What changes on 15 September 2026?

Cloudflare's 1 July 2026 post set new defaults from that date. For all new domains, Training and Agent are blocked by default on pages that display ads, while Search stays allowed. Multi-purpose crawlers are then judged on all their behaviours, with the most restrictive applicable rule winning.

Will Googlebot be blocked if I block training?

Cloudflare's 1 July 2026 post says multi-purpose crawlers such as Googlebot, Applebot and BingBot will be blocked by customers who have selected to block Training, including through the legacy Block AI bots service. Cloudflare offers an opt-out in zone Security settings, available at any time before 15 September 2026.

What are the four transparency requirements for mixed-use bots?

Per the 21 August post, a Search-and-Training bot must respect a no-training preference in robots.txt by any mechanism, give site owners a way to opt out of AI summaries, provide URL-level visibility into pages made available for training plus search metrics, and show publicly that disallowing training does not harm traditional search results.

Is there documentation for Bot Preference Sync?

Not yet. The expected path under AI Crawl Control returns a 404, and the AI Crawl Control changelog carries no entry. The page that does exist documents the older managed robots.txt setting, stamped 3 August 2026, which writes a blanket Disallow for eight named crawlers instead.

Does robots.txt actually stop AI crawlers?

No. Cloudflare's own documentation says compliance is voluntary: the file expresses preferences but does not prevent access at a technical level, and some operators disregard Disallow directives. Enforcement requires edge blocking through AI Crawl Control or WAF rules, which is a separate configuration from the preference file.

How eCorpIT can help

eCorpIT runs technical AI-search audits for teams whose discovery now depends on answer engines as much as on classical search results, and this is exactly the class of change those audits catch: a vendor default that rewrites a file nobody on the team owns. Our SEO, AEO and GEO audit service checks the served robots.txt, the content-signal state, the edge rules behind it, and whether the two agree. eCorpIT is CMMI Level 5, MSME certified and ISO 27001:2022 certified. Tell us your Cloudflare plan tier and current Training setting at /contact-us/ and we will tell you what changes on 15 September.

References

Cloudflare Blog, Say it once: introducing Bot Preference SynC, 21 August 2026.
Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, 1 July 2026.
Cloudflare Docs, robots.txt setting, last updated 3 August 2026.
Cloudflare Docs, AI Crawl Control overview.
Cloudflare Docs, Directives (formerly Robots.txt), last updated 23 April 2026.
Cloudflare Docs, Manage AI crawlers.
Cloudflare Docs, BotBase.
Cloudflare Docs, Verified bots.
Cloudflare Docs, AI Crawl Control changelog.
Cloudflare Radar, AI Insights and AI bot transparency.
Cloudflare Radar, public bots directory.
Cloudflare Docs, AI Crawl Control with Cloudflare WAF.

Last updated: 22 August 2026.

Alibaba Tumbles on Earnings Miss; Tesla Robotaxi Approval and Cloudflare Agent Surge Boost AI Stocks

StartupHub.ai — Fri, 21 Aug 2026 23:26:03 +0000

Alibaba Tumbles on Earnings Miss; Tesla Robotaxi Approval and Cloudflare Agent Surge Boost AI Stocks — alibaba tumbles earnings miss tesla robotaxi

In a day marked by significant sector-specific movements, Alibaba Group Holding Ltd. experienced a sharp decline, while Tesla Inc. and Cloudflare Inc. saw substantial gains, highlighting the dynamic nature of the AI stock landscape. This analysis delves into the key factors driving these shifts, including Alibaba's earnings miss, Tesla's robotaxi milestone, and Cloudflare's growing role in AI-agent traffic.

Alibaba's Earnings Setback and AI Spending Concerns

Alibaba (NYSE: BABA) slid 8.6% to $119.34, its most significant single-day drop in months. This downturn followed the release of its fiscal Q1 2027 results, which revealed adjusted earnings per ADS of $1.26, a 42% year-over-year decrease and significantly below the $1.85 consensus estimate. While cloud revenue growth reached its fastest pace in 22 quarters at 45%, investors were concerned by the revelation that AI capital expenditure is currently running at 4.5 times the rate of new cloud revenue growth. This imbalance between aggressive infrastructure spending and near-term returns is a pattern observed across the AI sector, as tracked by StartupHub.ai's global database of over 91,000 companies. For Alibaba to regain investor confidence, management must demonstrate a clearer path to narrowing this spending-revenue gap.

Tesla Secures Key Robotaxi Approval

Tesla Inc. (NASDAQ: TSLA) surged 5.1% to $362.86, extending its recent positive momentum. The catalyst for this rise was the approval granted by Nevada regulators for up to 5,000 Tesla robotaxis to operate in Las Vegas. This permit signifies a significant step towards a commercial rollout of Tesla's autonomous fleet, placing it ahead of many competitors. Concurrently, Tesla announced plans to showcase its electric Semi truck in Germany, underscoring its expanding presence in the commercial vehicle market. CEO Elon Musk remains optimistic about Tesla's ability to surpass Wall Street's revenue forecasts.

Cloudflare Dominates AI-Agent Network Traffic

Cloudflare Inc. (NYSE: NET) climbed 5.1% to $293.14 after exceeding second-quarter expectations and raising its full-year guidance. A pivotal announcement was that non-human and AI-agent traffic now constitutes over 50% of its total network volume. This milestone positions Cloudflare as a critical infrastructure provider for the burgeoning wave of agentic AI. Operating at the nexus of AI inference delivery and zero-trust security, Cloudflare is well-positioned to capitalize on key enterprise IT spending trends in 2026. The stock has seen a year-to-date increase of 49.6%, making it a notable player for investors seeking exposure to AI-agent infrastructure.

Broader Market Movements and Key AI Stocks

Other notable movers included SoundHound AI Inc. (NASDAQ: SOUN), which added 5.0%; Snowflake Inc. (NYSE: SNOW), up 3.6% driven by AI data platform spending; and Palantir Technologies Inc. (NYSE: PLTR), which rose 3.4% to continue a strong monthly performance. On the downside, Arm Holdings plc (NASDAQ: ARM) dropped 3.0%, and Intel Corp. (NASDAQ: INTC) slid 2.2%.

The broader AI stock universe, encompassing hardware, hyperscale cloud, and enterprise SaaS, continues to focus on NVIDIA Corp. (NVDA), Broadcom Inc. (AVGO), and Taiwan Semiconductor Manufacturing Co. (TSM) for hardware; Microsoft Corp. (MSFT) and Alphabet Inc. (GOOGL) for cloud infrastructure; and Palantir (PLTR) and Oracle Corp. (ORCL) for enterprise AI software. The market's attention has shifted from AI training clusters to inference delivery, AI-agent monetization, and autonomous deployment. The upcoming NVIDIA Q2 FY27 earnings report on August 26 is anticipated as a major catalyst for the sector. Investors are also closely watching the Jackson Hole Economic Symposium for signals on future interest rate policy, which could impact rate-sensitive tech stocks.

Conclusion: Navigating the AI Investment Landscape

The market's reaction to Alibaba's earnings miss, coupled with positive developments for Tesla and Cloudflare, underscores the evolving dynamics within the AI sector. While concerns about capital expenditure pacing persist, significant advancements in autonomous technology and AI infrastructure continue to drive innovation and investment. As the sector matures, the focus remains on companies demonstrating clear monetization strategies and sustainable growth in the rapidly expanding AI ecosystem. The upcoming earnings reports and economic symposiums will be crucial in shaping market sentiment for the remainder of 2026. Investors looking for insights into the broader market trends, including the tesla alphabet drag mega-cap lower earnings, will find valuable context in ongoing analyses.

This article was originally published on StartupHub.ai.

Your Worker Returned 500 and the Log Says `outcome: "ok"`

ushiro — Fri, 21 Aug 2026 12:00:00 +0000

I run AI Change Watch, a small independent project that
crawls what 15 AI vendors publish about their own models — deprecation tables, lifecycle pages, pricing
and SDK releases — and records every time one of them changes.

It runs on Cloudflare Workers, which means that when someone tells me "your site 500'd an hour ago",
the obvious tool is useless. wrangler tail is a live stream. It shows you what is happening now.
It cannot show you an hour ago.

There is a way to read the past, and there are four traps in it that cost me most of a day.

The part that works

Workers can write their invocation logs to a queryable store, and a REST endpoint reads it back. First
the worker has to be opted in — this is the whole config:

// wrangler.jsonc
{
  "observability": { "enabled": true }
}

Then you can ask for events in a time range:

curl -sX POST \
  "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/workers/observability/telemetry/query" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "queryId": "anything",
    "timeframe": { "from": 1786000000000, "to": 1786003600000 },
    "limit": 500,
    "view": "events",
    "parameters": {
      "datasets": ["cloudflare-workers"],
      "filters": [
        { "id": "f1", "key": "$workers.event.response.status",
          "type": "number", "operation": "eq", "value": 500 }
      ]
    }
  }'

from and to are epoch milliseconds, not ISO strings. A read-scoped API token is enough — the
one I already had for deploys worked unchanged.

Each event carries more than you would guess:

$workers.outcome                              ok | exceededCpu | canceled
$workers.cpuTimeMs  /  $workers.wallTimeMs
$workers.event.request.path  /  .search
$workers.event.request.headers['user-agent']
$workers.event.request.cf.asOrganization      the ASN owner
$workers.event.response.status
$metadata.error

That is the tool. Now the traps.

Trap 1: a rendered 500 is a successful invocation

I started by filtering on the field that sounds right:

{ "key": "$workers.outcome", "operation": "eq", "value": "exceededCpu" }

and found nothing, repeatedly, while the site was demonstrably returning 500s.

Because for every one of them:

$workers.outcome = "ok"

The worker ran. It produced a response. It returned it. That the response was an error page is not the
runtime's problem — the invocation succeeded. outcome describes the worker, not the HTTP result.

So outcome is the wrong axis for application errors. Filter on $workers.event.response.status for
what the user saw, and read $metadata.error for the throw. outcome is for failures the runtime
itself noticed: CPU limit, cancellation.

This is worth internalising because it inverts the usual relationship. In most stacks "the request
failed" and "the handler failed" are the same event. At the edge they are two different fields — and
the one with the friendlier name is the one that will not tell you.

Trap 2: a wide time window silently undercounts

This is the one that actually cost me the day.

I asked for 5xx across a 24-hour window, got nine events, and concluded I was chasing a single bug. The
same 24 hours, walked in 4-hour slices and concatenated, returned 956 — across 92 URLs and three
unrelated causes.

I re-ran the comparison today, on 404s, to check it was not a one-off:

24h asked as one query      →  26 events
same 24h in 4h slices       → 266 events

The single query returned 10% of what was there. Not a rounding difference — a different
conclusion. And nothing in the response says so: no truncation flag, no "results were sampled" field.
You get a well-formed answer that happens to be mostly missing.

So the loop, not the query:

const out = [];
for (let h = 24; h > 0; h -= 4) {
  const from = Date.now() - h * 3600_000;
  const to   = Date.now() - (h - 4) * 3600_000;
  const ev = await queryEvents({ from, to, limit: 500 });
  if (ev.length >= 500) console.warn(`slice ${h}h hit the limit — narrow it`);
  out.push(...ev);
}

The ev.length >= 500 check matters as much as the slicing. A slice that returns exactly your limit is
truncated, and you have to narrow that slice further. Without the warning you cannot tell "500 events
happened" from "500 events fit".

Trap 3: `exists` matches empty strings, and `includes` ignores case

Two smaller ones, both of which produced confidently wrong numbers before I noticed.

operation: "exists" matches a key that is present but empty. I wanted requests Cloudflare had
identified as verified bots:

{ "key": "$workers.event.request.cf.verifiedBotCategory", "operation": "exists" }

That field is present on every request and is "" on almost all of them, so the filter matched the
entire dataset and I briefly believed the whole site was bot traffic. Use exists only for keys
genuinely absent on what you are excluding — sec-fetch-mode is a real example, since non-browsers do
not send it.

operation: "includes" is case-insensitive. Filtering user agents for bot and for Bot returned
the identical 3,181 events. Convenient once you know; misleading if you were using case to separate two
populations.

Trap 4: event counts are not invocation counts

The events view and the dashboard's invocation count disagree, and both are right. On one day my web
worker showed 24,085 telemetry events against 8,772 invocations — roughly 2.7 events per
invocation.

So:

proportions — "what share of requests were 404s", "which UA dominates" — take from the events view
absolute totals — "how many requests did this worker serve" — take from workersInvocationsAdaptive in the GraphQL analytics API, not from counting events

Mixing them gives a number that is wrong by a factor you cannot see.

What I actually run now

// Past-tense debugging: "what 500'd between 3am and 4am".
async function queryEvents({ from, to, limit = 500, filters }) {
  const r = await fetch(
    `https://api.cloudflare.com/client/v4/accounts/${ACCOUNT}/workers/observability/telemetry/query`,
    { method: 'POST',
      headers: { Authorization: `Bearer ${TOKEN}`, 'Content-Type': 'application/json' },
      body: JSON.stringify({
        queryId: 'q', timeframe: { from, to }, limit, view: 'events',
        parameters: { datasets: ['cloudflare-workers'], filters },
      }) });
  const d = await r.json();
  return d?.result?.events?.events ?? [];
}

…called from the slicing loop above, with the results grouped in plain JavaScript rather than by asking
the API to group them. (view: "calculations" with a groupBy on a high-cardinality key returns only a
few groups, quietly — the same failure mode as trap 2: a well-formed answer that is mostly missing.)

The retention window is limited. I have reliably queried three days back and would not build a workflow
that assumes more; for anything you need to keep, pull it out and store it yourself.

One more, learned the embarrassing way: cf.asOrganization is the ASN owner, not the bot. Requests
from "Anthropic, PBC" turned out to be a crawler that robots.txt already allowed, and "Amazon
Technologies" was PerplexityBot. Identify declared crawlers by user agent; use the ASN only to catch
traffic whose user agent is lying.

The one-line version

wrangler tail is for watching. For asking, use the telemetry API — and remember that outcome: "ok"
means the worker succeeded, not that your user did, and that a query covering a wide window will hand
you a confident answer built from a tenth of the data.

The tracker this came out of is at aichangewatch.com — it watches AI
vendor docs for changes, and the 500s that started all this were a REST detail endpoint quietly falling
through to its collection endpoint.

Running a Monitoring SaaS on Cloudflare Workers + Supabase for Almost Nothing

Merlonix — Fri, 21 Aug 2026 10:41:00 +0000

Originally published on the Merlonix blog.

Merlonix monitors uptime, SSL/TLS, DNS, email authentication, blacklists, Certificate Transparency, Core Web Vitals, and MCP servers for agencies. A monitoring product has an unforgiving shape: it must run continuously, hit arbitrary customer-supplied hostnames, and stay up more reliably than the things it watches — while, in our case, keeping the infrastructure bill within a rounding error of zero until revenue exists to justify more.

This post is the real architecture, including the parts that bit us. Nothing here is a reference design we aspire to; every component named below is deployed and verifiable from the outside.

The shape of the system

Compute: eight Cloudflare Workers. One HTTP API worker (Hono) serves everything under api.merlonix.com. Seven background workers do the actual monitoring: a scheduler (cron), a check-runner and vendor-runner (queue consumers that execute checks), a vendor-fetcher, a dlq-consumer (dead-letter forensics), a browser-runner, and a db-backup worker. Each has its own wrangler config and its own deploy verifier.

Frontend: two static Cloudflare Pages projects. The marketing site and the app are both Next.js static exports — no SSR servers, no origin to fall over. Anything dynamic goes through the API worker. A Pages Function provides the thin middleware layer (redirects, custom-domain status-page routing).

Glue: Cloudflare Queues. The scheduler enqueues due work onto checks-q and vendor-q; the runners consume in batches of 10. Failures retry, and exhausted retries land in a dead-letter queue with a consumer that records the forensic payload instead of dropping it.

State: one Supabase Postgres. Every table that holds tenant data runs with forced row-level security — the API worker uses the service role deliberately and narrowly, and RLS is audited by a script that enumerates deny-all tables and cross-tenant probes. The migration ledger is past 270 forward-only migrations, applied to production by an idempotent runner. There is no second database; the queue messages carry IDs, and Postgres is the single source of truth.

The revenue-gated cron throttle

The scheduler's production cron fires every minute. But firing and working are different things:

/** Revenue-gated SLA throttle. */
export function isSweepDue(now: Date, hasLiveSubscription: boolean): boolean {
  if (hasLiveSubscription) return true;
  return now.getUTCMinutes() % 5 === 0;
}

With no live customer, a sweep only does real work on every 5th UTC minute — byte-for-byte the enqueue volume of a 5-minute cron, which costs effectively nothing while there's nobody to monitor but seeded assets. The moment any revenue-bearing subscription exists, the very next tick restores full per-minute cadence. No redeploy, no flag flip, no human. The SLA follows the money automatically.

The interesting part is the failure mode we shipped and later fixed. The hasLiveSubscription check queries the subscriptions table, and the original catch block was bare:

} catch { hasLiveSubscription = false; }

Read that as an SRE: a transient database blip on this one query would silently throttle paying customers' monitoring cadence from 1 minute back to 5 — no error, no alert, checks just quietly late. It's the worst kind of degradation: invisible, revenue-adjacent, and plausible-deniable. The fix keeps the safe degradation (bootstrap mode beats crashing the sweep) but fans the failure out to Sentry and structured logs under its own error code, so a sustained degradation pages before a customer notices. If you take one pattern from this post: when you degrade gracefully, make the degradation loud.

SSRF-guarding the public probes

Our free tools and monitoring checks fetch URLs and hostnames that strangers type into a form. That is the textbook server-side request forgery setup: the classic target is 169.254.169.254, the cloud metadata endpoint, where a successful internal request leaks the execution environment's credentials.

Workers add a twist: fetch() doesn't expose the resolved IP and gives you no way to pin one. So the guard works like this:

Pre-resolve via DNS-over-HTTPS (1.1.1.1) and check every A/AAAA record against a blocklist of loopback, RFC1918, link-local (metadata!), CGNAT, unspecified, and IPv6 unique-local/link-local ranges. Any private answer → reject before fetching.
Redirects are followed manually, once. A malicious server can 302 to http://127.0.0.1; we re-run the full SSRF check on the Location header before following, and a second hop is always rejected.
Responses are size-capped and time-capped so a hostile endpoint can't stall a worker or balloon memory.

And the honest residual, documented in the code rather than papered over: the runtime's fetch() performs its own DNS resolution, so a sub-TTL attacker could answer our DoH probe with a public IP and the runtime with a private one. IP-pinning isn't viable on Workers (TLS validates against SNI). The authoritative backstop is Cloudflare's platform egress policy — Workers cannot open connections into loopback/RFC1918/link-local regardless of DNS. Our DoH layer is defense-in-depth on top of that, and we say so, because an SSRF guard you overstate is worse than one you understand.

Watching the watcher, from a different cloud

A monitoring company that monitors itself with itself has a bootstrapping problem: if Cloudflare has an account-level bad day, the thing that would tell us is also having a bad day.

So the external watchdog doesn't run on Cloudflare at all. It runs inside Supabase — which sits on AWS — using pg_cron plus the http extension: every 5 minutes, a SECURITY DEFINER function curls merlonix.com and the API's /healthz and /readyz endpoints, tracks consecutive failures in a table, and posts to an operator Discord webhook after a sustained-failure threshold, with a single recovery notice when things come back. No third-party account, no additional bill, and — the actual point — no shared fate with the platform it watches.

Boring reliability plumbing that earns its keep

Every worker invocation writes a heartbeat row to a worker_runs ledger (worker name, outcome, duration). "Is the scheduler actually running?" is a SQL query, not a guess — and a failed heartbeat insert has its own alarm code, because observability that fails silently isn't observability.
Every deploy has a verifier script, and deploys aren't done until it passes: the API verifier checks live routes and CORS, the workers verifier catches a stale or forgotten worker plus queue/cron/binding drift via the Cloudflare REST API, and a headless-browser smoke pass loads the public pages and fails on console errors, first-party network failures, and layout regressions that string-matching verifiers are blind to. Rollback scripts are pre-written; a failed verifier means roll back, not debug-in-prod.
The dead-letter queue has a consumer. DLQs with no consumer are where failures go to be forgotten; ours records each exhausted payload so a storm becomes a diagnosable dataset. That mattered the day a decommissioned LLM model name plus a router that hard-threw instead of falling back flooded it — root-caused from the forensics, fixed, regression-guarded.

What it costs

Nearly nothing, and that's a design constraint, not an accident. Static Pages sites are free. Workers requests at our current scale sit comfortably inside the Workers plan floor. Supabase is on the free tier — the watchdog cron and RLS-forced Postgres both fit inside it. The paid-API checks that could cost real money (PageSpeed Insights, LLM-backed features) sit behind explicit flags, daily caps, and per-tenant meters, so the worst case of a bug is a rate-limit, not a bill. The whole stack is engineered so the monthly infrastructure bill stays within a rounding error of zero until customers exist — at which point the same cron throttle that saves money today upgrades their SLA on the next tick.

If you're building on this stack

The pattern that generalizes: cron fires cheap and constant; a pure function decides whether the tick does work. It gives you a testable throttle (isSweepDue is three lines and unit-tested), a zero-redeploy upgrade path, and one place where cadence policy lives. Pair it with loud degradation, verify every deploy from the outside, and put your last-resort watchdog on somebody else's cloud.

The product this architecture serves is Merlonix — monitoring for agencies, from uptime and SSL through MCP server health. The free tools run the same SSRF-guarded probe path described above; you can watch it work without signing up.

Building a streaming investigation assistant that tool-calls an MCP server and cites case IDs

Royal Simpson Pinto — Fri, 21 Aug 2026 09:30:28 +0000

I keep a public registry of documented AI-agent failures called AgentPostmortem: real incidents where an agent got prompt-injected, deleted a database, or spun into a runaway loop, each written up as a case file with an ID like APM-0048. The registry is useful, but browsing it is a chore. If I want to know why refund agents get prompt-injected, I have to guess search terms, open cases, and read them one by one.

So I built casebook-chat: a chat interface that does the searching for me. You describe an incident or ask about a failure mode, and the assistant searches the live registry, pulls the relevant case files, and answers in plain language while citing the real case IDs it used. The whole thing runs as a single Cloudflare Worker.

The core idea

The registry already speaks MCP (Model Context Protocol). It exposes three tools over JSON-RPC at mcp.agentpostmortem.com/mcp. Rather than reimplement search or copy the data into a new database, I wanted the chat model to call those tools directly and ground its answers in whatever they returned.

The three tools map cleanly onto how you actually investigate an incident:

search_cases(query) runs a full-text search over the failure case files and returns ranked summaries with case IDs.
get_case(id) fetches one case in full: outcome, verified facts, unknowns, and lessons.
similar_failures(description) takes a plain-language description of an incident and matches it against known failures.

The model decides which to call and when. Ask "why do refund agents get prompt-injected?" and it searches. Say "my agent deleted a database" and it reaches for similar_failures. Once it has a case ID, it can pull the full detail with get_case.

How it works

The stack is deliberately small. The browser runs a Vite and React app using @ai-sdk/react's useChat, which POSTs the message history to /api/chat. That endpoint is a Hono route inside a Cloudflare Worker, and the same Worker also serves the static UI through Workers assets. One deploy, one origin, no separate backend.

Inside the route, I call the Vercel AI SDK's streamText against Groq's llama-3.3-70b-versatile, passing the three tools defined with the AI SDK tool() helper and zod input schemas. Each tool's execute is just a fetch to the MCP server wrapping the arguments in a JSON-RPC tools/call:

search_cases: tool({
  description: "Full-text search over documented AI-agent failure cases...",
  inputSchema: z.object({ query: z.string() }),
  execute: async ({ query }) => callMcpTool("search_cases", { query }),
}),

Because these are genuine failure investigations, the model often needs more than one hop: search, find a promising case, then fetch its full detail before answering. I allow multi-step tool chains but cap the loop with stopWhen: stepCountIs(5), so a model that keeps calling tools cannot run away and burn the whole budget.

Streaming with tool calls rendered inline

The nice part is the streaming. The result comes back through toUIMessageStreamResponse(), which turns the run into a UI message stream where tool inputs and outputs arrive as typed message parts, not just text. On the client, useChat exposes each assistant message as an ordered list of parts, and I render them in order: text parts become markdown, and anything whose type starts with tool- becomes an inline activity chip.

Each chip is a collapsible component that reads the part's state field to show where the call is. While the model is still forming or running the call (input-streaming, input-available), it shows a spinner and the word "running". When the output lands (output-available) it flips to a checkmark and a one-line summary; on failure (output-error) it shows a cross and the error text. Expand a chip and you see the exact arguments the model sent and the raw result it got back, pretty-printed. So the tool calls appear woven into the answer as it streams, in the order they actually happened, and you can audit every step.

Citations that mean something

Grounding is enforced through the system prompt, not wishful thinking. The investigator is told to answer only from what the tools returned, to cite the real case IDs it used, and never to invent case IDs or facts. If the registry returns nothing relevant, it is instructed to say so plainly and label any general-knowledge answer as such. It also calls tools silently instead of narrating "let me search the registry", so the visible answer stays clean while the chips carry the process.

The MCP client itself is built to never throw. It retries once on a network error or a 5xx, times out each attempt at around eight seconds, and on persistent failure returns a structured "the registry is temporarily unavailable" string that the model is told to relay to the user rather than crash on. It also handles the fact that the MCP server can answer either as JSON or as a text/event-stream, parsing the last JSON-RPC frame out of the SSE body when needed.

One honest limitation

The grounding is only as strong as the prompt. There is no post-hoc verification step that checks the case IDs in the final answer against the IDs the tools actually returned. The system prompt tells the model not to invent IDs, and in practice llama-3.3-70b on Groq follows that well, but "the model was instructed not to hallucinate" is a softer guarantee than "the app refuses to emit an ID the tools did not surface". If I wanted a hard guarantee, I would parse the assistant's cited IDs and cross-check them against the tool outputs before rendering. That is the honest gap between "cites case IDs" and "provably cannot fabricate a case ID".

Two smaller notes: inference runs on the Groq free tier, so rate limits and the occasional slowdown apply, and 429s are caught and surfaced as a friendly "the model is busy" message rather than an error. And the registry data is community-documented, so the answers are only as complete as the casebook behind them.

Closing

What I like about this build is how little glue it needed. An MCP server that already exposes the right tools, an AI SDK that streams typed tool parts, and a single Worker to host both halves. The result is a chat that does not just talk about agent failures but shows its work: every claim traceable to a case file you can open yourself.

Code is here: https://github.com/AgentPostmortem/casebook-chat

Cloudflare Unveils Kitesurf: AI‑Focused Browser Beats Chromium on Power Use

10x Magazine — Fri, 21 Aug 2026 06:24:11 +0000

TL;DR: Cloudflare has unveiled Kitesurf, a cloud‑hosted browser optimized for AI agents, delivering up to 30% lower compute usage than Chromium and streamlining the creation of autonomous web bots.

Imagine a web browser that never needs a human hand to click, scroll, or type—because it was engineered for machines instead of people. Cloudflare announced exactly that today with Kitesurf, a lightweight, cloud‑native browser designed to run AI‑driven agents at scale. By moving the rendering engine to the edge and stripping out UI layers meant for human interaction, Kitesurf promises faster, cheaper automation for developers building everything from price‑tracking bots to intelligent search assistants.

What Is Kitesurf and Why It Matters?

Kitesurf is not a traditional desktop or mobile browser; it lives entirely in Cloudflare’s global network and exposes a programmable API that AI models can call directly. Instead of loading a full Chromium stack with a graphical interface, Kitesurf runs a headless environment that still supports modern web standards, JavaScript execution, and DOM manipulation. The key difference is its focus on agent‑centric workloads: tasks like form‑filling, data extraction, and multi‑step navigation that AI systems perform repeatedly.

Developers can now spin up isolated browser instances with a single API request, attach their preferred language model, and let the agent interact with live sites without worrying about resource contention. Cloudflare markets Kitesurf as a plug‑and‑play solution that removes the need for custom Docker images, Selenium grids, or third‑party headless browsers, reducing operational overhead and security exposure.

Performance Edge Over Traditional Browsers

In internal benchmarks, Cloudflare says Kitesurf consumes roughly 20‑30% less CPU cycles than a comparable Chromium headless instance when executing common automation scripts. The savings stem from two design choices:

Edge‑Hosted Execution – By running the browser close to the target website, network latency drops dramatically, and data never traverses the public internet, which cuts both time and bandwidth.
Trimmed Rendering Pipeline – Kitesurf removes UI‑only components such as paint layers, font rasterization for display, and GPU acceleration that are unnecessary for machine‑only interactions. The result is a leaner engine that still respects the same JavaScript event loop and security sandbox.

For enterprises that run thousands of concurrent bots, those efficiency gains translate into measurable cost reductions on Cloudflare’s pay‑as‑you‑go pricing model. Early adopters report being able to double the number of active agents on the same budget, while also seeing a 15% drop in page‑load failures caused by rate‑limiting or CAPTCHAs, thanks to the platform’s built‑in DDoS mitigation.

Implications for AI Development and the Browser Landscape

Kitesurf arrives at a moment when generative AI is being woven into everyday workflows. Companies are experimenting with agents that can browse the web, summarize articles, or even negotiate contracts in real time. Until now, the bottleneck has often been the browser layer—developers must balance fidelity against cost, and many resort to brittle scraping techniques.

By offering a purpose‑built, cloud‑scalable browser, Cloudflare lowers the entry barrier for building reliable AI agents. The platform’s API also integrates with popular model hosting services, allowing developers to chain language model inference with live web interaction in a single pipeline. This could accelerate use cases such as autonomous market research, real‑time compliance monitoring, and personalized shopping assistants.

Moreover, Kitesurf signals a shift in how the industry views browsers: no longer just a human interface but also a programmable compute substrate. If the model gains traction, we may see a new ecosystem of AI‑first extensions, security policies, and pricing tiers that treat browser time as a first‑class cloud resource.

Takeaway: Cloudflare’s Kitesurf reimagines the web browser as a lean, cloud‑native engine for AI agents, delivering lower compute costs and higher scalability than traditional headless Chromium. For developers building autonomous web bots, the service could become a fast‑track to production‑grade performance without the overhead of managing complex infrastructure.

Top 5 Cloudflare AI Gateway Alternatives for Enterprises in 2026

Kuldeep Paul — Thu, 20 Aug 2026 18:25:29 +0000

An analysis of the leading Cloudflare AI Gateway alternatives in 2026, comparing Bifrost against other top enterprise tools on latency, security, and VPC controls.

Routing high-concurrency LLM traffic through a public, managed-only proxy can introduce unnecessary network latency, data perimeter risks, and unpredicted platform costs. While Cloudflare AI Gateway offers convenient edge-based caching, many enterprise teams require greater deployment flexibility and deeper, self-hosted governance. Bifrost, an open-source AI gateway written in Go by Maxim AI, is designed to solve these infrastructure challenges by running natively inside an organization's private network. This article evaluates the top five Cloudflare AI Gateway alternatives for enterprises in 2026, comparing their architecture, performance, and security controls.

Why Enterprises Seek Cloudflare AI Gateway Alternatives

Cloudflare AI Gateway is a SaaS, closed-source product. While its integration with Cloudflare's global edge CDN makes it appealing for small-scale projects, enterprise infrastructure teams often encounter roadblocks when scaling production workloads:

Lack of VPC and Air-Gapped Deployment: Cloudflare is a managed service. All prompts, system messages, and model completions must leave the enterprise network perimeter and traverse Cloudflare's public infrastructure. For companies operating under strict regulatory frameworks (such as HIPAA, GDPR, or SOC 2), this external data routing introduces critical compliance and security risks.
Network Hop Latency: Public proxies add a network transit hop between the application server, the gateway, and the LLM provider. In multi-step agentic systems where a single user interaction triggers consecutive model calls, this added edge latency degrades the user experience.
Granular, Hierarchical Budgeting: Cloudflare has introduced spend and rate limits, but it lacks the capability to delegate, partition, and track dynamic budgets across hundreds of internal teams, external clients, or individual developer keys from a single control plane.
Agentic Tool Integration (Model Context Protocol): Modern enterprise AI applications use the Model Context Protocol (MCP) to connect LLMs to local databases, file systems, and internal APIs. Cloudflare is focused primarily on traditional model APIs, leaving a security and governance gap for multi-agent tool execution.

The Top 5 Cloudflare AI Gateway Alternatives

The following five alternatives represent the leading enterprise-grade AI gateways in 2026, prioritized by performance, compliance, and deployment flexibility.

1. Bifrost: Best for Performance, VPC Isolation, and Agentic Workflows

When routing mission-critical workloads, Bifrost represents the standard for performance and network control. Bifrost, an Apache-2.0 licensed, Go-based gateway, functions as a high-performance proxy in front of 20+ model providers. It integrates seamlessly with existing codebases as a drop-in replacement for major SDKs, requiring developers to adjust only their base URL parameters.

# Example Bifrost configuration for multi-provider routing and fallbacks
providers:
  - name: openai-primary
    type: openai
    api_key: ${OPENAI_API_KEY}
    weight: 70
  - name: anthropic-backup
    type: anthropic
    api_key: ${ANTHROPIC_API_KEY}
    weight: 30

fallback_chain:
  - primary: openai-primary
    failover: anthropic-backup
    on_codes: [429, 500, 503]

Bifrost is engineered specifically for high throughput. It introduces just 11 microseconds of overhead per request at 5,000 requests per second under sustained load. These metrics are verified by public performance benchmarks, showing that it maintains sub-millisecond latency profiles where Python-based proxies run into execution bottlenecks.

To minimize model token spend, Bifrost implements semantic caching. This system matches incoming prompts against a vector store of historical requests, allowing the gateway to instantly replay cached completions for highly similar queries instead of routing them to expensive upstream models.

Furthermore, Bifrost utilizes virtual keys as its core cost-control abstraction. These virtual credentials enforce hierarchical rate limits and cost caps across individual consumers, teams, and departments. On the agentic front, its built-in Model Context Protocol (MCP) gateway supports a dynamic Code Mode, which lets LLMs orchestrate multiple tools locally, resulting in up to 50% fewer tokens and 40% lower execution latency.

Pros: Extremely low latency (11 microseconds under load); full in-VPC or air-gapped deployment; granular hierarchical budgets; native MCP client and server integration.
Cons: Requires operations capacity to self-host, although a managed enterprise option is available for teams wanting a zero-ops control plane.

2. LiteLLM: Best for Developer Compatibility and Fast Prototyping

For small-scale projects or fast prototyping, LiteLLM is a widely used open-source Python proxy. It normalizes input and output schemas across 140+ upstream model providers, allowing developers to switch between various models using a single OpenAI-compatible SDK format.

LiteLLM is packaged as a Docker container, making it easy to deploy on standard cloud compute instances. It includes database-backed rate limiting, basic team tracking dashboards, and native Slack alert integrations for key failures.

However, because LiteLLM is built on Python's async framework, it introduces significant runtime latency and CPU overhead under high concurrency. In sustained enterprise environments where hundreds of requests execute simultaneously, this processing overhead can cause latency to spike, making it less suitable for performance-critical agent chains.

Pros: Broad multi-model normalizations; easy to deploy via Docker; active open-source community.
Cons: Higher latency overhead compared to compiled Go gateways; advanced enterprise features (OIDC, audit logs) are locked behind its commercial tier; Python runtime introduces concurrency limitations.

3. Kong AI Gateway: Best for Teams Already Running the Kong Mesh

Enterprises that have already consolidated their web API traffic on the Kong API platform often use Kong AI Gateway to manage their LLM endpoints. Rather than deploying a separate proxy, this solution runs as a suite of AI-specific plugins layered onto Kong's legacy Nginx/Lua-based core.

Kong's architecture uses plugins to handle standard API tasks, such as token-based rate limiting, OAuth authentication, and route transformations. Its AI-specific plugins add prompt injection protection, PII redaction, and multi-provider load balancing.

While Kong is highly stable, if your organization does not already use the broader Kong ecosystem, deploying it purely for LLM traffic adds unnecessary configuration and infrastructure overhead. Furthermore, because it was designed for standard REST APIs, it lacks native AI-native tools like interactive prompt playgrounds, automated evaluation datasets, or Model Context Protocol orchestration.

Pros: Reuses existing web API security, routing, and access control patterns; extensive plugin catalog.
Cons: Heavy operational footprint for greenfield setups; Lua runtime configuration can be complex; lacks deep AI-native lifecycle features.

4. OpenRouter: Best for Zero-Ops Managed Multi-Model Access

For teams that prioritize ease of use and zero infrastructure management over data perimeter control, OpenRouter represents a compelling managed alternative. It acts as an API aggregator and model marketplace, providing a single endpoint for hundreds of commercial and open-weight models.

OpenRouter manages the upstream API keys and offers a unified billing system, letting developers test and switch between models (such as Claude, Llama, and Mistral) without maintaining accounts with individual model providers.

The main drawback for enterprise buyers is security and compliance. Since OpenRouter is a managed SaaS, every prompt is forwarded through their public edge nodes. This setup prevents self-hosted deployment inside an isolated VPC, which is a common requirement for enterprises handling proprietary source code or confidential client data.

Pros: Completely managed; unified billing; access to hundreds of open-source and closed-source models.
Cons: Data leaves the corporate boundary; added network hops increase total latency; no on-premise governance or local vector caching.

5. Azure API Management (GenAI Policies): Best for Azure-Native Architectures

Large enterprises whose infrastructure is entirely committed to the Microsoft cloud often route their AI traffic using Azure API Management. Microsoft has introduced specialized generative AI policies to help cloud architects manage, secure, and monitor their enterprise LLM endpoints.

These native policies handle token-based rate limiting (such as throttling based on Tokens Per Minute), circuit breaking across regional Azure OpenAI deployments, and secure credential storage within Azure Key Vault.

Azure's solution excels in high-security, Azure-only cloud architectures. However, routing traffic to non-Azure models (like Anthropic or locally hosted open-weight models) is highly manual, and the platform lacks the native semantic caching and developer-friendly playground tools found in dedicated AI gateways.

Pros: Native enterprise compliance inside Azure; robust token rate-limiting; seamless Entra ID and Key Vault integrations.
Cons: Highly complex multi-cloud configurations; heavily optimized for Azure OpenAI at the expense of other model providers; no native tool or agent orchestration capabilities.

Comparing Alternatives on Key Enterprise Dimensions

The table below outlines how the leading Cloudflare AI Gateway alternatives compare across the architectural criteria most critical to enterprise platform teams.

Dimension	Bifrost	LiteLLM	Kong AI Gateway	OpenRouter	Azure API Management
Latency Overhead	Sub-millisecond (11µs P99)	High (Python asyncio bottleneck)	Moderate (Lua/Nginx pipeline)	High (Public SaaS edge hops)	Moderate (Cloud gateway routing)
Deployment Model	VPC / On-Premise / Hybrid	VPC / On-Premise	Hybrid / On-Premise	Managed SaaS Only	Managed Cloud (Azure-bound)
Hierarchical Budgeting	Deep (Virtual Key, Team, Client)	Basic (API Key only)	Basic (Consumer limits)	Basic (Account limits)	Basic (Subscription limits)
MCP & Agent Support	Native (Client & Server Gateway)	None	Basic (Custom Lua plugins)	None	None
Cost Optimization	Semantic Caching & Code Mode	Basic Caching	Basic Caching Plugins	None	None

Enterprise AI Security: Extending Gateway Controls to the Endpoint

Deploying a server-side gateway is only the first phase of a comprehensive enterprise AI security strategy. While backend application traffic can be routed through a centralized corporate endpoint, employees routinely download local desktop applications, access web-based LLM chat portals, or configure terminal-based coding agents on their corporate laptops, creating a major shadow AI compliance blind spot.

Beyond centralized gateway routing, Bifrost applies robust server-side governance and security controls, such as virtual keys, budget limits, and immutable audit logs. To extend these exact policies to individual employee workstations, Bifrost Edge (currently in alpha) operates as an endpoint governance agent that runs natively on macOS, Windows, and Linux. This combined gateway and endpoint architecture automatically redirects all local developer AI traffic through the enterprise's central policy layer without requiring any manual reconfiguration of client applications.

Using the Bifrost Edge agent, security and platform teams can enforce app governance to permit only approved desktop tools, map and block unverified third-party scripts via MCP governance, and enforce data loss prevention (DLP) rules using native endpoint security patterns. This ensures that sensitive corporate data, raw intellectual property, and hardcoded database credentials never leave the laptop, maintaining a secure data perimeter across the entire enterprise fleet.

How to Choose the Right AI Gateway for Your Enterprise

Selecting the ideal gateway involves evaluating your performance requirements, existing architecture, and compliance boundaries:

Choose Bifrost if your applications require sub-millisecond routing, deployment inside a private VPC or air-gapped network, semantic caching to lower token costs, or advanced Model Context Protocol (MCP) tool governance across both backend servers and employee endpoints.
Choose LiteLLM if your team is standardizing on Python-based infrastructure, is running low-concurrency prototype workloads, and does not require enterprise-grade low-latency performance under load.
Choose Kong AI Gateway if your IT infrastructure is already operating on the Kong API mesh, allowing you to easily reuse existing API management, monitoring, and authentication plugins.
Choose Azure API Management if your application architecture is fully committed to the Microsoft Azure tenant and your LLM usage is exclusively focused on Azure OpenAI instances.
Choose OpenRouter if you are building an early-stage prototype, need immediate access to hundreds of public models, and do not need to enforce strict on-premise data governance or network isolation.

Engineering and platform teams evaluating Cloudflare AI Gateway alternatives can explore the Bifrost GitHub repository to inspect the open-source code, or request a Bifrost demo to see how native endpoint AI governance and high-performance routing operate at enterprise scale.

Sources

I Built a Free AI Image Prompt Gallery + Browser-Based Image Toolkit — Here's What I Learned

Wenjie Zhang — Thu, 20 Aug 2026 00:19:34 +0000

If you've ever spent twenty minutes hunting for the right prompt to get Nano Banana or GPT Image to output exactly the poster style you had in mind, this post — and the tool I built — is for you.

I recently launched HereImg, a site that combines two things I personally kept needing and never found bundled together:

A searchable gallery of AI image prompts (organized by model and category, not just a random Twitter thread)
A set of fast, free, browser-based image editing tools for the boring-but-essential stuff — resize, crop, compress, remove background, etc.

Here's why I built it, what's under the hood, and how it might save you some time.

The problem: prompt engineering for images is still trial-and-error

Text-to-image models like Nano Banana, GPT Image, and Seedream are incredible, but getting consistent, high-quality output still comes down to prompt phrasing. Most people either:

Scroll through scattered social media posts hoping to find a prompt that matches their use case, or
Burn through credits re-rolling generations with slightly different wording

I wanted a place where prompts are organized like a real reference library — by model (Nano Banana / GPT Image / Seedream) and by category (Food & Drink, Illustration & 3D, Photography, Poster Design, Product & Brand, UI & Graphic) — so you can browse examples close to what you're trying to make, copy a working prompt, and tweak it instead of starting from zero.

Every prompt in the gallery is paired with its actual generated output, so you're not guessing what a phrase like "cinematic rim lighting, 35mm" will actually produce.

The other half: tools you need before and after generating

AI generation is rarely the last step. You still need to resize the output for Instagram, crop it for a thumbnail, strip the background for a product shot, or compress it before uploading. So I bundled a Magic Tools suite alongside the prompt gallery:

Social Media Resizer (pre-set dimensions for major platforms)
Image Cropper / Resizer / Rotator & Flip
Compressor & Format Converter (JPG/PNG/WebP)
Brightness & Contrast, Filters, Grayscale
Color Picker (extract hex codes / palettes)
Watermark tool
AI Background Remover

The key design decision: all the "free tools" run entirely client-side in the browser. Nothing gets uploaded to a server. That means no signup, no file size anxiety, no waiting on an upload queue — and your images never leave your device. Only the AI-powered features (like background removal) touch a backend, and those are clearly marked with a credit cost.

Under the hood

For fellow builders curious about the stack: HereImg runs on Cloudflare Workers, which has been a great fit for this kind of product —

Edge deployment means the tool feels instant no matter where in the world someone opens it
Cheap to run at scale, which matters a lot as an indie/bootstrapped project
Pairs naturally with client-side image processing, since the heavy lifting (canvas/WASM operations) happens in the visitor's browser rather than on my infra

If there's interest, I can write a follow-up post specifically on the Workers + D1 setup, the KV caching strategy for the prompt gallery, and some of the deployment gotchas I hit along the way (happy to share the pain so you don't repeat it).

Try it out

🖼️ Browse the prompt gallery
🛠️ Magic Tools suite
🎁 New accounts get 20 free credits to try AI generation/background removal

It's free to use, no login required for the browser-based tools, and I'd genuinely love feedback from this community — especially if you spot a prompt category I'm missing or a tool you wish existed.

If you build something with it (or find a prompt that works great), drop it in the comments — I'll add good ones to the gallery.

Cloudflare OS Launches Open Platform for AI Agents

Felipe L — Wed, 19 Aug 2026 13:27:30 +0000

What Happened

Cloudflare released Cloudflare OS, an open platform for building AI agents, applications, and automated workflows. The OS bundles services, APIs, and tooling that let developers launch agents anywhere from the edge to the cloud, with built‑in scaling, security, and observability.

Cloudflare OS delivers a unified runtime, a marketplace for reusable components, and integration hooks that connect agents to external services, databases, and other agents. The code and SDKs are open source on GitHub, and the platform works with existing workflow engines like n8n, Zapier, and custom orchestration frameworks.

The launch aligns with the industry’s push toward edge‑first AI, where latency and privacy matter. By providing a managed runtime that runs near users, Cloudflare OS cuts round‑trip times for agent‑driven tasks while keeping data on the edge whenever possible.

Why This Matters for Builders

Scalable Edge Runtime: Deploy n8n workflows or custom AI agents directly to Cloudflare’s edge network. Reduce latency for time‑sensitive tasks such as real‑time data enrichment or user‑specific personalization.
Unified API Surface: A consistent set of APIs for authentication, storage, and messaging cuts the friction of integrating multiple third‑party services. Teams can focus on business logic instead of plumbing.
Marketplace for Components: A curated library of pre‑built agent modules—NLP, data transformation, and more—can be plugged into existing workflows. This speeds prototyping and lowers the barrier to adding AI capabilities.
Observability & Security: Built‑in logging, tracing, and fine‑grained access controls let production‑grade workflows be monitored and audited out of the box. This is a major advantage in compliance‑heavy environments compared to generic cloud VMs.
Open‑Source Flexibility: The core is open source. Teams can fork, extend, or embed the runtime into their own infrastructure for hybrid or on‑prem deployments.

A team that already uses n8n can deploy a new agent on Cloudflare OS for low‑latency transformations, import a community‑built NLP component from the marketplace, chain it with existing tasks, and rely on Cloudflare’s observability stack to monitor performance.

FAQ

Q: Can I run my existing n8n workflows on Cloudflare OS without rewriting them?

A: Yes. Cloudflare OS includes a runtime wrapper that accepts n8n’s JSON workflow definitions and executes them in the edge environment. Minimal configuration is needed to connect to your data sources.

Q: Does Cloudflare OS support custom AI models, or only pre‑built ones?

A: You can host your own model containers or call external model endpoints. The marketplace offers pre‑built models for common tasks, but you can also deploy your own models to the edge or cloud as needed.

Q: What about cost and scaling?

A: Cloudflare’s edge network charges per request and per GB of data processed. For high‑volume, low‑latency workloads, this can be cheaper than running dedicated VMs. The OS automatically scales agent instances based on traffic, so you pay only for what you use.

Originally published on Automations Cookbook.

Traffic Anomaly or Outage? What a 30% Drop Actually Means

Umesh Malik — Wed, 19 Aug 2026 13:05:31 +0000

TL;DR

Traffic anomaly or outage? The question every on-call engineer dreads has a mechanical answer: compare the drop against a baseline matched by time-of-day and day-of-week, then check whether the deviation's timing lines up with an independent, verifiable cause. Cloudflare answered it in minutes for the August 12, 2026 solar eclipse — traffic fell 15-30% along the path of totality, and as much as -46.7% in the hardest-hit countries — by comparing five-minute traffic buckets against the median of the same slot on the three preceding Wednesdays, then confirming the drop's shape matched the eclipse's geometrically computed timing.

What actually happened during the eclipse

On August 12, 2026, a total solar eclipse crossed a path starting near Alaska around 15:35 UTC and sweeping through the North Atlantic into Europe. Cloudflare's Radar team documented the traffic impact as the moon's shadow moved: "regions along the path of totality saw traffic fall by roughly 15% to 30%" at the moment of maximum obscuration.

Individual countries moved even more — the full range across affected regions was "9.3 to -46.7%" — with Iceland, Spain, and Portugal registering the steepest declines, alongside smaller drops in Ireland, the UK, and France. Countries with only shallow partial coverage, like Sweden, Denmark, Poland, and Switzerland, showed close to nothing. That's a bigger swing than most teams would tolerate silently: a 30-46% drop in request volume, in a five-to-twenty-minute window, is the kind of number that pages an on-call engineer at 3 a.m.

The interesting part isn't that traffic dropped — outdoor daylight events pulling people off their phones is intuitive. The interesting part is how Cloudflare proved it was the eclipse and not a routing problem, a DNS issue, or a partial regional outage, all of which produce the exact same symptom: a sudden request-volume cliff.

Traffic anomaly or outage: the method that actually decides it

Cloudflare's stated methodology was specific: "We compared eclipse day against the same weekday: the median of the three previous Wednesdays, matched slot-by-slot on time-of-day." Traffic was bucketed into five-minute windows, and each window on eclipse day was checked against the median of that identical five-minute window on the prior three Wednesdays — not against yesterday, not against a rolling daily average, and not against a single reference day.

Then came the step that actually establishes causation rather than mere correlation: they checked the timing. The obscuration percentage at each location was computed geometrically from real sun and moon positions, and the traffic dip's shape — onset, trough, and recovery — was checked against that computed obscuration curve at each location.

Cloudflare's own framing: "Paired with the precise timing of the drops, the trend of the data demonstrates that the eclipse itself was the primary driver of the decline." A drop that lines up minute-for-minute with an independently computable physical event is not the same kind of evidence as a drop that merely happened during the day.

Why naive baselines lie

Most homegrown anomaly detectors compare against something too crude to survive contact with real-world variance:

Naive baseline	What breaks it
"Compare to yesterday"	Day-of-week effects (Wednesday ≠ Tuesday) dwarf small real signals
"Compare to a flat daily average"	Erases the intraday shape — traffic isn't flat across 24 hours, so every trough looks anomalous
"Compare to last week, same day"	One sample is still noisy; a single unusual prior week poisons every comparison after it
"Alert on any % deviation"	No mechanism to separate a real external cause from ordinary variance, so every threshold is a guess

Cloudflare's baseline avoids all four failure modes at once. Matching by time-of-day removes the intraday shape. Matching by weekday removes the day-of-week effect. Taking the median of three weeks, not one, removes single-week noise without being distorted by one outlier week the way a mean would be.

Building this yourself: a baseline that doesn't cry wolf

You don't need Cloudflare's edge network to apply the same logic to your own traffic, request-rate, or error-rate monitoring:

Bucket at the resolution of the event you care about. Five minutes is a solid default for web traffic — coarse enough to smooth per-request noise, fine enough to catch anything that resolves within 15-20 minutes. If your real events are shorter, tighten the bucket.
Match by time-of-day and day-of-week, not just by date. A Wednesday-at-3pm bucket should be compared to prior Wednesdays at 3pm, never to Tuesday at 3pm or to Wednesday at 9am.
Take the median across 3+ prior matched windows, not the mean of one or two. The median is robust to a single unusual prior week (an outage, a holiday, a promo) in a way a two-sample average never is.
Require a second, independent signal before calling it a real anomaly. Cloudflare had the eclipse's geometric timing. You might have a deploy timestamp, a known maintenance window, a marketing campaign's send time, or a correlated metric (CPU, upstream latency) moving in the same shape. Deviation alone is a symptom; deviation plus a matching independent cause is a diagnosis.
Only page on deviation without an independent cause. If the drop matches nothing you can explain, that's exactly the case worth waking someone up for — you've now filtered out the explainable noise, so what's left is genuinely worth investigating.

Common mistakes

The single biggest mistake is treating "deviation from baseline" and "root cause identified" as the same finding. They aren't. A baseline tells you that something changed; it never tells you why. Teams that stop at step 1-3 above end up with dashboards that are technically correct and practically useless — every real anomaly looks identical to a genuine incident until someone manually checks for an external cause, which is exactly the step Cloudflare didn't skip.

The second mistake is picking a bucket size and baseline window once and never revisiting them as traffic patterns shift — a baseline tuned for pre-pandemic office-hours traffic, for instance, will misfire constantly against post-2020 traffic shapes. Re-validate your baseline's assumptions whenever you meaningfully change what "normal" looks like for your service.

The takeaway

A 15-30% traffic drop is not, by itself, evidence of anything. It's the shape of the drop against a properly matched baseline, plus an independent explanation for its exact timing, that turns a scary number into either "expected, stand down" or "genuinely worth paging someone." Cloudflare had a solar eclipse as its independent signal. Most teams have deploy logs, feature-flag timestamps, and marketing calendars sitting right there, unused, the next time a dashboard spikes red.

If you're building alerting on top of Cloudflare Workers, see how Cloudflare Access and removing unnecessary beacon scripts affect what you're even measuring. For the discipline of tracing a symptom back to a verified root cause, the same rigor shows up in debugging an EEVDF latency regression and in running a zero-downtime dual-write migration — both are exercises in not trusting the first plausible explanation. And if your traffic anomaly turns out to be automated rather than seasonal, verifying crawler IPs instead of trusting User-Agent strings is the next place to look.

FAQ

How do you tell a real traffic anomaly from an outage? You compare the drop against a baseline built from the same time-of-day and day-of-week, not against yesterday or a flat average. If the deviation lines up precisely with a known external cause — a scheduled event, a regional pattern, a correlated signal — and recovers on the same timeline that cause predicts, it's an anomaly, not an outage. An outage has no such external correlate and typically doesn't self-correct on a predictable schedule.

Why is comparing traffic to yesterday a bad baseline? Day-of-week effects are large enough to swamp small real signals — enterprise traffic on a Wednesday can differ from a Tuesday by more than the anomaly you're trying to detect. A single prior day is also one noisy sample, so any randomness in that one day becomes randomness in your alert threshold. A multi-week, same-weekday, same-time-slot median averages that noise out before you ever compare it to today.

What is Cloudflare Radar and what did it measure for the 2026 eclipse? Cloudflare Radar is Cloudflare's public internet-traffic-and-trends dashboard, drawing on request data from its global edge network. For the August 12, 2026 total solar eclipse, Radar's team compared HTTP request volume in five-minute buckets against a baseline built from the three preceding Wednesdays at matching times of day, and found traffic changes ranging from +9.3% to -46.7% across affected countries, with the deepest drops in Iceland, Spain, and Portugal.

What bucket size should I use for anomaly detection? Five minutes is a reasonable default for web traffic: coarse enough to smooth out per-request jitter, fine enough to catch a drop that starts and ends within 15-20 minutes. Match your bucket size to the shortest real event you care about — if you need to catch something that resolves in under five minutes, bucket at one or two minutes instead.

Does differential privacy or noise correction matter for this kind of baseline? Not for this specific case — aggregate HTTP request counts across a whole country aren't sensitive per-user data, so Cloudflare's eclipse analysis didn't need privacy-preserving noise. That technique matters for a different problem: when the metric itself could reveal something about an individual, which is a concern in on-device telemetry pipelines, not in coarse regional traffic monitoring.

Sources

Cloudflare, "Total eclipse of the Internet: traffic impacts in Iceland, Spain, and Portugal" — the traffic percentages, methodology, and timing correlation cited throughout.
Cloudflare Radar — the public dashboard the analysis draws from, for readers who want to explore live global traffic trends.

Originally published at umesh-malik.com

Keep reading on umesh-malik.com:

DEV Community: cloudflare

AI crawler user agents are self-reported: 468 real fetches, 991 fake ones

A User-Agent is a claim, not evidence

The query (works on a free zone)

One UA string, two rows

One of the agents cannot exist

Classify the path, not just the client

The result: 468 content fetches against 991 scans

I Built a Crypto Market Intelligence App with React Native, Supabase & Cloudflare Workers AI

The problem: alert fatigue

The stack

Architecture

The feature I'm most proud of: the AI semantic scanner

⚠️ The ES256 vs RS256 gotcha

Lessons learned along the way

1. External sites will block your cloud IPs

2. Data freshness is a feature

3. Alert fatigue is the real enemy

4. Keep the app architecture boring

The result

Cloudflare Bot Preference Sync will write your robots.txt from 21 August 2026 - and it ignores your custom rules

Cloudflare Bot Preference Sync will write your robots.txt from 21 August 2026 - and it ignores your custom rules

What Cloudflare actually announced

The part the announcement does not lead with

The docs have not caught up

Why the 15 September date is the real deadline

How to tell whether this is you

India-specific considerations

What is still unknown

FAQ

How eCorpIT can help

References

Alibaba Tumbles on Earnings Miss; Tesla Robotaxi Approval and Cloudflare Agent Surge Boost AI Stocks

Alibaba Tumbles on Earnings Miss; Tesla Robotaxi Approval and Cloudflare Agent Surge Boost AI Stocks — alibaba tumbles earnings miss tesla robotaxi

Alibaba's Earnings Setback and AI Spending Concerns

Tesla Secures Key Robotaxi Approval

Cloudflare Dominates AI-Agent Network Traffic

Broader Market Movements and Key AI Stocks

Conclusion: Navigating the AI Investment Landscape

Your Worker Returned 500 and the Log Says `outcome: "ok"`

The part that works

Trap 1: a rendered 500 is a successful invocation

Trap 2: a wide time window silently undercounts

Trap 3: exists matches empty strings, and includes ignores case

Trap 4: event counts are not invocation counts

What I actually run now

The one-line version

Running a Monitoring SaaS on Cloudflare Workers + Supabase for Almost Nothing

The shape of the system

The revenue-gated cron throttle

SSRF-guarding the public probes

Watching the watcher, from a different cloud

Boring reliability plumbing that earns its keep

What it costs

If you're building on this stack

Building a streaming investigation assistant that tool-calls an MCP server and cites case IDs

The core idea

How it works

Streaming with tool calls rendered inline

Citations that mean something

One honest limitation

Closing

Cloudflare Unveils Kitesurf: AI‑Focused Browser Beats Chromium on Power Use

What Is Kitesurf and Why It Matters?

Performance Edge Over Traditional Browsers

Implications for AI Development and the Browser Landscape

Top 5 Cloudflare AI Gateway Alternatives for Enterprises in 2026

Why Enterprises Seek Cloudflare AI Gateway Alternatives

The Top 5 Cloudflare AI Gateway Alternatives

1. Bifrost: Best for Performance, VPC Isolation, and Agentic Workflows

2. LiteLLM: Best for Developer Compatibility and Fast Prototyping

3. Kong AI Gateway: Best for Teams Already Running the Kong Mesh

4. OpenRouter: Best for Zero-Ops Managed Multi-Model Access

5. Azure API Management (GenAI Policies): Best for Azure-Native Architectures

Comparing Alternatives on Key Enterprise Dimensions

Enterprise AI Security: Extending Gateway Controls to the Endpoint

How to Choose the Right AI Gateway for Your Enterprise

Sources

I Built a Free AI Image Prompt Gallery + Browser-Based Image Toolkit — Here's What I Learned

The problem: prompt engineering for images is still trial-and-error

Trap 3: `exists` matches empty strings, and `includes` ignores case