DEV Community: Jangwook Kim

Official GEO is a subtraction list plus one Search Console switch

Jangwook Kim — Fri, 14 Aug 2026 06:44:40 +0000

The official GEO document is not a shopping list. It is a subtraction list, plus one Search Console switch.

Google Search Central published Optimizing your website for generative AI features on 15 May 2026 and touched it again on 10 July. Third-party AEO/GEO checklists still open with llms.txt, then chunking, then a special schema.org type, then a rewrite "for the model." The official mythbusting section deletes that column first.

I did not log into Search Console for this piece. I did fetch the public bytes this site already serves. Ranking is not in scope.

What AI Overviews actually pick up

Two names, one index. AI Overviews attach a short gist and supporting links to a hard question. AI Mode is the conversational surface for comparisons and multi-step reasoning. Both sit inside Google Search. Both pull live pages. Google describes the machinery as RAG on top of the core ranking systems, plus query fan-out: a lawn-weeds question can spawn separate retrievals for herbicides, chemical-free removal, and prevention.

Eligibility is older than the names. A page has to be indexed and allowed to show a snippet. AI features and your website puts it this way:

There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.

Source: AI features and your website

No extra technical bar. No special optimization. The same page still wants the technical requirements, the spam policies, and people-first content. Meeting all of that still does not guarantee a crawl, an index, or a serving slot. That hedge is the same hedge Search has always used.

Snippet eligibility is a lever I already measured. nosnippet and max-snippet:0 close the page as direct input to AI Overviews and AI Mode. The write-up lives in robots snippet directives. I am not rerunning that parser today. A layer landed above it.

The optimization guide adds this: besides the usual technical requirements, a site must be included in Search generative AI features in Search Console before it is eligible to appear there. That is a property setting, not a template ticket.

Four backlog items the official guide deletes

The mythbusting section treats "AEO" and "GEO" as nicknames for optimizing the search experience. The point fits in one sentence.

From Google Search's perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.

Source: Optimizing your website for generative AI features

What drops out of an engineering backlog:

Item on third-party lists	Official stance	What to do in code
`llms.txt` and other "AI files"	Google Search does not use them. Creating them neither helps nor hurts visibility or rankings	Do not add one for Google Search. Keep it only if some other system actually reads it
Chunking copy for the model	Not required. The systems can follow more than one topic on a page	Size the page for readers
Rewriting only for AI	Synonyms and meaning are already understood. You do not need a page per long-tail variant	Keep the human draft. Farming variant pages collides with scaled content
A special schema.org type for generative search	Not required. There is no special markup to add	Leave rich-result markup in place. Do not invent an "AI Overview schema"

The llms.txt line is blunt.

Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.

Source: the LLMS.txt item in the same optimization guide

Harmless and useless, for Google Search. Fine to keep for some other consumer. Off the Google Search board. If robots.txt already splits training bots from search bots, today's job is not another file. It is dropping the idea that Google Search reads this one.

Read the structured-data paragraph twice. It is not required for generative search and there is no dedicated type. It still helps rich-result eligibility, so keep it in the wider SEO plan. I do not read that as "delete JSON-LD." I read it as "do not add a type because AI Overviews exist." Structured data has never guaranteed rankings. It does not guarantee citations either.

Inauthentic mentions around the web are, officially, not as helpful as they sound. Core ranking looks at quality. Other systems block spam. Generative features depend on both. That is not a sprint ticket.

Third-party tools that claim "internal" Google metrics get the same haircut. No third-party tool has access to internal ranking or AI systems. The third-party SEO guidance says to check AEO/GEO advice against official docs. Using a tool in a workflow is fine. Treating its number as Google's number is not.

The Search Console switch, and who inherits it

"Included in Search Console" points at the Search generative AI control. Path: Settings > Search generative AI.

Three states. Include the site's links and content in generative AI features. Exclude them. Inherit from the parent. Include is the default for every property. Exclude removes the site from AI Overviews, AI Mode, and generative features in Discover: no links, no grounding input, no impressions or traffic from those features.

The limit, in the help page's own words:

This control only affects whether your content can appear in certain Search generative AI features; this control isn't used as a ranking or inclusion signal affecting other parts of Search.

Source: Search generative AI control

Not a ranking signal for the rest of Search. Not a training switch either. Training limits sit on Google-Extended. Full removal from Search is noindex. After the control goes live, exclusion usually lands in 1〜2 days. Cache can stretch that.

This is an ownership problem. Flip exclude on the domain property and every child URL-prefix property that still inherits follows it. A blog living at https://example.com/blog/ can fall out of generative features because someone touched the parent. Clean HTML and a careful robots.txt do not reopen that layer.

The control and the report are still on a subset of sites. The 3 June 2026 announcement of the Generative AI performance report is explicit about that. The report counts impressions, not clicks or positions. Search Labs experiments are out. A missing screen does not mean the site is excluded. It may not be in the rollout, or it may not have enough generative impressions yet.

I did not open Search Console for this article. I will not claim the menu exists on this property. What I can claim is the documented default (include) and the documented inheritance.

For a team, invert the order of work. Read the parent property first. Write down whether each child inherits or overrides. Only then audit robots meta and the live robots.txt. Reverse that and you can spend a week on markup while the site is simply not included. The help page already says 1〜2 days, so an unchanged report the same afternoon is not a reason to revert. If the people who ship HTML and the people who own Search Console are different groups, this layer never shows up in a pull request.

The live robots.txt was not the file in git

After the official subtraction list, I did not add a file. I fetched what a crawler is offered. 14 August 2026, public URLs on https://jangwook.net only.

curl -sI https://jangwook.net/llms.txt
# HTTP/2 404

curl -sL https://jangwook.net/robots.txt | wc -l
# 106

# public/robots.txt in git: 45 lines, 1,101 bytes
# live response: 106 lines, 2,937 bytes

llms.txt is 404. So are LLMs.txt and the www host. That matches a site that did not plant a file Google Search says it ignores.

robots.txt did not match git. The repo file is 45 lines: training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) are disallowed, search bots and * only lose the cross-language URLs. The live response prepends a CDN-managed block and becomes 106 lines. User-agent: * carries Content-Signal: search=yes,ai-train=no,use=reference. Several training and extended bots get another Disallow: /. The repo body is still there, after the prefix.

I did not find Content-Signal listed as a supported rule in the Search Central robots.txt material I used for this piece. Presence in the live file and consumption by Googlebot are different claims. I am not making the second one. Treat the token as a third-party convention. Do not hang Google Search eligibility on it.

Eight pages (/, /ko/, /en/, /ko/blog/, three posts, /ko/contact/) all returned HTTP 200. Zero <meta name="robots"> tags. Zero data-nosnippet attributes. One post's description mentions the word nosnippet. That is copy, not a directive. The template only emits a robots tag when noindex is on.

JSON-LD on the home page included Organization, Person, and WebSite. Posts carried BlogPosting, WebPage, and BreadcrumbList. Under today's official text, that markup is not a ticket into generative search. It stays on the rich-result and entity side, the same split I used when FAQ rich results ended and the Q&A markup stayed. Validation and visibility were never the same event.

The live/git split is the measurement. If GEO work means "add a file to the repo," the file a crawler already reads has grown outside the repo. Diff the deployed URL, not public/robots.txt against itself.

Agents also read the accessibility tree

The guide's last stretch is browser agents: booking, comparing specs. Different job from search citations. Same surface. web.dev's agent-friendly note lists three views: screenshots, raw HTML, the accessibility tree.

The accessibility tree keeps roles, names, and states, and drops visual noise. It is the same tree a screen reader uses. Style a div as a button and the DOM-only path never sees a button. The screenshot path may see where it sits and still miss what it does. Semantic HTML and label for are not a score chase. They are how a machine avoids guessing the wrong action.

The optimization guide talks about semantic HTML as readability and assistive parsing, not "perfect code." The web is not valid HTML. Google can still read it. The reason to use semantic elements anyway is not only Googlebot. Agents walk the same tree.

The leftover work is dull. Prefer button and a. Wire labels with for. Do not hide hit targets under a transparent overlay. Do not let the layout jump by category. That is WCAG work. It is not a new agent file format.

web.dev also says a screenshot-only path is slow and expensive, a backup when structure is muddy. Put the main path on the tree and the DOM and the text a crawler reads comes from the same markup an agent uses. That is the opposite of stuffing hidden search copy.

What leaves the sprint, what stays

The official GEO page is not asking for more surface area. It deletes the market list, then keeps the technical Search surface and one Search Console switch.

Leave the sprint:

A new llms.txt for Google Search, a special AI markdown file, a generative-search-only schema.org type
Chunking and AI-only rewrites, and a page per query variant
Third-party "internal" metrics as a deploy gate

Keep:

Indexing and snippet eligibility. No stray nosnippet in the template. noindex only where you meant it
Fetch the live robots.txt and diff it against git. Confirm a CDN prefix did not invert training bots and search bots
The generative AI control on parent and child properties. Default is include. A missing UI can mean "not in the rollout," not "excluded"
Semantic HTML and the accessibility tree. Agent guidance is existing markup
Rewrite the purpose line on rich-result JSON-LD from "required for generative search" to "rich-result eligibility"

The smallest commands:

curl -sI https://example.com/llms.txt | head -n 1

curl -sL https://example.com/robots.txt > /tmp/live-robots.txt
diff -u public/robots.txt /tmp/live-robots.txt

python3 - <<'PY'
import re, sys, urllib.request
html = urllib.request.urlopen(sys.argv[1]).read().decode("utf-8", "ignore")
print("robots meta:", re.findall(r"<meta[^>]+name=[\"']robots[\"'][^>]*>", html, re.I))
print("data-nosnippet attrs:", len(re.findall(r"<[^>]+data-nosnippet", html, re.I)))
PY
https://example.com/your-page/

Those three lines still cannot see the Search Console switch. That layer is a property setting. A pull request will not catch it.

Nothing here promises an AI Overview. Include can stay on, snippets can stay allowed, the page can be indexed, and Google still does not owe a citation. I measured the layers of eligibility, not the size of an effect.

If you line up the repo, the live URL, and the property setting and cannot see which layer is open, bring that layer. Matching official text to the bytes you actually ship is the work I do.

Sources: Google Search Central generative AI optimization guide (updated 2026-07-10), AI features and your website, third-party SEO guidance, Generative AI performance reports announcement (2026-06-03), Search Console Help Search generative AI control and Generative AI performance report, web.dev Build agent-friendly websites (all official). The four English block quotes were fetched from those pages, whitespace-folded, and checked against the source; each quote sits next to its URL. Live fetch: 2026-08-14, robots.txt, llms.txt, and eight pages on https://jangwook.net, curl plus HTML parse. Data: data/official-geo-gsc-control-probe-2026.json. Figures: scripts/chart-official-geo-gsc-control.py. Search Console was not opened. Content-Signal appears in the live robots.txt and was not confirmed as a supported Search Central robots.txt rule. Structured data and this switch do not guarantee rankings.

The gpt-5 Alias Points at a Model OpenAI Deletes Dec 11

Jangwook Kim — Fri, 14 Aug 2026 00:45:11 +0000

A support-ticket summarizer that has run quietly for a year stops returning anything on a Friday in December. Nobody deployed. Nobody touched a config file. The line in the code still says model: "gpt-5", the same three words it has said since launch, and that line now points at something OpenAI has switched off.

The date is already on a calendar. OpenAI has two model shutdowns inside the next four months, and the second one is the expensive one for most teams. There's also a comfortable assumption about who it hits, and the assumption is wrong.

Who this actually breaks

OpenAI's deprecation page lists two waves. The first, announced April 22 and landing October 23, 2026, clears out the old furniture: gpt-3.5-turbo-0125, gpt-4-0613, gpt-4-turbo, gpt-4o-2024-05-13, o1-2024-12-17, o3-mini-2025-01-31, and their fine-tuned variants. The second, announced June 11 and landing December 11, 2026, removes what most production systems are running today: gpt-5-2025-08-07, gpt-5-mini-2025-08-07, gpt-5-nano-2025-08-07, gpt-5-pro-2025-10-06, o3-2025-04-16, and o3-pro-2025-06-10.

Read that list and an obvious reading suggests itself. Every ID on it carries a date, so the teams at risk must be the ones who pinned a dated snapshot for reproducibility. Everyone who typed the plain short name is presumably riding some maintained rolling release and can ignore the whole thing.

We checked that against the API instead of assuming it. It doesn't hold.

What we ran

Effloow Lab ran an OpenAI API check on August 14, 2026 against one standard company API account. Two things were tested. First, we asked the API to list every model the account can see and compared that list against both shutdown waves. Second, and this is the part that produced the finding, we sent a trivial request (the literal prompt "Reply with the single word: ok") to each undated model name, then read back which model the API said had answered.

Every response carries a model field. It reports the specific dated version that served the request, whatever you typed. It's the API telling you in writing what your traffic is really hitting.

Commands, raw response data, and everything that failed are in the public lab note.

What came back

What the code says	What the API says answered	Switched off
`gpt-5`	`gpt-5-2025-08-07`	Dec 11, 2026
`gpt-5-mini`	`gpt-5-mini-2025-08-07`	Dec 11, 2026
`gpt-5-nano`	`gpt-5-nano-2025-08-07`	Dec 11, 2026
`gpt-5-pro`	`gpt-5-pro-2025-10-06`	Dec 11, 2026
`o3`	`o3-2025-04-16`	Dec 11, 2026
`gpt-5.6-sol`	`gpt-5.6-sol`	not scheduled

Five out of five. On this account, on this date, every plain undated name in the gpt-5 and o3 families resolved to precisely the version on the December kill list.

So the comfortable reading collapses. Writing gpt-5 doesn't put you on a maintained rolling release. It puts you on gpt-5-2025-08-07 with a friendlier label. The team that pinned a date at least left a string in the repo that a search will find. The team that used the short name has nothing to grep for and the same deadline.

There's a second, quieter result. The account's model list returned 126 model IDs, and two officially deprecated IDs (gpt-4-1106-preview and o3-pro-2025-06-10) were already missing from it. A published shutdown date is when access is guaranteed to end, not when it starts to end. Some things go early.

The replacement that isn't a replacement

Two rows of OpenAI's migration table deserve separate attention, because a find-and-replace across your codebase will get them wrong.

Start with the shape of the mapping. It isn't one-to-one. Per the deprecation page, gpt-5-mini-2025-08-07 moves to gpt-5.6-terra and gpt-5-nano-2025-08-07 moves to gpt-5.6-luna, while gpt-5-2025-08-07 and o3-2025-04-16 both go to gpt-5.6-sol. Swapping every old ID for one new ID quietly changes the size and price class of half your traffic.

The pro tier is worse. Both gpt-5-pro-2025-10-06 and o3-pro-2025-06-10 are documented as replaced by gpt-5.6-sol with reasoning.mode set to pro. The "pro" capability stopped being a separate product and became a setting. Change the model name alone and you land on a cheaper, different configuration than the one you were paying for. Nothing errors to tell you.

We measured what switching that setting on costs. Same model, same prompt, only the reasoning object changed:

Prompt sent	Setting	Input tokens billed
A 13-token instruction	standard (`effort: low`)	13
The same 13-token instruction	`mode: pro`	1,530
A 167-token instruction	standard (`effort: low`)	167
The same 167-token instruction	`mode: pro`	2,145

In plain terms: pro mode attached roughly fifteen hundred to two thousand extra tokens of input to a request that was one short sentence (+1,517 and +1,978 across the two pairs). On the short prompt that's about 117 times the input you thought you were sending. At volume, that difference is your bill.

Treat it as a budgeting fact rather than a verdict. Two request pairs is not a benchmark, and we make no claim about whether the output is any better. The point is narrower. The parameter isn't free, and a migration that flips it on everywhere without measuring will show up in the next invoice.

Can this survive your workflow?

Ask something narrower than "do we use OpenAI." Ask where a silent model swap or a hard stop would actually hurt:

Support ticket triage and summarization. Built early, rarely touched since, usually on the plain gpt-5-mini name. Highest chance of an unfindable string.
Order and invoice processing. A hard stop here means a queue backing up rather than a slightly worse answer. Loud, expensive, and it'll happen on a weekend.
CRM enrichment and internal automation. Often written by whoever needed it, deployed once, owned by nobody in particular. No one is reading OpenAI's deprecation page on their behalf.
Anything with a fine-tuned model. Fine-tunes on a retiring base have no string-swap path at all. That's a retraining project with a lead time, and October 23 is the date on it.
Anything using pro-tier reasoning. Both routes into it collapse into one parameter. Cost and behaviour both move.

If one of those is yours and has no clear owner, look at it this month rather than in November.

Want someone to run this audit against your repositories and hand you the list? Effloow's Proof Studio does exactly this kind of claim-bound technical check, and our services page covers the write-up and migration documentation side.

When to act, and when to skip

Act now if: you have production traffic on any gpt-5, gpt-5-mini, gpt-5-nano, gpt-5-pro, or o3 name, dated or not; you run fine-tuned models on a 2024–2025 base; you use pro-tier reasoning; or you can't say from memory which model IDs live in your codebase.

Skip this if: you're already on the gpt-5.6 family (gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna), or on gpt-5.1 through gpt-5.5, none of which appear on either shutdown list as of this writing. Also skip if your only OpenAI usage is voice, realtime, or video. Those retire on separate schedules with their own dates and aren't covered here.

Don't treat a passing test suite as evidence. Everything on both lists still answers today. That's the trap. Nothing fails until the date, and then it all fails at once.

The honest limits of this check

Three things to hold against these results.

Model availability is scoped to an account. Our list of 126 models reflects one organization's tier and entitlements on one day. Yours will differ, which is exactly why the useful move is running the check yourself instead of citing ours.

Alias resolution is observed behaviour, not a written contract. OpenAI could repoint gpt-5 at a newer snapshot tomorrow without announcing it, and that would be a fix rather than a problem. It also means today's observation has a shelf life. Re-run it.

Error messages lie about causes. When we requested o3-pro-2025-06-10, the API returned a 404 saying the organization must be verified to use that model, saying nothing about deprecation. From the error text alone, "retired" and "you aren't entitled to this" look identical. Any audit script that probes models and parses error strings will mislabel some results.

We also didn't compare quality, latency, or cost between any old snapshot and its replacement. That comparison is unmeasured here, and we won't imply it.

What to do differently after reading this

Stop auditing for pinned date strings only. Search for the short names in the same pass: "gpt-5", "gpt-5-mini", "gpt-5-nano", "gpt-5-pro", "o3". Cover .env files, deployment configs, infrastructure-as-code, notebooks, and any prompt-management or routing layer. Model IDs sprawl across all of them and rarely sit in just one.

Then call the API once for each hit and read back the model field. That's the only answer reflecting reality rather than intent. Two minutes per service, and it turns "we think we're fine" into a fact.

Escalate instead of auto-fixing in two cases: fine-tuned models on retiring bases, and anything touching pro-tier reasoning. Neither one is a text substitution.

What Effloow added

OpenAI's deprecation page tells you which dated IDs end and when. It doesn't tell you what your undated alias currently resolves to, and that's where most of the exposure sits. We ran the resolution check against the live API and published the mapping, plus the token cost of the reasoning.mode: pro swap that the migration table implies but never prices. Complete commands, raw output, and failures are in the public lab note.

Related reading: our OpenAI Assistants API sunset port covers the same problem shape at the endpoint level, and the prompt tooling sunset migration scanner covers building the repository-scan side of this work. If billing behaviour under failure is your concern, see the spend limit 429 fail-closed audit.

Bottom Line

The short model name never protected you. It resolved to the same doomed snapshot as an explicit pin, minus the searchable string. Audit both this month, and read the model field back from a real request instead of trusting what your code says.

For your engineers

Everything below is method. Reproduce it before trusting it.

Environment. Python 3, standard library only (urllib.request, json). One standard OpenAI API account, not organization-verified for restricted models. Run date 2026-08-14.

Step 1: enumerate what the account can see.

req = urllib.request.Request(
    "https://api.openai.com/v1/models",
    headers={"Authorization": f"Bearer {key}"},
)
ids = sorted(m["id"] for m in json.load(urllib.request.urlopen(req))["data"])

Returned 126 IDs. Diff that set against both deprecation lists. gpt-4-1106-preview and o3-pro-2025-06-10 were already absent. Note that the bare string gpt-5.6 is not a valid ID; only gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna resolve.

Step 2: resolve aliases via the Responses API.

payload = {
    "model": "gpt-5",
    "input": "Reply with the single word: ok",
    "max_output_tokens": 16,
    "reasoning": {"effort": "low"},
}
# then read response["model"]

At max_output_tokens: 16 the reasoning models return status: "incomplete". Expected, and irrelevant here. The model field populates regardless, and it was the only field under test. Cost per probe is a few tokens.

Step 3: price the pro-mode parameter. Identical model (gpt-5.6-sol) and identical prompt, varying only the reasoning object between {"effort": "low"} and {"mode": "pro"}. Read usage.input_tokens. Results: 13 → 1,530 on the short prompt, 167 → 2,145 on the longer one. Two pairs, single account, single date. Treat it as a signal to measure your own workload, not as a published figure.

Step 4: error shape reference. A nonexistent ID (gpt-5-2025-08-08) returns HTTP 400 with code: "model_not_found". An entitlement failure (o3-pro-2025-06-10 on an unverified org) returns HTTP 404 with a verification message. Neither is a deprecation signal. If you build a probe-based scanner, treat 404-with-verification-text as "unknown" rather than "retired".

Primary sources.

OpenAI model deprecations — both shutdown waves, ID lists, and replacement mappings (fetched 2026-08-14)
GET https://api.openai.com/v1/models — live account model inventory
POST https://api.openai.com/v1/responses — alias resolution and token accounting
Effloow lab note — commands, raw results, failures, and limitations

Voice, realtime, and video model retirements run on separate schedules and are deliberately out of scope here.

Three of ten never became an element: where robots meta actually lands

Jangwook Kim — Thu, 13 Aug 2026 06:47:14 +0000

I ran ten HTML documents through a spec-compliant parser and counted where <meta name="robots"> actually ended up. Two stayed in the head. Five moved to the body. In the remaining three, no meta element was built at all.

That third bucket is why I'm writing this. Head versus body is a difference you can see. Element versus no element is invisible until you parse. And from the point of view of anything reading directives, a node that was never built is far worse than a node in the wrong place.

What actually closes the head

Start with the mechanism, because everything below depends on it.

A browser doesn't use your HTML string as written. It tokenizes it, then builds an element tree according to a fixed set of rules. That process is tree construction, and it carries a piece of state called the insertion mode. While the parser works through <head>, it sits in the "in head" mode.

The question that matters is when that mode ends. Most of us assume it ends at the </head> we typed. It doesn't. The moment anything appears that can't live in the head, the parser closes the head, opens the body, and reprocesses that content there. Your </head> is one of several exits, not the only one.

So the boundary between head and body isn't a value you declare. It's a result the parser computes. You can write a perfectly correct directive and still have no control over which side of that line it lands on. What the values themselves do is a separate topic, and I covered the directives one by one in the meta line that decides whether AI Overviews cite your page. Today is only about location.

The line Google added in March

Google's robots meta documentation has always said the same thing about placement: "Place the robots meta tag in the <head> section of a given page."

Then, on March 24, 2026, the Search Central documentation update log records a note added to the robots meta docs about how Google processes tags outside the HTML head. Here's that note, from the robots meta tag specification:

Google Search doesn't enforce placement of meta robots in the HTML head and will respect robots meta tags in the body section of an HTML document as well.

Clear enough. The recommendation is still the head; the tolerance is now written down.

I'd push back on reading this as permission to start putting directives in the body. Almost nobody authors a robots meta tag in the body on purpose. In practice, a robots meta tag sitting in the body is a tag someone wrote in the head that the parser carried out of it.

Which means the note isn't handing you new freedom. It's taking one class of silent failure and making it stop being a failure, for Google. The failure itself is still there.

Ten placements, one spec parser

I built ten documents in a throwaway sandbox. Every one contains the same tag, <meta name="robots" content="noindex">, and the only variable is where that line sits. I parsed them with parse5 8.0.1, which implements the HTML Standard's tree construction algorithm and lets you flip the scripting flag on and off.

After parsing, I walked the tree looking for meta[name=robots] and recorded the element's ancestor chain. When no element turned up, I also checked whether the original string survived somewhere as a text node.

Markup	scripting on	scripting off
A. in head (baseline)	`head > meta`	`head > meta`
B. in head, after a comment	`head > meta`	`head > meta`
C. in head, after stray text	`body > meta`	`body > meta`
D. in head, after a `<div>`	`body > meta`	`body > meta`
E. body, first child	`body > meta`	`body > meta`
F. body, last child	`body > meta`	`body > meta`
G. inside `<noscript>`	no element (text)	`head > noscript > meta`
H. inside `<template>`	separate fragment	separate fragment
I. in head, unclosed `<title>`	no element (text)	no element (text)
J. body, inside a `<div>`	`body > div > meta`	`body > div > meta`

C and D are the accident I described above. A single text node reading hello inside the head was enough to push the following meta into the body. One <div> did the same. Comments are allowed in the head, so B came through untouched.

That gap is where real sites break. A tag manager snippet, a banner injected by the server, one non-whitespace character left behind by a template. Any of them can close the head early. And robots meta isn't the only thing that gets pushed out. Your canonical and your hreflang go with it.

C, D, and the deliberate E, F, and J all fall under what Google now says it respects. So far, so reassuring.

A noindex inside noscript does the opposite of what it looks like

G is the row I stared at longest.

Wrapping noindex in <noscript> looks defensive. The intent reads as "keep the directive around even where scripts don't run." The actual behavior is inverted.

The HTML Standard defines how noscript works, and the wording is unusually blunt. The original is in the noscript element section of the WHATWG HTML Standard:

The noscript element is only effective in the HTML syntax, it has no effect in the XML syntax. This is because the way it works is by essentially "turning off" the parser when scripts are enabled, so that the contents of the element are treated as pure text and not as real elements.

With scripts enabled, the contents are plain text. Not elements. No meta node gets built, and with no node there's nothing to respect.

So which side is Google on? JavaScript SEO Basics answers it in one sentence: "Google Search runs JavaScript with an evergreen version of Chromium." Scripting on.

A robots directive inside <noscript> doesn't exist in the tree Google sees. It isn't ignored for being in the wrong place. There's simply no element to read.

There's a second fork here worth knowing about. I fed the same markup to jsdom 30.0.1 and changed nothing but the options.

runScripts=undefined      -> meta element present: true  | noscript text length: 0
runScripts=outside-only   -> meta element present: true  | noscript text length: 0
runScripts=dangerously    -> meta element present: false | noscript text length: 38

The default says the element is there. Turn scripting on and the same library says it isn't. Same string, same parser, opposite answers. And when teams wire jsdom into CI, they overwhelmingly leave the defaults alone. That's how you end up with a check that passes on a directive Google never sees.

H and I are simpler. Content inside <template> goes into a separate DocumentFragment rather than the document tree, so document.querySelector can't reach it and nothing reads it as a directive. I is an unclosed <title>, and since title swallows its content as text, everything after it became part of the title string. That one loses the element regardless of the scripting flag.

My own checker was stricter than Google

Knowing where the directive lands, I turned to the code that goes looking for it. Internal linters, SEO crawlers, prerender verification: the lookup is almost always a one-line selector. I took the two forms I see most often and ran the same fixtures through jsdom 30.0.1 again.

Markup	`document.head.querySelector`	`document.querySelector`
A. in head (baseline)	found	found
C. in head, after stray text	missed	found
D. in head, after a `<div>`	missed	found
E. body, first child	missed	found
G. inside `<noscript>`	found	found
H. inside `<template>`	missed	missed
I. in head, unclosed `<title>`	missed	missed
K. injected by JS after parse	raw false / after script true	raw false / after script true

The head-scoped lookup missed the directive in C, D, and E. Those three are exactly the placements Google documented as respected. My checker had become stricter than Google.

Strict isn't the problem. The direction is. It reports "no directive" on pages where the directive exists and works. Then in G it reports "directive present" when Google's tree contains nothing. Two errors, pointing opposite ways.

The document-scoped lookup caught C, D, and E correctly, and waved G straight through. Neither selector covers the table on its own.

Anything the renderer has to build is a weaker guarantee

K is the case where the initial HTML has no robots meta and a script appends one to document.head. Absent right after parsing, present after the script runs. Obvious in isolation. On the search side, that "after" comes with a condition attached.

The same JavaScript SEO doc describes the rendering order:

Googlebot queues all pages with a 200 HTTP status code for rendering, unless a robots meta tag or header tells Google not to index the page.

And then the warning that matters:

When Google encounters the noindex tag, it may skip rendering and JavaScript execution, which means using JavaScript to change or remove the robots meta tag from noindex may not work as expected.

Meeting a noindex may cost you rendering and JavaScript execution entirely, so removing that noindex with JavaScript may never happen. Google's own advice is to keep noindex out of the original page code if you want the page indexed.

Lay that sentence over the parsing results and one rule falls out. Bytes in the initial HTML are the strongest guarantee you have, and everything that depends on the renderer is weaker than that. <noscript>, <template>, and JS injection are all weaker, for different reasons. The first two produce no element even when the renderer runs. The last one produces nothing unless it does.

If you want the layer before the crawler arrives, I wrote that up in controlling AI crawlers properly with robots.txt. This post is the layer after.

Five things to ask the parser before you ship

Turning the results into checks:

Search the whole document. Use document.querySelector, not document.head.querySelector. A head-scoped lookup reports "missing" on placements Google respects.
Check the ancestor chain once you find it. If template or noscript sits anywhere above the element, treat the directive as absent. Present-or-missing isn't enough; you need present-and-where.
Parse with scripting enabled. Google renders with evergreen Chromium, so match that condition. The jsdom default sits on the other side.
Make head-versus-body a warning, not an error. Google respects the body placement. But finding it there tells you the head closed early, so go check whether canonical and hreflang got pushed out alongside it.
Put noindex in the initial HTML or leave it out. Don't toggle it with JavaScript. That's Google's recommendation, not my preference.

Points 2 and 3 come to roughly this much code. Short enough to drop into an existing checker.

import { parse } from 'parse5';

export function findRobotsDirective(html) {
  const doc = parse(html, { scriptingEnabled: true }); // same condition as Google
  const stack = [{ node: doc, path: [] }];
  while (stack.length) {
    const { node, path } = stack.pop();
    const name = node.tagName ?? node.nodeName;
    if (node.tagName === 'meta') {
      const attrs = Object.fromEntries(node.attrs.map((a) => [a.name, a.value]));
      if ((attrs.name ?? '').toLowerCase() === 'robots') {
        return { content: attrs.content, path: [...path, name] };
      }
    }
    // a template's contents live in its content fragment, not in childNodes
    if (node.tagName === 'template') stack.push({ node: node.content, path: [...path, name] });
    for (const child of node.childNodes ?? []) stack.push({ node: child, path: [...path, name] });
  }
  return null;
}

// usage: a template or noscript ancestor means the directive is never read
const found = findRobotsDirective(servedHtml);
const dead = found?.path.some((n) => n === 'template' || n === 'noscript');

I pointed this at my own site too. Three pages from the build output, same parser: 60 children in the head, zero of them elements that can't live there. No early close.

That's not a result worth bragging about, though. BaseHead.astro only emits robots meta when a noindex value is set, so most of my pages don't carry the tag at all. The head didn't stay quiet because it's well guarded. It stayed quiet because there was nothing to push out. The pages that do emit a directive, like the 404, are the risky ones, and I didn't count those this time.

One more boundary on the measurement. I measured how parse5 and jsdom implement the HTML Standard, not what Googlebot does. Both libraries implement the standard algorithm and Google uses Chromium, so I expect the tree construction to agree. Expecting is inference, not measurement. I didn't test Bing or any other crawler and I'm claiming nothing about them. And a directive sitting in the right place is a question of indexing and display eligibility, never of ranking.

Only Google got lenient

One thing nagged at me the whole way through. Google deciding not to enforce placement looks like an engineering team accepting reality. The web's HTML is broken, and salvaging a directive out of a broken head serves users better than discarding it. That's a sound call.

What bothers me is that the leniency landed in exactly one place. My build pipeline still checks the head. No other engine has published the same sentence. And above all, a head that closed early is still a bug, whatever survived it. Robots meta making it through says nothing about the canonical that was sitting next to it.

So I've decided not to read the note as "the body is fine." I read it as "if you found it in the body, go find out where the head closed." That version is more useful at work. How long it stays useful, I don't know. Once other engines write the same line and frameworks take over head management completely, this check can disappear. We're not there.

Tracing where a directive vanishes inside a rendering pipeline is part of what I do for a living. The contact page is the way in.

Sources: Google Search Central's Robots Meta Tags Specifications, JavaScript SEO Basics, Latest Google Search Documentation Updates, and WHATWG's HTML Standard, The noscript element (all official). The four block quotes were pulled from those pages and compared against the fetched source on the spot, with the source link placed next to each quote. Measurement setup: 10 fixture documents in a throwaway sandbox directory, parse5 8.0.1, jsdom 30.0.1, Node 22.22, macOS, measured 13 August 2026. Probes are scripts/probe-robots-meta-placement.mjs and scripts/probe-robots-meta-consumer.mjs, raw data is data/robots-meta-placement.json and data/robots-meta-consumer.json, figures come from scripts/chart-robots-meta-placement.py. What I measured is the tree construction result of two libraries, not Googlebot's actual processing. I did not check Bing or other crawlers. Robots directives govern indexing and display eligibility, not ranking.

CSS Cannot Hear the Escape Key: Seven Tooltips Against SC 1.4.13

Jangwook Kim — Wed, 12 Aug 2026 06:48:09 +0000

Your accessibility scan is green. It has no opinion whatsoever about your tooltips.

That's not a dig at the scanner. WCAG 2.2 Success Criterion 1.4.13 asks three questions that a static DOM cannot answer: can you close this thing, can you move the pointer onto it, and how long does it stay. Answering those requires pressing a key, dragging a pointer, and waiting. Automated rules don't do any of that, so they say nothing, and silence reads like approval.

So I built the same tooltip seven different ways in a throwaway sandbox and measured the three requirements one at a time. Every CSS-only version failed Dismissible. The native popover="hint" passed Dismissible without a line of code from me, then failed Hoverable. One implementation out of seven passed all three.

What the criterion is actually pointed at

Worth grounding this before the numbers. SC 1.4.13 is Level AA. It arrived in WCAG 2.1 and carried into 2.2. Its subject is additional content that appears when you hover or focus a trigger and disappears when you stop. Tooltips are the obvious case. Hover-opened mega menus, profile cards that pop up over a username, help bubbles beside a form field, all of it counts.

What doesn't count is spelled out too. The third note attached to the criterion says this, and the source is the W3C's WCAG 2.2 Recommendation:

This criterion applies to content that appears in addition to the triggering component itself. Since hidden components that are made visible on keyboard focus (such as links used to skip to another part of a page) do not present additional content they are not covered by this criterion.

A skip link that reveals itself on focus is out of scope. Something separate from the trigger has to appear before you're in 1.4.13 territory. Draw that line first or you'll audit half your page against the wrong criterion.

It's also worth knowing who the three requirements protect. All three point at people using screen magnification and people whose pointer control isn't precise. At 400% zoom a tooltip covers a serious fraction of the viewport, and if there's no way to clear it, the text underneath is simply gone. Someone with an unsteady hand moves the mouse toward the tooltip body to read it and loses the tooltip on the way. And a popup that vanishes after a couple of seconds may as well not exist for a slow reader. Dismissible, Hoverable and Persistent map onto those three situations one for one.

What the normative text says

The normative text is short. Here it is verbatim from the W3C's SC 1.4.13:

Where receiving and then removing pointer hover or keyboard focus triggers additional content to become visible and then hidden, the following are true:

Dismissible: A mechanism is available to dismiss the additional content without moving pointer hover or keyboard focus, unless the additional content communicates an input error or does not obscure or replace other content;

Hoverable: If pointer hover can trigger the additional content, then the pointer can be moved over the additional content without the additional content disappearing;

Persistent: The additional content remains visible until the hover or focus trigger is removed, the user dismisses it, or its information is no longer valid.

Exception: The visual presentation of the additional content is controlled by the user agent and is not modified by the author.

Two clauses deserve a second read.

First, Dismissible carries a conditional. If the additional content communicates an input error, or if it doesn't obscure or replace other content, the requirement is waived. A tooltip that covers nothing needs no dismissal mechanism. Those are rare, though. I measured the popup rectangle against the paragraph directly below it in all six DOM variants, and all six overlapped.

Second, the exception only covers presentation the browser controls and the author hasn't touched. The first note on the same page names the case:

Examples of additional content controlled by the user agent include browser tooltips created through use of the HTML title attribute [HTML].

So title sits outside the criterion. This is where the common misreading starts: outside the criterion is not the same as passing it. A title tooltip still doesn't appear on touch devices, and you control neither when it shows nor how long it stays. The exception defers the verdict rather than fixing anything.

Seven implementations, one ruler

One static HTML page in a temp directory outside the repo. Same button, same sentence ("Rate limit: 60 requests per minute per API key."), same 8px gap between trigger and popup. All seven rows share that much; the only variable is the machinery that opens and closes the thing.

	Implementation	Opens on
V1	`title` attribute	drawn by the browser
V2	CSS `:hover` only	pointer
V3	CSS `:hover` + `:focus-visible`	pointer, keyboard
V4	CSS `:has()` + padding bridge	pointer, keyboard
V5	JS hover/focus/Escape + 150ms grace	pointer, keyboard
V6	JS + 2s auto-hide	pointer, keyboard
V7	native `popover="hint"`	pointer, keyboard

Playwright did the judging. Each check reloads the page so state can't leak between probes, and five things get measured.

Does hovering open it
Does tabbing to it open it
Dismissible: with it open and the pointer held still, does Escape close it
Hoverable: walking the pointer from trigger centre to popup centre in twelve steps, is it still alive at the end
Persistent: after hovering, does it survive five seconds of doing nothing

The Hoverable probe is the one worth explaining. A single mouse.move to the destination skips the hit-testing in between and sails straight over the gap, which would have quietly passed every variant. Twelve interpolated steps at 20ms each is closer to what a real hand does.

const path = 12, from = center(tb), to = center(pb);
for (let i = 1; i <= path; i++) {
  await page.mouse.move(
    from.x + (to.x - from.x) * i / path,
    from.y + (to.y - from.y) * i / path
  );
  await page.waitForTimeout(20);
}
r.hoverable = await page.locator(v.tip).isVisible();

Environment: Chromium 143.0.7499.4 headless, Playwright 1.57.0, Node 22.22, viewport 900×1400. I ran the script twice and all seven rows came back identical.

CSS cannot hear the Escape key

V2, V3 and V4 contain no JavaScript. All three failed Dismissible. That isn't sloppy implementation, it's structural. Dismissible wants a way to close the content without moving pointer or focus, and in practice that way is the Escape key. CSS has no selector that reacts to a keystroke.

The W3C's Understanding document uses exactly that example when it explains the requirement, pressing Escape to clear a tooltip without touching the mouse (my summary, not a verbatim quote). A CSS-only tooltip has no route to this requirement unless the waiver applies.

You can aim for the waiver. Build a tooltip that doesn't obscure or replace anything and Dismissible drops away. That's why I measured overlap as well, and all six overlapped the paragraph below. With the most common arrangement, absolutely positioned right under the trigger, not covering the following content is the harder thing to achieve. Reserve space in the layout flow and drop the popup into it and the waiver holds, but at that point you've built an accordion.

Here's the working judgement I take away. "You can build a tooltip with pure CSS" usually means "two of 1.4.13's three". If the third one drags a keyboard listener in anyway, managing state in JS from the start is the shorter path. A component that felt finished after two :hover lines ships unfinished, and that was the most repeatable finding in the whole run. I saw the same shape measuring Escape and inert on modals, except there at least everyone agreed the thing was supposed to close. Nobody expects that of a tooltip.

Keep the 8px in the eye, take it out of the box

Hoverable is where V3 and V4 parted ways. Same technology, same opening triggers. The difference wasn't the selector, it was the size of the box.

V3 gives the popup margin-top: 8px. It looks 8px away and it hit-tests 8px away. The instant the pointer leaves the trigger and lands in that strip, :hover releases and the popup is gone. The script measured the distance from trigger bottom to popup top at exactly 8px.

V4 builds the same 8px out of padding instead. The popup box touches the trigger, and 18px of transparent padding inside the box pushes only the visible content down. Measured gap: 0px. It looks detached and hit-tests as continuous. Add a :has() rule so hovering the popup itself keeps the open state, and Hoverable falls out of pure CSS.

/* popup touches the trigger; the breathing room lives inside the box */
#tip { display: none; margin-top: 0; padding-top: 18px; background: transparent; }
#tip .inner { background: #111827; color: #f9fafb; border-radius: 6px; padding: 10px 12px; }

.anchor:has(.trigger:hover) #tip,
.anchor:has(.trigger:focus-visible) #tip,
.anchor:has(#tip:hover) #tip { display: block; }

The other route is time. V5 keeps the real 8px gap and, instead of closing on mouseleave, schedules a close 150ms later and cancels that schedule if the pointer reaches the popup. While the pointer crosses the empty strip, the popup is still there.

let timer;
const open  = () => { clearTimeout(timer); tip.classList.add('open'); };
const close = () => tip.classList.remove('open');
const soft  = () => { clearTimeout(timer); timer = setTimeout(close, 150); };

trigger.addEventListener('mouseenter', open);
trigger.addEventListener('focus', open);
trigger.addEventListener('blur', close);
trigger.addEventListener('mouseleave', soft);
tip.addEventListener('mouseenter', open);
tip.addEventListener('mouseleave', soft);
document.addEventListener('keydown', (e) => { if (e.key === 'Escape') close(); });

Twenty lines, and the only one of seven that passed all three. Nothing clever in it. Listen for Escape, forgive the gap, don't hide on a timer.

The 150ms is my number, not the spec's. It was enough for a headless browser travelling in a straight line. How much a slow hand on a curved path needs, I didn't measure, and sweeping the grace value to find the breaking point is the obvious next run. The padding bridge is the safer of the two, because a gap of zero removes the question entirely.

popover gives you exactly half

V7 taught me the most. The popover attribute hands state management and the top layer to the browser. popover="hint" is the value aimed at tooltips, and per the WHATWG HTML Standard the auto and hint states have light dismiss and respond to close requests while manual does not (my summary, not a verbatim quote). Close requests include the Escape key.

The measurement matched. V7 closed on Escape with the pointer sitting still. It's the only one of the seven that got Dismissible without a line of code from me. A shipped platform feature absorbing an entire accessibility requirement is a good argument for why the value exists.

Then it failed Hoverable, for a boring reason. popover manages whether the thing is open, not when it should open or close. Build a hover tooltip on top of it and you still call showPopover() on mouseenter and hidePopover() on mouseleave, and that mouseleave fires as the pointer enters the 8px gap. Top layer or not, the pointer hasn't arrived yet.

The split is clean. CSS gives you Hoverable and Persistent and never Dismissible. popover gives you Dismissible and not Hoverable. The two halves don't overlap, so passing all three means keeping the grace timer or the padding bridge even when you adopt popover. Next time you read that popover solves tooltip accessibility, that's the sentence to check.

The tooltip that disappears after two seconds

V6 starts from a kindness. Tooltips lingering on screen feel intrusive, so hide it after two seconds. Plenty of UI libraries ship that default, and I've written it myself.

Held the pointer still, waited five seconds, and the popup was gone. Persistent, failed. The criterion allows three ways for the content to leave: the trigger is removed, the user dismisses it, or the information stops being valid. Elapsed time isn't among them.

You might wonder whether "no longer valid" can be stretched to cover a timer. Sometimes it genuinely does. A countdown, a one-time code with an expiry, anything whose content is bound to the clock. A sentence describing an API rate limit is just as true two seconds later. That's not grounds for taking it away from someone who reads slowly.

V6 lost Hoverable too, separately from the timer, because it also closes immediately on mouseleave. Missing all three at once turns out to be easy.

Zero of axe's 105 rules look at this

I ran axe-core 4.13.0 with all seven tooltips forced open. Two violations came back, landmark-one-main and region, both caused by my fixture having no landmarks and neither related to tooltip behaviour.

The rule list explains it. axe-core 4.13.0 carries 105 rules and exactly zero of them is tagged wcag1413.

axe-core 4.13.0 total rules: 105
rules tagged wcag1413: 0 []

Not a criticism of the tool. Dismissible, Hoverable and Persistent aren't properties you can read off a static DOM; you have to press, drag and wait. The blind spot I catalogued when counting axe's rule tags per success criterion shows up here intact. A green score and compliance with this criterion have nothing to do with each other.

Which means the scope of this run needs stating plainly. One engine (Chromium 143), one gap value (8px), one hand-built fixture. These numbers say nothing about violation rates on real sites and nothing about how other rendering engines behave. Touch input and the experience through assistive technology weren't measured at all. And my fixture passing three probes is not the same as a conformance verdict on a real page, which folds in the waivers and the context around them.

Delete the 8px or grant the 150ms

Short list of what I'd now hold to.

Escape lives outside CSS. If the tooltip covers anything behind it, attach a keyboard listener. Lean on the waiver only when you're certain it covers nothing.
Keep the gap in the eye and out of the box. 8px of margin is a pointer trap; 8px of padding isn't. What decides the verdict is hit-test distance, not visual distance.
If the gap has to stay, grant a grace period. Don't close on mouseleave; schedule it 100–200ms out and cancel on entry.
Never hide on a timer. Auto-hide is defensible only when the content itself is bound to the clock.
Wire the open/close logic yourself even with popover. What the browser takes off your hands stops at Dismissible.
Check by hand. No rule in the scanner covers this. Push the pointer into the popup, press Escape, wait five seconds. Three motions.

One thing I still haven't settled. Butting the popup against the trigger makes Hoverable certain, but it also blurs the boundary between the two, and there's a point where you can no longer tell where the button ends. How thick the padding has to get before that edge reads again, and whether that route really beats keeping the gap and granting the grace period, this fixture couldn't tell me. The next run sweeps the grace value and puts both routes on the same ruler.

If you're staring at a screen layered with dropdowns, menus and bubbles and can't tell where to start on these three, ask. Translating criterion text into component code, and making a script rather than a person re-check the verdict, is the work I do. Contact route is on my profile.

Sources: the W3C's WCAG 2.2 Success Criterion 1.4.13 Content on Hover or Focus (W3C Recommendation), Understanding SC 1.4.13, and the WHATWG HTML Standard, The popover attribute (all official). The success criterion text and notes 1 and 3 were copied verbatim after checking them against the Recommendation on the spot, with the source link placed next to the quote. The Understanding document and the HTML Standard are summarised and linked rather than quoted. Measurement environment: one static HTML page in a temporary sandbox directory (seven tooltip implementations), Chromium 143.0.7499.4 headless, Playwright 1.57.0, Node 22.22, viewport 900×1400, 8px gap between trigger and popup, axe-core 4.13.0, measured 12 August 2026. Probe script scripts/probe-hover-focus-1413.mjs, fixture scripts/fixtures-hover-focus-1413.html, raw data data/hover-focus-1413-probe.json. The script was run twice with identical results. Every verdict comes from this engine, this fixture and this gap value, and is not a statement about conformance of real sites or about other rendering engines. Touch input, assistive technology behaviour, and the browser-drawn title tooltip (not observable from the DOM) were not measured.

The markup said 'read this aloud' and pointed at 13 paragraphs

Jangwook Kim — Tue, 11 Aug 2026 06:46:28 +0000

article p:first-of-type looks like it selects one paragraph. It selected thirteen.

That was the median across my sample. On the widest page it grabbed twenty-four. And this selector wasn't sitting in a stylesheet where a wrong match would be visible. It was inside structured data, telling any voice assistant that reads my pages which parts to speak.

Two kinds of structured data, two ways to rot

Most structured data carries its own values. You write the title into headline, the date into datePublished, the name into author. Whatever you wrote is what ships, so mistakes surface: an empty title ships empty, a malformed date trips a format error.

speakable doesn't work that way. A SpeakableSpecification holds no text at all. It holds a cssSelector or an xPath — an address into the document. The value lives in the DOM; the markup only points at it. The idea is that an author, not a heuristic, decides which sentences a voice surface should read out.

Pointers have a different failure profile from values. When the target disappears, the markup stays perfectly healthy: valid JSON, correct type, all required properties present. Rename one CSS class and the instruction starts aiming at empty space, while your build stays green and every test passes. Nothing in a normal pipeline is watching that relationship. Among the structured data types Google documents, speakable is effectively the only one built this way.

And a pointer can rot in two directions. It can reach nothing, or it can reach far too much. My site had managed both at once.

What Google actually promises about speakable

Before touching anything I reread the speakable documentation. It sets its terms in the first paragraph:

This feature is in beta and subject to change. We're currently developing this feature and you may see changes in requirements or guidelines.

The audience is narrower than most people assume:

The speakable property works for users in the U.S. that have Google Home devices set to English, and publishers that publish content in English.

A one-person technical blog shipping Korean, Japanese, English and Chinese editions is not in that sentence. The selector guidance is one line:

Use either cssSelector or xPath; don't use both.

And the usual disclaimer, which I'd rather quote than paraphrase:

Google does not guarantee that features that consume structured data will show up in search results.

Meanwhile Speakable is still on the supported list in the Search Gallery. Those are separate facts, and blurring them is how a site ends up carrying markup on the theory that it might pay off someday. Worth remembering that Google has been shrinking that list, not growing it: the documentation update log shows a deprecation notice added to the FAQ rich result on 2026-05-08 ("This feature will no longer appear in Google Search starting May 7, 2026.") and the docs removed on 2026-06-15, with practice problem docs deleted back on 2026-01-06. When the FAQ result died I argued for keeping the Q&A markup anyway, because that vocabulary carries its own text and other parsers can still read it. The argument collapses for pointers. An address with nothing at the end of it is nothing, to every reader.

Running the selectors instead of reading them

Here's what my pages had been emitting:

{
  "@type": "SpeakableSpecification",
  "cssSelector": [
    "article h1",
    "article h2",
    "article p:first-of-type",
    ".article-summary"
  ]
}

Title, headings, lede. Clear intent. Intent isn't the thing that ships, though, so I opened the built HTML with jsdom and executed all four. Node 22.22, jsdom 29.1.1, against the dist as of 2026-08-11. Of 1,336 blog pages, 1,332 carried a SpeakableSpecification; I parsed a sample of twenty, five from each language.

Selector	Matches per page (median)	Total across 20 pages	Verdict
`article h1`	1	24	two h1s on 4 pages
`article h2`	9	229	every section heading
`article p:first-of-type`	13	272	over-matching
`.article-summary`	0	0	matches nothing

The zero has a boring explanation: no component on this site uses that class. Whether it existed once or was only ever planned, I couldn't settle from the commit history. What's certain is that the selector rode along on 1,332 pages without ever pointing at anything.

There's a trap worth naming here. Grep dist for article-summary and it hits on all 1,332 pages, which reads like confirmation. Open one and the string appears exactly once — as the selector's own value inside the JSON-LD.

...akableSpecification","cssSelector":["article h1","article h2",
"article p:first-of-type",".article-summary"]},"url":"https://jangwo...

The pointer shows up in a text search because of its own name. String matching cannot detect this class of rot. Selectors have to be executed.

Where those 272 paragraphs actually lived

The thirteen interested me more than the zero, so I grouped every matched paragraph by its parent element.

Parent element	Paragraphs matched	What it really is
`li`	73	paragraphs inside list items
`div.item-content`	60	related-post recommendation cards
`blockquote`	59	pull quotes
`header.article-shell__header`	20	post header
`div.article-prose`	20	the actual lede
`div.text-center`	20	layout chrome
`div.flex-1`	20	layout chrome

Twenty out of 272 were what I meant. The cause is the selector definition rather than anything exotic: :first-of-type means first sibling of that type under its own parent, not first occurrence in the document. Combine it with a descendant combinator that walks the whole subtree, and every container holding paragraphs contributes one.

The div.item-content row is the one that stung. Those sixty are the blurbs on related-post cards — machine-written navigation copy. Had a voice surface honored this annotation, it would have treated three recommendation blurbs as core spoken content on equal footing with the article's opening sentence. My own thesis was outvoted by my own sidebar.

I recognize the shape of this from auditing text fragment deep links, where 14 of 15 code-block citations broke. Same species of bug. Pointers rarely break when you write them; they break when everything around them moves.

What each tool can and cannot see

The schema.org Schema Markup Validator caught the null selector. Feeding it a live URL returned three objects and exactly one error: NO_MATCHES_FOUND, isSevere: true, naming .article-summary. That's more than syntax checking — the validator runs your selectors against the fetched document, which is a genuinely useful behavior that I suspect most people never exercise.

It said nothing about the thirteen, and it shouldn't have. Matching many nodes is legal; speakable accepts an array and multiple targets are a supported pattern. Nothing here is invalid. It's just wrong.

Failure mode	Schema validator	Build	Text grep	What actually catches it
Selector matches 0 nodes	caught (severe)	no	no	deploy gate
Selector over-matches	no	no	no	a count ceiling you write
Used outside its stated audience	no	no	no	human judgment

That third row has no tooling answer. Whether a multilingual personal blog should ship a feature documented for U.S. English Google Home users is a question no linter will settle for you.

The fix, and the assertion that keeps it fixed

Two selectors instead of four, each scoped to hit exactly once:

// src/components/BaseHead.astro
const speakableSchema = articleData ? {
  '@context': 'https://schema.org',
  '@type': 'WebPage',
  'speakable': {
    '@type': 'SpeakableSpecification',
    'cssSelector': ['.article-shell__header h1', '.article-prose > p:first-of-type']
  },
  'url': canonicalURL.toString()
} : null;

The load-bearing change is the child combinator. .article-prose > p:first-of-type only considers direct children of the prose container, so paragraphs nested in lists and blockquotes never enter the candidate set. Re-measured across the same twenty pages:

Selector	Pages matched	Nodes per page
`.article-shell__header h1`	20 / 20	1
`.article-prose > p:first-of-type`	20 / 20	1

Then a postbuild assertion, so this can't silently rot again. Open the output, run the selectors, fail on zero, fail when a paragraph selector blows past a ceiling.

// scripts/validate-speakable.mjs (core)
for (const selector of selectors) {
  const count = dom.window.document.querySelectorAll(selector).length;
  if (count === 0) {
    failures.push(`${file}: "${selector}" matches nothing`);
  } else if (/\bp\b|paragraph/.test(selector) && count > MAX_PARAGRAPH_MATCHES) {
    failures.push(`${file}: "${selector}" matches ${count}`);
  }
}

Point it at the pre-fix dist and it fails exactly 40 times: twenty pages times two bad selectors.

❌ validate-speakable failed (40)
  - dist/en/blog/en/45-day-analytics-report-2025-11/index.html:
      "article p:first-of-type" matches 24 (limit 2)
  - dist/en/blog/en/45-day-analytics-report-2025-11/index.html:
      ".article-summary" matches nothing
  ...

When I built the CI validation for JSON-LD, I checked syntax and required properties. That check would have passed this markup every single day, because the syntax was immaculate. Pointer-valued properties need a separate question asked of them: how many nodes does this actually resolve to?

One honest boundary. I'm not claiming this improves my search presence. I'm not an English-language news publisher serving U.S. Google Home users, Google states outright that structured data guarantees nothing about appearing in results, and I have no evidence that LLM crawlers read speakable at all — so I won't write as though they do. What I fixed is the accuracy of a statement my site makes to machines. Two true lines beat 1,332 pages of a false one. The sample is twenty pages, not the full corpus, and I'd rather say so than round up.

Before you ship markup that points instead of carries

Search your structured data for cssSelector and xPath first. Properties holding addresses need their own care.
Verify selectors by executing them, never by grepping. A selector string always matches itself.
Any time you see :first-of-type, :first-child, or a descendant combinator, count the matches. Intent of one and a result in double digits means you want >.
The schema validator flags zero matches as severe and waves over-matching straight through. The ceiling is yours to write.
When renaming CSS classes, search the JSON-LD too. No linter knows your stylesheet and your structured data share a class name.
Read the audience restrictions before adding a type at all. Beta notices and country or language limits usually sit in the first paragraph of the docs.

One thing I haven't resolved: whether keeping those two lines is the right call when I know I'm outside the documented audience. Deleting them removes something to maintain. Keeping them leaves one machine-readable statement that the heart of the page is its title and its opening paragraph. The gate now guarantees that statement is true, which tipped me toward keeping it. Ask me again in six months.

Wiring structured data into deploy gates is the kind of work I take on. Contact routes are on my profile.

Chrome's agent skill: 138 guides, 2 on a11y, 0 on search

Jangwook Kim — Mon, 10 Aug 2026 06:54:51 +0000

Chrome shipped Modern Web Guidance at I/O 2026. It's a bundle of skills that pushes web platform knowledge into your coding agent, and the announcement says it helps you build web experiences that are "more accessible, performant, and secure." Installing it takes one line of npx.

The first thing I did after that line finished was run list and count the guides. 138 of them, across 15 categories. Two sit in the accessibility category. There is no category for structured data or crawlability at all.

The count isn't a flaw by itself. But there's a gap between the developer who reads those three words and installs, and where the corpus actually puts its weight. Today I measure that gap, then write down what goes where to close it.

An agent skill is a scope declaration, not a feature

Let's line up the terms first. An agent skill is a bundle of instructions your coding agent opens when it hits a particular kind of task. The core of it is a single SKILL.md, which says when to open the bundle and what to do once it's open. The agent reads that trigger description on every task and decides for itself whether to fire. So installing a skill isn't really adding a tool. It's drawing a line around the agent's judgment.

Baseline deserves a paragraph too. Baseline is the web.dev classification for whether a web feature is stable enough to rely on across the major browsers. Two stages matter: Newly available means it has just landed in every major engine, and Widely available means enough time has passed that it's safe essentially everywhere. Any honest answer to "can I use this yet" eventually comes back to that data.

Modern Web Guidance welds the two together. Here is the definition from the Chrome for Developers docs, verbatim.

Modern Web Guidance is a set of skills that embed web platform expertise, best practices, and browser compatibility data directly into your coding agents.

The I/O 2026 post describes the Baseline hook. Also verbatim.

It integrates directly with Baseline, letting you focus on what you want to build while your tools automatically figure out the right features and fallbacks to use within your chosen Baseline target.

The intent is clear and I think the direction is right. Injecting current compatibility data at the moment of work beats hand-stripping 2019-vintage CSS idioms out of model weights every single time. What matters is the shape of the map being injected. Off the map, the agent goes back to its weights. And it won't tell you that's what happened.

What actually lands in your repo

On npm, the package first appeared on 2026-04-30 and reached 96 versions by 2026-08-03. That's a release every three days. The latest at the time I measured was 0.0.180, Apache-2.0, 36.6 MB unpacked across 198 files. Read all of this as a snapshot of an early-preview 0.0.x.

I ran the install in an empty directory.

npx modern-web-guidance@latest install

The installer brings up an interactive screen and detects your agents. Mine reported this.

✓ ./.agents/skills/modern-web-guidance
  universal: Amp, Antigravity, Antigravity CLI, Codex, Cursor +12 more
  symlinked: Claude Code

That's 1.2 MB and 140 files under .agents/skills/modern-web-guidance/, plus a skills-lock.json at the root. Claude Code gets a symlink. Every guide is plain Markdown, so you can open one and read it. I like that a lot. When the agent gives you an answer, you can go find the text it was reading.

The install ends with a telemetry notice. Straight from the repository README:

Google collects anonymous usage statistics (such as search queries, guide retrievals, and installation) to improve the reliability, relevance, and performance of the tool.

Note that search queries are explicitly in scope. On an internal repo, query strings pick up project context fast. One environment variable turns it off.

export DISABLE_TELEMETRY=1

Worth mentioning: the installer prints third-party security assessments next to the package. My run showed Socket at 0 alerts and Snyk at Med Risk. Those are third-party scanner verdicts, not an official Google rating, and I didn't chase down the reasoning behind them. Treat them as reference values.

Where the 138 guides pile up

The list command dumps every guide as JSON. Counted by category, it splits like this.

Category	Guides
ui-behaviors	29
performance	24
visual-design	16
forms	15
css	14
ui-atoms	9
js	8
security	7
built-in-ai	4
ui-components	4
webmcp	3
accessibility	2
css-layout	1
html	1
privacy	1
search / structured data	0

The top five categories hold 98 guides, 71% of the corpus. The weight sits on making things that appear on screen.

One point of fairness. Two accessibility guides is not the total volume of accessibility content. One of those two, the umbrella accessibility guide, runs 7,131 tokens and is the third largest document in the whole package. Its table of contents moves through navigability and structure, semantic HTML and ARIA, accessible names, document metadata and language, keyboard and focus management, then alternate text and media. The fifteen forms guides carry label and autofill accessibility inside them too. So "only two accessibility guides" is a statement about category labels, not about substance.

The structural difference still stands, though. The UI side has twenty-nine narrow this-situation-use-this-feature guides laid out flat. Accessibility has its detail folded into one thick primer. In a system where the agent retrieves narrow answers by search, that difference shows up in the output.

Quality inside the covered territory deserves its own verdict. I opened optimize-image-priority in the performance category, and it's good. It does not stop at the usual "put fetchpriority="high" on your LCP image."

4. **Optimize lazy loading**: Never use `loading="lazy"` on the LCP image.
   For standard below-the-fold images, `loading="lazy"` is sufficient...
   Avoid adding `fetchpriority="low"` to these images, as you want them to
   load at normal priority once the user scrolls to them.

It separates ordinary below-the-fold images from images that are technically above the fold but not initially visible, like carousel slides behind the first one or a mega menu. Teams get that distinction wrong constantly, and getting it wrong costs you bandwidth contention rather than LCP. If that resolution of advice runs through a decent share of the 138, then yes, there is real territory where this tool beats the weights. The question is where that territory ends.

Three kinds of question, one tool

So I probed it. Twenty-two queries: six on UI and CSS, ten on accessibility, six on search and structured data. For each I logged the top result's similarity and how many results came back. I wrote the queries as plain English task instructions, which is the favorable condition here, since the corpus is English.

npx modern-web-guidance@0.0.180 search "add JSON-LD structured data for local business"

[{"id":"language-model","description":"...","category":"built-in-ai",
  "tokenCount":1984,"similarity":0.357}]

One hit, and it's the guide for the browser's built-in language model API. Nothing to do with structured data.

By group:

Query group	Queries	Mean top-1 similarity	Empty
UI / CSS	6	0.643	0
Accessibility	10	0.508	0
Search / structured data	6	0.267	2

The UI queries land cleanly. "custom styled select dropdown" pulled custom-select-picker-layouts at 0.724; "view transition between pages" pulled cross-document-transitions at 0.703. All six found their answer inside the right category.

Accessibility queries always return something, but the aim wanders. In seven of ten, at least one accessibility-category guide appeared somewhere in the top five. The other three are the interesting ones. "associate a label with a form input" returned five forms guides about autofill and nothing else. "minimum target size for touch controls" landed on the css primer. "reflow at 400% zoom without horizontal scroll" landed on defer-work-until-scroll-ends, a performance guide. That last one is badly off: reflow at 400% zoom is a viewport-dimension problem, not a scroll-performance problem. I measured that exact case and found the damage was vertical, not horizontal — an answer this corpus doesn't contain anywhere.

The search group looks nothing like the other two. "canonical link tag for duplicate pages" and "sitemap and robots.txt for a static site" both returned zero results, meaning nothing cleared the tool's threshold. "render meta description and title tags" came back with an accessibility guide at 0.378, and "get cited by AI search answers" came back with the language-detection API guide at 0.362.

Now the limits. Twenty-two queries is a probe, not a benchmark. Reword a query and the similarity moves. The threshold and cutoff belong to the tool; I didn't touch them. And a miss at the top does not prove the agent writes bad code, because the umbrella guides are thick enough that the right passage sometimes lives inside a wrong-looking hit. Google's own eval in the README (July 6, 129 tasks and 1,071 assertions) reports codex_cli going from 57% to 84%, a 27-point lift, and claude_code from 52% to 87%, a 35-point lift. Those are their numbers. I did not run their suite.

What I can say from my own measurements is narrower: this corpus is strong at building the screen and silent about whether the page gets found and cited.

Put the Baseline target in a project file

This part you can use today. The installed SKILL.md (package source, GoogleChrome/modern-web-guidance) spells out how browser support gets decided. The default, verbatim:

All guides assume Baseline Widely available features are safe to use without fallbacks.

Declare nothing and the agent treats only Widely available as unconditionally safe, attaching fallbacks to everything below that line. If your project can be more aggressive, say so. There's no required format; prose in AGENTS.md or CLAUDE.md is enough. The year-target rule is in SKILL.md as well: for a Baseline YYYY target, a feature qualifies when its "Baseline since" date is on or before that year.

Here's the shape I use.

## Browser Support

Baseline target: Baseline 2024.
Newly available features are allowed when feature-detected.
Accept fallback code only if it stays under 20 lines and adds no dependency.
If it can't, change the implementation instead of writing the fallback.

Four lines, and the agent stops asking. The data it needs is already inside the guides: 74 of the 138 state Baseline status in the body. The image priority guide, for instance, carries this line.

Baseline status for Fetch priority: Newly available.
It's been Baseline since 2024-10-29.

I timed the calls too. The first run spent 10.7 seconds pulling the package down; once cached, three consecutive runs took 2.09s, 1.15s, and 1.22s. Since SKILL.md demands the agent run search first on every HTML, CSS, or client-side JS task, assume that round trip attaches to each task. The context cost is the bigger one. Search returns a tokenCount per guide: narrow guides run 900 to 3,000 tokens, while the umbrellas are css at 7,755, accessibility at 7,131, and performance at 5,599. Open two umbrellas and you've spent 15,000 tokens of context. That's not an argument against it. Just know the budget you're spending.

Build a second rules layer where the corpus is empty

The behavior that matters most is what the agent does when search comes back empty. SKILL.md tells it to fall back to list and browse everything. But for a subject that isn't in the list either, the conclusion the agent reaches is "this project has no rule about that." No rule reads as freedom. Freely written JSON-LD tends to get injected client-side, canonicals go missing, and title tags get assembled somewhere inside a component.

So I ship a second rules layer alongside the skill, covering only the axes the corpus leaves alone.

## Search & structured data (outside the skill corpus — project rules)

- Structured data ships in server-rendered HTML. Never inject it client-side.
- Every page carries a self-referencing canonical; localized pages cross-reference via hreflang.
- title and meta description come from the route definition, not assembled inside a component.
- Body text must exist in the response HTML without JS, including content inside tabs and accordions.
- Any JSON-LD change runs schema validation in CI.

## Accessibility acceptance (what automated checks won't catch)

- Judge every new overlay against WCAG 2.4.11, walking Shift+Tab in reverse as well.
- Interactive targets satisfy 2.5.8's 24x24 CSS px on both width and height.
- No two-dimensional scrolling at a 320x200 viewport (1.4.10).
- Verify all three separately, regardless of what axe reports.

That second block isn't arbitrary. Its items map onto exactly the queries where the aim wandered. I once matched axe's rules against WCAG success criteria to see which criteria a scanner actually decides, and the lesson transfers directly: a tool won't tell you what it can't see. Someone has to write that list by hand and pin it up.

The first line of the search block isn't taste either. It's the conclusion from measuring what really changes when LocalBusiness markup ships server-side versus getting attached with JavaScript, compressed into one sentence of policy.

A 30-minute check before you adopt it

None of this means don't install it. I did, and I plan to keep it. But five things are worth checking yourself first, and together they take about half an hour.

Run npx modern-web-guidance@latest list and count how many guides cover the work your team actually does. If a domain isn't there, drop it from your expectations up front.
Take five real task descriptions from your last sprint and feed them to search unchanged. If more than half come back under 0.5, the tool doesn't fit your team's main battlefield yet.
Put a Baseline target sentence in AGENTS.md or CLAUDE.md. If you skip it, make sure everyone knows the default is Widely available.
Write project rules for the axes the corpus omits: search and structured data, plus the accessibility criteria no scanner decides. Copy the blocks above and cut them down to your situation.
If sending query strings outside your org is a problem, put DISABLE_TELEMETRY=1 in the shell profile and tell the team.

One question I still can't answer. As skills like this multiply, how should an agent handle "I searched and found nothing"? Right now it quietly returns to its weights. There are tasks where reporting the absence and stopping would be the better move, and I don't yet see how the skill format could express that difference.

Deciding what to delegate to an agent and what to hold down with rules is less a coding problem than an agreement problem. If you need someone to help write that agreement, my contact page is open.

Sources: Chrome for Developers' Modern Web Guidance, Get started, and 15 updates from Google I/O 2026; the GoogleChrome/modern-web-guidance repository README; web.dev's Baseline (all official). The three English block quotes (the definition, the Baseline integration sentence, and the fallback default) were each checked against those documents and the installed SKILL.md at the time of writing, with the source linked next to the quote. Measurement environment: modern-web-guidance 0.0.180, Node 22.22, macOS, a throwaway sandbox directory, measured 2026-08-10. Raw guide list in data/mwg-guide-list.json, the 22 query results in data/mwg-query-probe.json, the probe script in scripts/probe-modern-web-guidance.mjs, figures generated by scripts/chart-modern-web-guidance.py. Twenty-two queries is a probe, not a benchmark, and results shift with query wording. The 27 to 35 point improvements in the README are Google's own published evaluation, not something I reproduced. Category counts follow the tool's own labels; I did not measure accessibility material embedded inside guides filed under other categories.

Your Prompt Tooling Has a Deadline Your Monitoring Can't See

Jangwook Kim — Mon, 10 Aug 2026 00:46:33 +0000

Picture a support-triage feature that's run fine for eight months. Nobody has touched it. Then one Tuesday it starts throwing errors, and the on-call engineer discovers the failing call isn't to a model at all. It's to a small helper endpoint the team used once, during setup, to tidy up a prompt. The vendor announced its removal in a release note back in July. Nobody on the team read release notes in July.

That's the shape of the risk this quarter. Two AI vendors are switching off prompt-management and evaluation tooling, their shutdown dates 15 weeks apart, and the nearer one is seven days from the date of this article.

So we went looking for something specific. If you aren't reading release notes, does anything else tell you? Your logs, your dashboards, your API gateway. Do they warn you?

We checked. They don't.

What's actually being switched off

Five things, from two vendors, on two dates. All of it read straight from the vendors' own pages rather than from summaries.

What goes away	Vendor	Date	Vendor's suggested replacement
Experimental prompt tools API (generate, improve, templatize)	Anthropic	2026-08-17	None stated. Requests return an error after removal.
Legacy Workbench (saved prompts, variables, evals)	Anthropic	2026-08-17	Updated Workbench, but saved prompts/variables/evals aren't supported there
Reusable prompt objects and the v1/prompts API	OpenAI	2026-11-30	Move the prompt content into your application code
Evals dashboard and API	OpenAI	2026-11-30 (read-only from 2026-10-31)	Promptfoo
Agent Builder	OpenAI	2026-11-30	Agents SDK, or ChatGPT Workspace Agents

Two rows deserve more attention than they usually get.

The first is the Anthropic row about saved work. An updated Workbench exists, so it's tempting to read this as "the tool moved." It didn't. Anthropic's note says saved prompts, variables, and evals aren't supported in the updated Workbench, and points you at an export option in the banner and under Organizational Settings. If your prompt library lives there and nobody exports it, it's gone on 17 August.

The second is the OpenAI Evals row, which carries two dates instead of one. Evals go read-only on 31 October, a full month before the shutdown. The first thing your team notices, then, won't be an outage. It'll be a release gate that quietly stops recording new results while continuing to look healthy.

What we ran

Two things, both in a throwaway sandbox.

First we called the live APIs, read-only. We created no prompts, no evals, no agents, and spent nothing on model tokens. The question was simple: what does an ordinary API key see today?

Then we wrote a scanner that reads a codebase and reports every reference to the retiring surfaces, with the deadline and the days remaining attached to each hit. We ran it against a small test repo built for the purpose: four files with problems, one clean file as a control.

What actually happened

Nothing warns you. That's the finding that matters. A long-standing web standard, RFC 8594, defines a Sunset response header for exactly this situation — a server telling clients "this endpoint has an expiry date." Across every request we made, to both vendors, on endpoints both companies have publicly scheduled for removal, not one response carried a Sunset header. Not one carried a Deprecation header either.

Put plainly: your API gateway can't flag this. Your monitoring can't alert on it. A dashboard tracking third-party API health shows green right up until the day the endpoint stops answering. The deadline exists in exactly one place, a documentation page somebody has to remember to open.

One endpoint is already gone on our account. OpenAI's reusable-prompts REST path returned a 404 when we called it. Interpreting that took care, because a 404 can mean two different things, so we also tested a path we knew was live-but-wrong and a path we knew had never existed. OpenAI answers those differently. A real endpoint with a bad ID gives you a descriptive error explaining what it couldn't find; a path that doesn't exist gives you an empty response. The prompts path gave us the empty one, on both the list and the single-item route.

So on the key we used, that surface is unreachable 112 days before its published shutdown date. We're deliberately not claiming OpenAI removed it early for everyone. It could be account-tier gating. It could be a staged rollout, or a surface only ever offered through the SDK rather than as a public REST collection. We can't tell from outside and we're not going to guess. What we can say is that the published date and what our account actually returns don't match, which is a reason to check your own account rather than trust the calendar.

The Evals API, meanwhile, is in perfect health. It answered normally, with no warning of any kind, 112 days before it shuts down.

The scanner found 11 references in the test repo, sorted so the most urgent came first. The two Anthropic hits landed at the top with "7 days left" beside them. The clean control file was correctly left alone. Run in gate mode, the scanner failed the build whenever a deadline fell inside 30 days.

One honest note on accuracy. Of those 11 findings, one was wrong. The scanner flagged a line reading def poll_run(eval_id, run_id) because it matched on eval_id, which there is just a function's parameter name, not a reference to the vendor's service. Call it 9% noise. We left it in rather than tuning it away, because the narrower rule that would suppress it also misses genuine eval_id entries in config files. Treat the output as a triage list, not a defect count.

Can this survive your workflow?

The question worth asking isn't "do we use OpenAI's prompt tooling." It's "would we know if we did." Some concrete cases where the answer is usually no:

Support ticket triage and routing. Built once, works well, sits untouched for a year. Also the most likely place for a stored prompt ID rather than inline prompt text.
Anything with a quality gate before deploy. If a release check writes to OpenAI Evals, the 31 October read-only date turns your gate into decoration a month before anyone sees an error.
Internal automations built by someone who has since moved teams. Agent Builder workflows are the classic case. The person who exported the flow isn't the person who owns the service now.
Prompt libraries kept in a vendor console. The Anthropic Workbench case. Not code, so not in your repo, so no scan finds it. Somebody has to log in and export by 17 August.
Billing and CRM writes driven by an agent. Lower volume, higher blast radius. A silent failure there costs more than a failed chat reply.

Time cost, honestly stated: scanning your repos takes minutes. The migration work behind a hit varies enormously. Moving a stored prompt back into your own codebase is small. Rebuilding an Agent Builder workflow on a different SDK is not, and that one has 112 days on it, which sounds comfortable until you push it through a normal planning cycle.

What to do differently this week

Three things, in this order, because the deadlines aren't equally close.

Today, if you use Anthropic's console: log in to the legacy Workbench and export saved prompts, variables, and evals. There are seven days left and nothing in your codebase will remind you. This is a browser task, not an engineering task.
This week: grep your repositories for pmpt_, eval_id, v1/prompts, v1/evals, agent_builder, and v1/experimental/. Our scanner does exactly this with the dates attached, but the point is the answer, not the tool.
Before 31 October: if any release gate writes to OpenAI Evals, move it or accept that it stops recording results a month before it starts failing loudly.

If the scan comes back empty, stop. That's a legitimate result and it's worth ten minutes to establish.

When to use this, and when to skip it

Worth doing if: you have production code calling either vendor that predates mid-2026; you have a release gate wired to a vendor eval product; anyone on your team built prompts in a vendor console; or you can't answer "which of our services use stored prompt objects" from memory.

Skip it if: you already pass prompt text inline from your own repository, keep evaluations in your own harness, and have never opened either vendor's visual builder. That's a real and increasingly common setup. If it describes you, none of the five deadlines apply. Don't run a migration project to discover you had nothing to migrate.

Skip the scanner specifically if your prompt IDs live in a database or a secrets manager rather than in files. It reads source files. It'll report a clean repo and be wrong.

What this doesn't cover

Worth being blunt about the limits, because they're what make the rest usable.

We could not verify the current state of Anthropic's three experimental endpoints. We tried without credentials and got a 404 back from all three. Then we sent the same request to an endpoint name we invented, and got a byte-identical 404. Anthropic returns the same response to unauthenticated callers whether the route exists or not, so the probe proved nothing. We're reporting it as inconclusive rather than dressing it up.

The scanner is regex over lines. A reference split across two lines, or a prompt ID assembled from string fragments at runtime, gets missed. The ID patterns assume the current pmpt_ and eval_ prefixes. And the deadlines live in the scanner's own rule table, so if a vendor moves a date, somebody updates the table by hand.

None of this measures whether the suggested replacements are any good. OpenAI points Evals users at Promptfoo and Agent Builder users at the Agents SDK. Whether those are equivalent for your use case is a separate question we didn't test.

What Effloow added

The vendor pages tell you what's being removed and when. What they don't tell you is whether anything will warn you, or how to find your own exposure.

Our contribution is three things the primary sources don't contain. A tested answer to whether these endpoints signal their own expiry at runtime: no, on both vendors, with no Sunset or Deprecation header anywhere. A recorded observation that OpenAI's reusable-prompts path is already unreachable on a live account 112 days before its published date, with the control tests that separate "gone" from "wrong ID." And a runnable, dated scanner published with its real output, false positive included.

The full command log, the raw probe responses, the control tests, and the scanner's complete output are in the public lab note.

If deprecation-tracking as an ongoing discipline is the real problem, our audit of OpenAI spend-limit fail-closed behavior covers the adjacent case where a vendor limit surfaces as an error your client mishandles, and our Assistants API sunset port walks a comparable migration end to end.

Want this audited across your services and turned into a dated remediation plan rather than a list of grep hits? That's the work Effloow's Proof Studio packages, and you can start a conversation through our services page.

For your engineers

Everything below is method. It's separated deliberately so the section above stays readable.

Environment. macOS, Python 3 standard library only (urllib.request), no vendor SDKs. Credential read from .env as OPENAI_API_KEY, never logged. Anthropic requests sent unauthenticated; Effloow holds no Anthropic key. All OpenAI requests were GET. No objects created, no model tokens consumed. Sandbox at /tmp, disposable.

Probe results, 2026-08-10:

Request	Status	Body signature
`GET /v1/definitely_not_a_route_effloow` (unrouted control)	404	empty
`GET /v1/vector_stores/vs_effloow_nonexistent` (routed control, bad id)	404	descriptive JSON error
`GET /v1/prompts`	404	empty
`GET /v1/prompts/pmpt_effloow_nonexistent_probe`	404	empty
`GET /v1/evals`	200	`{"object":"list","data":[]}`
`GET /v1/evals/eval_effloow_nonexistent_probe`	404	descriptive JSON error
`GET /v1/models` (health control)	200	model list

Deprecation and Sunset headers: absent on every row above, and on all three unauthenticated Anthropic probes.

Anthropic route-existence probe, and why it failed. POST to /v1/experimental/generate_prompt, /improve_prompt, and /templatize_prompt, unauthenticated, empty JSON body. All three returned 404 {"type":"error","error":{"type":"not_found_error","message":"Not found"}}. The control, /v1/experimental/does_not_exist_effloow, returned the identical body. An unauthenticated 404 carries no signal here. Determining the real state needs a key with Console access.

Scanner design. Five dated rules, each carrying its own deadline, severity, remediation string, and the vendor URL that sets the date. Findings sort by days-remaining ascending, so the nearest deadline is always first. An extension allowlist skips binaries; .git, node_modules, vendor, dist, build, and .venv are excluded. Two output modes, human-readable and --json. A --fail-within-days N flag returns exit code 1 when any finding falls inside the window, which is what makes it usable as a CI gate. A --today override keeps the tool deterministic in tests instead of dependent on the wall clock.

Fixture and run. Five files: a TypeScript module using prompt: {id: "pmpt_..."} plus a raw v1/prompts fetch, a Python Evals client, a Python module calling two Anthropic experimental endpoints, a YAML pipeline config carrying prompt_id, eval_id, and an Agent Builder export reference, and one clean control passing an inline prompt. Result: 11 findings, control file not flagged, both Anthropic hits ranked first at 7 days remaining. --fail-within-days 30 exited 1.

Reproduce. Full source for the probes, the scanner, and the fixture, plus raw responses, are in the lab note. Re-run against your own repository root, and pass --today explicitly if you want reproducible output.

Primary sources, read directly:

OpenAI deprecations: https://developers.openai.com/api/docs/deprecations
Claude Platform API release notes, 17 July 2026 entry: https://platform.claude.com/docs/en/release-notes/api
Claude Console legacy Workbench: https://platform.claude.com/workbench
Updated Claude Workbench: https://platform.claude.com/playground

Vendor-stated dates are theirs. The probe results, the control methodology, the scanner, and its error rate are ours.

How to Self-Host n8n with Docker — AI Workflow Automation Guide 2026

Jangwook Kim — Sun, 09 Aug 2026 11:43:05 +0000

n8n can be used as a managed cloud service or as a self-hosted workflow automation platform. The cost model is different in each mode: n8n Cloud is priced around workflow executions, while a community self-hosted deployment shifts the bill to the server, storage, monitoring, backups, and maintenance you operate yourself.

n8n is a source-available workflow automation platform with native AI capabilities and a visual node editor. The official GitHub README describes it as a fair-code platform with 400+ integrations, custom code support, AI agent workflows, and self-hosting support. That makes it useful when you want AI calls, vector database queries, webhooks, API requests, and human review steps in one workflow surface.

This guide walks through self-hosting n8n with Docker Compose, from initial setup to an AI workflow pattern with Ollama for local LLM inference. It is a source-verified implementation guide, not a claim that Effloow ran this exact stack in production for this article.

The goal is a practical deployment path: PostgreSQL persistence, safe credential handling, Ollama integration for private inference where appropriate, and production hardening steps before you expose the instance to the internet. If you have not settled on how to run those local models yet, our Docker Model Runner vs Ollama comparison weighs the two deployment paths before you wire one into n8n.

Fact-check status: Pricing, release, starter-kit, and integration claims here were checked against official n8n pages on 2026-06-11. Exact hosting costs, competitor bills, workload capacity, and performance numbers vary by provider and workload, so this guide treats them as planning inputs rather than benchmark results.

What Is n8n?

n8n (pronounced "nodemation") is an open-source, source-available workflow automation platform. You connect nodes on a visual canvas to build automations — triggers, data transformations, API calls, AI model invocations — and n8n executes them on schedule or in response to events.

Feature	Detail
GitHub stars	About 192,000 on GitHub when checked on 2026-06-11
Current stable version	2.25.6 in the official release notes when checked on 2026-06-11
License	Sustainable Use License (source-available)
Integrations	400+ integrations according to the official README and starter-kit docs
AI capabilities	LLM chains, AI agents, vector stores, embeddings
Deployment	Docker (self-hosted), n8n Cloud, or npm

Sources checked

n8n pricing for cloud plan structure, execution-based pricing, and self-hosted Business/Enterprise notes.
n8n release notes for stable and beta version status.
n8n Self-hosted AI Starter Kit docs for the included n8n, Ollama, Qdrant, and PostgreSQL components.
n8n GitHub repository for the README description, license model, current star count, and release metadata.

Why n8n stands out in 2026

n8n is strongest when AI is one part of a broader automation: a webhook receives data, an HTTP node enriches it, an AI node classifies or drafts output, a database node stores state, and a human approval step gates sensitive actions.

The platform ships with dedicated nodes for:

LLM chains — connect any model (OpenAI, Anthropic, Google, Ollama) to structured prompts with output parsing
AI agents — autonomous agents with tool use, memory, and multi-step reasoning
Vector stores — Qdrant, Pinecone, Supabase, and more for RAG workflows
Embeddings — generate and query embeddings from multiple providers
Document loaders — ingest PDFs, web pages, and databases into your AI pipelines

If you have used Dify for AI-specific workflows, think of n8n as the broader automation layer. Dify excels at RAG chatbots and prompt engineering. n8n excels at connecting AI to everything else — your CRM, email, databases, APIs, and file systems. When the whole job is retrieval over a large document set, a dedicated engine such as RAGFlow handles chunking and ingestion more thoroughly than n8n's general nodes, and you can still trigger it from an n8n workflow.

n8n's execution model

This matters for understanding the cost advantage. Zapier charges per "task" — every individual action in a workflow counts. A 5-step workflow triggered 1,000 times burns 5,000 tasks. Make.com charges per "operation" — similar granularity.

n8n counts per "execution." One execution = one complete workflow run, regardless of how many nodes it contains. A 20-node workflow that processes 1,000 webhook events uses 1,000 executions, not 20,000.

On self-hosted n8n, there is no execution limit at all. You pay for the server. That is it.

Why Self-Host n8n?

The main reason to self-host is control. You decide where data lives, which model endpoints are allowed, how long execution logs are retained, and which network paths are exposed. Cost can also be lower for some workloads, but only after you include server time, backups, upgrades, monitoring, and incident response.

n8n Cloud pricing snapshot

Plan	Price	Executions/month	Active workflows
Starter	20€/mo, billed annually	2.5K	Unlimited
Pro	50€/mo, billed annually	10K	Unlimited
Business	667€/mo, billed annually	40K	Unlimited
Enterprise	Contact sales	Custom	Unlimited

Source checked: n8n pricing on 2026-06-11. Verify current pricing before purchasing because plan names, annual discounts, included features, and execution quotas can change.

Self-hosted cost model

Component	Monthly cost
VPS or local server	[DATA NOT AVAILABLE]
Domain name, if needed	[DATA NOT AVAILABLE]
SSL certificate (Let's Encrypt)	Free
Backups and monitoring	[DATA NOT AVAILABLE]

Self-hosted Community Edition does not impose the same hosted-plan workflow execution quota, but your actual ceiling is your machine, database, worker configuration, queue mode, and operational discipline. Treat any "$5/month unlimited automations" claim as marketing shorthand unless it is tied to your own measured workload.

When n8n Cloud makes more sense

Self-hosting is not always the right choice. n8n Cloud is worth the premium if:

You need multi-user access control without configuring LDAP/OIDC yourself
You want zero maintenance — no Docker updates, no backups to manage
Your team does not have anyone comfortable with basic server administration
You need enterprise SSO and audit logging out of the box

For solo developers, small teams, and anyone who already manages their own infrastructure, self-hosting can be a strong path. If that describes you, our guide to self-hosting your entire dev stack covers the broader infrastructure context.

Comparison with other platforms

We covered this in detail in our Zapier vs Make vs n8n comparison, but the decision should be based on the current pricing pages and your own workflow counts, not a generic benchmark table:

Question	Why it matters
How many complete workflow runs do you expect per month?	n8n Cloud pricing is based on full workflow executions.
How many steps does each workflow contain?	Some automation tools bill per task or operation, while n8n emphasizes execution-based pricing.
Does sensitive data need to stay inside your network?	Self-hosting gives you the network boundary, but also the security burden.
Who will patch, back up, and monitor the system?	Self-hosting is only cheaper if operations are handled reliably.
Do you need SSO, SAML, LDAP, audit logging, or enterprise support?	These may require n8n Business, Enterprise, or another managed option.

Prerequisites

Before starting, make sure you have:

A server or local machine with at least 2 CPU cores and 2GB RAM (4GB recommended for AI workloads)
Docker Engine 24+ and Docker Compose v2 installed
A domain name (for production with SSL — optional for local development)
Basic comfort with the terminal

Recommended servers

For a dedicated self-hosted n8n instance, we recommend:

Local development: Any machine with Docker installed (Mac, Linux, or Windows with WSL2)
Production VPS: A small Linux VPS with enough RAM for n8n, PostgreSQL, and your execution history. Check current provider pricing directly before committing.
AI workloads with local LLMs: A larger server or a machine with a GPU if you plan to run non-trivial Ollama models alongside n8n.

If you plan to run Ollama alongside n8n for local AI inference, budget at least 8GB of RAM. Small models like Llama 3.2 (3B) run well on 8GB; larger models like Llama 3.1 (70B) need a GPU server. See our Ollama self-hosting guide for model sizing details.

Method 1: n8n Self-Hosted AI Starter Kit (Recommended)

n8n maintains an official Self-Hosted AI Starter Kit that bundles n8n with PostgreSQL, Ollama, and Qdrant. The official docs describe it as a way to get started with self-hosted AI workflows, but also warn that it is for testing and proof-of-concept use unless you customize, secure, and harden it for production.

Step 1: Clone the repository

git clone https://github.com/n8n-io/self-hosted-ai-starter-kit.git
cd self-hosted-ai-starter-kit

Step 2: Configure environment variables

cp .env.example .env

Edit the .env file with secure values:

# .env — CHANGE THESE VALUES
POSTGRES_USER=n8n_user
POSTGRES_PASSWORD=your-secure-password-here
POSTGRES_DB=n8n

N8N_ENCRYPTION_KEY=generate-a-random-32-char-string
N8N_USER_MANAGEMENT_JWT_SECRET=generate-another-random-string
N8N_DEFAULT_BINARY_DATA_MODE=filesystem

# Uncomment for Mac/Apple Silicon running Ollama locally
# OLLAMA_HOST=host.docker.internal:11434

Generate secure random strings for the encryption keys:

openssl rand -hex 32  # Run twice — once for each key

Important: The N8N_ENCRYPTION_KEY encrypts stored credentials. If you lose it, all saved credentials become unreadable. Back it up somewhere safe.

Step 3: Start the stack

Choose the command that matches your hardware:

# NVIDIA GPU (recommended for AI workloads)
docker compose --profile gpu-nvidia up -d

# AMD GPU (Linux only)
docker compose --profile gpu-amd up -d

# CPU only (works everywhere, slower AI inference)
docker compose --profile cpu up -d

# Mac / Apple Silicon (uses host Ollama — install Ollama separately)
docker compose up -d

The first run pulls several Docker images and downloads the Llama 3.2 model (about 2GB). This takes a few minutes depending on your connection speed.

Step 4: Access n8n

Open your browser and navigate to:

http://localhost:5678

You will see the n8n setup screen. Create your admin account — this is the owner account for your instance. Choose a strong password.

What the starter kit includes

The Docker Compose stack runs five services:

Service	Image	Purpose	Port
n8n	`n8nio/n8n:latest`	Workflow automation engine	5678
PostgreSQL	`postgres:16-alpine`	Persistent data storage	5432 (internal)
Ollama	`ollama/ollama:latest`	Local LLM inference	11434
Qdrant	`qdrant/qdrant`	Vector database for RAG	6333
n8n-import	`n8nio/n8n:latest`	One-time demo data import	—

The starter kit includes demo workflow material for understanding n8n's AI capabilities. Treat demo workflow IDs and seed content as examples that may change in the upstream repository.

Method 2: Minimal n8n Setup (Without AI Stack)

If you only need workflow automation without the AI components, a simpler setup with just n8n and PostgreSQL is lighter on resources.

Docker Compose for minimal n8n

Create a docker-compose.yml file:

volumes:
  n8n_storage:
  postgres_storage:

services:
  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - postgres_storage:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -h localhost -U ${POSTGRES_USER} -d ${POSTGRES_DB}']
      interval: 5s
      timeout: 5s
      retries: 10

  n8n:
    image: n8nio/n8n:2.25.6
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_USER=${POSTGRES_USER}
      - DB_POSTGRESDB_PASSWORD=${POSTGRES_PASSWORD}
      - DB_POSTGRESDB_DATABASE=${POSTGRES_DB}
      - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}
      - N8N_USER_MANAGEMENT_JWT_SECRET=${N8N_USER_MANAGEMENT_JWT_SECRET}
      - N8N_DIAGNOSTICS_ENABLED=false
    volumes:
      - n8n_storage:/home/node/.n8n
    depends_on:
      postgres:
        condition: service_healthy

Create the .env file:

POSTGRES_USER=n8n_user
POSTGRES_PASSWORD=your-secure-password
POSTGRES_DB=n8n

N8N_ENCRYPTION_KEY=your-random-encryption-key
N8N_USER_MANAGEMENT_JWT_SECRET=your-random-jwt-secret

Start it:

docker compose up -d

This minimal setup is lighter than the full AI starter kit, but actual RAM use depends on workflow concurrency, execution logging, custom code nodes, and data volume. Measure your own instance before sizing production infrastructure.

Building Your First AI Workflow

Now that n8n is running, let us build a practical AI workflow. We will create an AI-powered content summarizer that:

Receives a URL via webhook
Fetches the web page content
Sends it to an LLM for summarization
Returns the summary as a structured JSON response

Step 1: Create a new workflow

In the n8n editor (http://localhost:5678), click "Add Workflow" in the top-right corner. Name it "AI Content Summarizer."

Step 2: Add a Webhook trigger

Click the + button to add a node
Search for "Webhook" and select it
Set the HTTP Method to POST
Set the path to summarize
Under "Respond," select "Using 'Respond to Webhook' Node" — this lets us return the AI-generated summary

Step 3: Add an HTTP Request node

Add a new node after the Webhook
Search for "HTTP Request"
Set the URL to: {{ $json.body.url }} — this reads the URL from the incoming webhook payload
Set the method to GET
Under "Options," set Response Format to "String" — we want the raw HTML

Step 4: Add an AI chain

Add a "Basic LLM Chain" node
Connect it to the HTTP Request output
Configure the model:
- If using Ollama (from the starter kit): select Ollama Chat Model and choose llama3.2
- If using an API: select OpenAI Chat Model or Anthropic Chat Model and add your API credentials
Set the prompt:

Summarize the following web page content in 3-5 bullet points. Focus on the key facts and actionable information. Return the summary as JSON with the format: {"title": "...", "bullets": ["...", "..."], "word_count": N}

Content:
{{ $json.data }}

Step 5: Add a Respond to Webhook node

Add a "Respond to Webhook" node at the end
Set "Respond With" to "All Incoming Items"

Step 6: Test it

Activate the workflow, then call it:

curl -X POST http://localhost:5678/webhook/summarize \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

You should receive a JSON response with the AI-generated summary.

This basic pattern — trigger → fetch data → AI processing → structured output — is the foundation for most AI automation workflows. From here, you can add error handling, conditional branches, database storage, and notification steps.

Advanced: AI Agent Workflows with Tool Use

n8n's AI Agent node goes beyond simple LLM chains. It creates autonomous agents that can reason, use tools, and take multi-step actions — similar to what you would build with LangGraph or CrewAI in code, but configured visually.

What an AI Agent can do in n8n

An n8n AI Agent node combines:

An LLM (any supported model) for reasoning
Tools that the agent can call: HTTP requests, code execution, database queries, web search
Memory for multi-turn conversations
Output parsers for structured responses

Building a research agent

Here is a workflow pattern for an AI research agent:

Webhook trigger — receives a research question
AI Agent node configured with:
- Model: Ollama (llama3.2) or Claude/GPT-4 via API
- Tools:
  - SerpAPI or Google Search node — for web research
  - HTTP Request tool — for fetching specific pages
  - Code tool — for data processing and calculations
- System prompt: "You are a research assistant. Search the web to answer questions thoroughly. Cite your sources with URLs."
Respond to Webhook — returns the agent's research report

The agent can decide which configured tools to use based on the question and intermediate results. For production, constrain tools carefully and require human approval for actions that modify external systems or expose sensitive data.

Agent tool configuration

To add tools to an AI Agent node:

Click the AI Agent node
Under "Tools," click "Add Tool"
Select from available tool nodes (Calculator, Code, HTTP Request, SerpAPI, etc.)
Each tool gets a name and description that the LLM uses to decide when to invoke it

The key insight is that the tool descriptions matter as much as the system prompt. Write clear, specific descriptions so the agent knows when each tool is appropriate.

Memory for multi-turn conversations

For chatbot-style workflows, add a Window Buffer Memory or Postgres Chat Memory sub-node to the AI Agent. This gives the agent conversation context across multiple interactions.

With PostgreSQL already in your stack, Postgres Chat Memory is the natural choice — sessions persist across n8n restarts and you get full conversation history in your database.

n8n vs Zapier vs Make: When to Choose What

We wrote a detailed comparison of all four platforms. Here is the decision framework:

Choose Zapier if:

You need the largest integration library (7,000+ apps)
Your team is non-technical and needs the simplest interface
Budget is not a primary constraint
You need enterprise compliance certifications

Choose Make.com if:

You need complex multi-branch workflows at scale
You want the best price-to-operations ratio on a managed platform
Your workflows involve heavy data transformation
You do not want to manage infrastructure

Choose n8n (self-hosted) if:

You want AI workflow automation as a first-class feature
You want execution capacity governed by your own infrastructure rather than a hosted-plan quota
Data sovereignty matters — GDPR, client contracts, or privacy requirements
You are comfortable with Docker and basic server management
You want to run local LLMs via Ollama without API costs

Choose n8n Cloud if:

You want n8n's features without managing infrastructure
You need managed scaling and high availability
Your execution volume fits within the tier pricing

Production Deployment Tips

Running n8n locally is one thing. Running it reliably in production requires a few more steps.

Reverse proxy with Caddy

For production deployments, put n8n behind a reverse proxy with automatic SSL. Caddy is the simplest option.

Add Caddy to your docker-compose.yml:

  caddy:
    image: caddy:2-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data

Create a Caddyfile:

n8n.yourdomain.com {
    reverse_proxy n8n:5678
}

Add WEBHOOK_URL=https://n8n.yourdomain.com to your n8n environment variables so webhook URLs resolve correctly.

Backups

Your n8n data lives in two places:

PostgreSQL database — workflow definitions, credentials (encrypted), execution logs
n8n_storage volume — binary data, encryption keys

Automate daily backups:

#!/bin/bash
# backup-n8n.sh
BACKUP_DIR="/backups/n8n/$(date +%Y-%m-%d)"
mkdir -p "$BACKUP_DIR"

# Database backup
docker compose exec -T postgres pg_dump -U n8n_user n8n > "$BACKUP_DIR/n8n-db.sql"

# Volume backup
docker run --rm -v n8n_storage:/data -v "$BACKUP_DIR":/backup alpine \
  tar czf /backup/n8n-storage.tar.gz -C /data .

echo "Backup completed: $BACKUP_DIR"

Run this daily via cron:

0 3 * * * /path/to/backup-n8n.sh

Updating n8n

Pin your n8n version in docker-compose.yml (for example, the current stable version from the n8n release notes) instead of using latest, and update deliberately:

# 1. Backup first
./backup-n8n.sh

# 2. Update the image tag in docker-compose.yml

# 3. Pull and restart
docker compose pull n8n
docker compose up -d n8n

Check the n8n release notes before updating. Major version bumps may include breaking changes to workflow nodes.

Credential encryption

n8n encrypts stored credentials using the N8N_ENCRYPTION_KEY environment variable. If this key is lost or changed, all saved credentials become unreadable.

Best practices:

Generate a strong key: openssl rand -hex 32
Store it outside the server (password manager, encrypted backup)
Never commit .env files to version control
Set the key before creating any credentials — changing it later requires re-entering all credentials

Resource monitoring

For a typical self-hosted n8n instance:

Metric	What to watch
CPU	Code nodes, high workflow concurrency, and local model calls
RAM	n8n process, PostgreSQL, execution history, and Ollama models
Disk I/O	Execution logs, binary data, database writes, and backups
Network	Webhook bursts, API calls, and model-provider latency

If running AI workloads with Ollama, monitor Ollama separately from n8n. Local model memory needs vary by model, quantization, context length, and concurrency.

Five Practical AI Automation Examples

Here are five n8n workflows you can build today with the self-hosted AI stack.

1. Email triage agent

Trigger: New email arrives (IMAP or Gmail node)
AI Agent: Classifies email as urgent/normal/spam, extracts action items
Actions: Labels email, creates tasks in your project management tool, sends Slack alert for urgent items
Model: Llama 3.2 (3B) via Ollama — fast enough for classification tasks

2. RSS-to-summary pipeline

Trigger: RSS Feed trigger (check every hour)
AI Chain: Summarizes each new article in 3 bullet points
Output: Posts summaries to a Slack channel or Discord server
Model: Any LLM — this is a straightforward summarization task

3. Customer support auto-responder

Trigger: Webhook from your support system
AI Agent with tools: Searches knowledge base (Qdrant vector store), drafts a response
Human review: Sends draft to Slack for approval before sending
Model: Claude or GPT-4 via API for quality-sensitive responses

4. Document processor

Trigger: File uploaded to a watched folder or S3 bucket
Processing: Extract text from PDF, split into chunks, generate embeddings
Storage: Store embeddings in Qdrant for later RAG queries
Model: Ollama for embeddings, any LLM for subsequent queries

5. Competitor monitoring dashboard

Trigger: Scheduled (daily)
Web scraping: HTTP Request nodes fetch competitor pages
AI Analysis: Compares changes, identifies new features or pricing updates
Output: Generates a daily digest email with structured comparison data
Model: Llama 3.2 via Ollama for cost-free daily runs

Each of these workflows can run on self-hosted infrastructure, but the right server size depends on actual frequency, data volume, model choice, and concurrency. Price the equivalent managed workflow from current vendor pages before making a migration decision.

Connecting n8n to Local LLMs via Ollama

If you used the AI Starter Kit, Ollama is already running. Here is how to configure n8n to use it.

Adding Ollama as a credential

In any AI node, click the Model parameter
Select "Ollama Chat Model" or "Ollama"
Create new credentials:
- Base URL: http://ollama:11434 (inside Docker network) or http://localhost:11434 (if Ollama runs on the host)
Select your model (e.g., llama3.2)

Pulling additional models

To download more models into your Ollama instance:

docker compose exec ollama ollama pull mistral
docker compose exec ollama ollama pull codellama
docker compose exec ollama ollama pull nomic-embed-text  # For embeddings

For a complete guide on Ollama model selection, hardware requirements, and optimization, see our Ollama + Open WebUI self-hosting guide.

When to use local LLMs vs API models

Use case	Recommended model	Why
Classification, tagging	Small local model or API model	Use local models for private, high-volume internal tasks after checking quality.
Summarization	Local model or API model	Match model quality to the risk of the summary being wrong.
Customer-facing content	Higher-quality reviewed model path	Require human review when customers see the output.
Code generation	Code-capable local or API model	Depends on repository complexity and review process.
Embeddings	Local or hosted embedding model	Choose based on privacy, latency, and retrieval quality.

The hybrid approach works well: use local Ollama models for high-volume, internal tasks and API models for customer-facing or quality-critical outputs.

Frequently Asked Questions

Is n8n really free to self-host?

The n8n Community Edition is free to self-host under the Sustainable Use License. You can use it for your own projects and business automations. The license restricts reselling n8n as a hosted service. For most self-hosting use cases, it is effectively free.

How many workflows can self-hosted n8n handle?

There is no artificial Community Edition workflow count in the same way a hosted quota is expressed, but the practical limit is your server, database, execution logging, queue mode, and workflow design. For high-volume scenarios, load-test your own workflows, scale vertically or horizontally, and consider n8n queue mode with multiple workers.

Can I use n8n with OpenAI and Anthropic APIs?

Yes. n8n has native credential types for OpenAI, Anthropic, Google AI, Hugging Face, and many more. Add your API key in the credentials settings and select the corresponding model in any AI node. You can mix and match providers within the same workflow.

How do I update n8n without losing data?

Pin your version, backup your database and volumes, update the image tag in docker-compose.yml, then run docker compose pull && docker compose up -d. Your workflows, credentials, and execution history are stored in PostgreSQL and survive container updates.

Is self-hosted n8n secure enough for production?

n8n encrypts stored credentials, supports LDAP/OIDC/SAML authentication, and provides role-based access control. For production, add a reverse proxy with SSL (Caddy or Nginx), keep the instance updated, restrict network access to port 443, and use strong encryption keys. The self-hosted version gives you more security control than any cloud platform because you own the network boundary.

Can n8n replace Zapier for a small team?

For many technical-team use cases, yes. n8n covers the same integration patterns: triggers, API calls, data transformation, scheduling, custom code, and AI nodes. Check the n8n integrations page and the current pricing pages for each vendor before deciding whether migration saves money.

How does n8n compare to Dify for AI workflows?

They serve different purposes. Dify is purpose-built for AI: RAG chatbots, prompt engineering, model management. n8n is a general-purpose automation platform with strong AI capabilities added. Use Dify when your entire workflow is AI (chat interfaces, document Q&A). Use n8n when you need AI as part of a broader automation that connects to external services, databases, and APIs.

Production Lessons to Apply

Three practices matter before you treat a self-hosted n8n instance as production infrastructure:

Pin your n8n version. Avoid :latest for production. Pin to a stable release, read the release notes, and update after a backup and staging check.
Set up execution log pruning. n8n stores execution data, and that can grow quickly for high-frequency workflows. Configure pruning and backup retention intentionally.
Treat the AI Starter Kit as a starting point. The official docs say it is for getting started and proof-of-concept work. Harden networking, credentials, data retention, and access control before production use.

What Effloow Added

n8n's docs cover the product feature by feature. What a developer deciding to self-host needs is a path that ends in a working AI workflow plus the judgment about when not to bother. We added:

Two named deployment paths — the AI Starter Kit (Recommended) versus a minimal Compose setup — so you pick by whether you want the bundled Ollama/Qdrant stack or just the automation engine.
A self-host-vs-managed decision (the n8n vs Zapier vs Make section) that says when a hosted tool is the cheaper answer than running your own instance.
Production hardening lessons — version pinning, execution-log pruning, Starter-Kit caveats — pulled to the front so they aren't discovered after an incident.

The value is the runnable path and the operational judgment, not a restatement of the node catalog.

Getting Started: Your First 30 Minutes

Here is a concrete plan for your first session with self-hosted n8n:

Minutes 0-10: Clone the AI Starter Kit, configure .env, run docker compose --profile cpu up -d
Minutes 10-15: Open http://localhost:5678, create your admin account, explore the demo workflow
Minutes 15-25: Build the AI Content Summarizer workflow from the tutorial above
Minutes 25-30: Test the webhook, iterate on the prompt, try switching between Ollama and an API model

After that, explore n8n's workflow templates library — there are hundreds of community-built workflows you can import and customize.

Self-hosting n8n can be a high-leverage move for developers and small teams building AI workflows, especially when data control and customization matter. The honest tradeoff is operational responsibility: you own the server, the upgrade path, the backups, the secrets, and the blast radius of every workflow you enable.

n8n is open-source software maintained by n8n GmbH. This guide covers the self-hosted Community Edition. For managed hosting with support, see n8n Cloud.

Best Open Source AI Tools for Developers: Evidence Matrix

Jangwook Kim — Sun, 09 Aug 2026 11:36:24 +0000

Open source AI tooling is no longer one category. A developer choosing "open source AI tools" in 2026 may mean a coding agent, a workflow builder, a local model runner, a chat interface, or model weights. Treating those as one ranked list creates a bad buying decision: the best tool for editing code is not the best tool for serving a private chat UI, and neither solves workflow orchestration on its own.

This guide addresses the usual listicle problem by using a source-derived matrix. The table below was checked on 2026-06-15 against official project pages and GitHub repository metadata. It is not a hands-on benchmark, not a popularity ranking, and not an affiliate review. Where a public primary source did not support a metric, the cell says [DATA NOT AVAILABLE].

If you are comparing proprietary and open coding agents together, read the AI coding agents comparison. If your main decision is whether to self-host model inference, start with the self-hosting LLMs vs cloud APIs guide.

Source Snapshot

Tool or project	Primary job	Public source signal checked on 2026-06-15	License signal	Use it when	Skip it when
Aider	Terminal coding agent	46,207 GitHub stars	Apache-2.0	You want AI edits that fit a git-first terminal workflow	Your team needs a visual IDE approval surface
Cline	IDE coding agent	63,288 GitHub stars	Apache-2.0	VS Code or JetBrains users need plan/act agent control inside the editor	You want a pure terminal workflow
Dify	LLM app and workflow platform	145,203 GitHub stars	Dify Open Source License, based on Apache 2.0 with additional conditions	You need RAG, workflow design, app packaging, and observability in one platform	You require a simple permissive license with no extra terms
n8n	Workflow automation with AI nodes	192,528 GitHub stars	Fair-code / Sustainable Use style licensing	You need AI connected to SaaS tools, webhooks, databases, and business workflows	Your legal policy only allows OSI-permissive dependencies
Ollama	Local model runner	174,167 GitHub stars	MIT	You want a simple local model server and CLI for open models	You need centralized enterprise governance out of the box
llama.cpp	Low-level local inference engine	116,536 GitHub stars	MIT	You need direct control over inference, quantization, build flags, or embedded deployment	You want the easiest first-time local AI setup
Open WebUI	Self-hosted chat and model interface	141,519 GitHub stars	Repository license not normalized by GitHub API	You want a ChatGPT-style interface for local and remote models	You only need an API endpoint, not a user-facing UI
Qwen3	Open-weight model family	27,300 GitHub stars	License metadata not normalized by GitHub API	You need open model weights for dense and MoE experiments	You need one-click operational tooling rather than model weights
Qwen3.6	Newer Qwen model family	3,563 GitHub stars	Apache-2.0	You want the latest Qwen-family release line to evaluate	You need a mature tooling ecosystem around one stable model family

The star counts above are exact GitHub API snapshots from this review, not claims about quality. GitHub stars can reflect attention, documentation, timing, or community size. They do not prove that a tool is reliable in your stack.

Decision Checklist

Use this checklist before you install anything:

Question	If yes, start with	Why this route is safer
Do you need AI to edit code in an existing repository?	Aider or Cline	They are coding-agent products, not general workflow platforms.
Do you need a private ChatGPT-like interface for a team?	Ollama plus Open WebUI	Ollama serves local models; Open WebUI provides the user interface.
Do you need an internal LLM app with RAG and workflows?	Dify	It is closer to an application platform than a model runner.
Do you need AI to move data across Slack, GitHub, CRM, email, or databases?	n8n	Workflow automation is its core job; AI is one node category in a larger automation graph.
Do you need to tune inference behavior or embed local inference deeply?	llama.cpp	It gives lower-level control than wrapper tools.
Do you need model weights to evaluate locally?	Qwen3 or Qwen3.6	These are model families, so the next decision is runtime, hardware, and license fit.

The practical rule: choose by job first, license second, popularity third. Popularity helps you find documentation and community answers, but it should not decide architecture.

Worked Example: A Private Support Triage Stack

Imagine a small SaaS team wants an internal support triage assistant. The assistant should summarize tickets, draft labels, search internal docs, and avoid sending customer data to a hosted chatbot by default.

Input requirements

Customer tickets stay inside the company network.
Support staff need a browser UI, not a terminal.
Engineers need an automation path that can later write labels back to GitHub or a helpdesk.
The first version should be reversible if model quality is not good enough.

Output architecture

Layer	Tool choice	Reason
Local model runtime	Ollama	It provides the local model server and a simple CLI/API surface.
User interface	Open WebUI	It gives support staff a chat-style interface without making them run terminal commands.
Workflow automation	n8n	It can later connect ticket events, Slack notifications, GitHub issues, and approval steps.
Evaluation model candidate	Qwen3 or Qwen3.6	Open weights can be evaluated locally before any paid hosted model is introduced.
Engineering escape hatch	llama.cpp	If Ollama's defaults are not enough, engineers can move lower in the inference stack.

What this example deliberately does not claim

It does not claim the stack has been run on Effloow customer tickets.
It does not claim Qwen3 or Qwen3.6 will meet your quality bar.
It does not claim a cost saving percentage, because no workload size, hardware cost, or support volume was measured.
It does not claim Open WebUI or n8n is secure by default; network exposure, authentication, secrets handling, and workflow permissions still need review.

This is the kind of decision output a useful comparison should give: a first architecture, the reason each piece is present, and the claims that still require testing.

Coding Agents: Aider vs Cline

Aider and Cline are both open source coding agents, but they optimize for different work habits.

Choose Aider when your workflow already centers on a terminal and git. Its official project description is AI pair programming in your terminal, and its documentation emphasizes git-aware editing. That makes it a good fit for developers who want file edits, diffs, and commits in the same loop they already use.

Choose Cline when your developers want a visible plan/act flow inside an IDE. The official project describes it as an autonomous coding agent available as an SDK, IDE extension, or CLI assistant. Its editor-first posture makes review and approval more accessible for teams that are not comfortable delegating file edits from a terminal-only interface.

Neither tool should be approved for production repositories just because it is popular. The first internal trial should be a throwaway repository with known tests. Give the agent a small issue, record the diff, record commands it ran, and decide whether the review burden went down or up.

App Platforms: Dify vs n8n

Dify and n8n are often grouped together because both can build AI workflows. They should not be evaluated as substitutes without checking the job.

Dify is the better starting point when the artifact is an AI application: a RAG assistant, an agentic workflow, a prompt-managed internal app, or a prototype that needs user-facing behavior and observability in one place. Its repository describes it as a production-ready platform for agentic workflow development.

n8n is the better starting point when the artifact is business automation: watch a webhook, enrich a row, call an LLM, notify a channel, update a CRM, and wait for approval. Its official repository describes a workflow automation platform with native AI capabilities and 400+ integrations.

The license distinction matters. Dify's repository says its license is based on Apache 2.0 with additional conditions. n8n uses fair-code style licensing. That does not make either tool unusable, but it does mean your legal and deployment policies should be checked before embedding them in a commercial product.

Local Inference: Ollama, llama.cpp, and Open WebUI

Ollama is the default first step for many local AI experiments because it turns model running into a simple developer workflow. Its official repository describes the project as a way to get up and running with models including Qwen, Gemma, DeepSeek, gpt-oss, Kimi, and others.

llama.cpp is the lower-level engine choice. Use it when you need build control, inference tuning, platform-specific optimization, or a deeper understanding of how the model is served. If the team only wants to try a local model this afternoon, start with Ollama. If the team needs to ship an optimized local runtime, evaluate llama.cpp directly.

Open WebUI is the interface layer. Its documentation describes a self-hosted AI platform that supports Ollama and OpenAI-compatible APIs and can operate offline. It is useful when the audience is broader than engineers, because a browser interface changes adoption more than another CLI command.

Open Models: What the Tool List Cannot Decide

Qwen3 and Qwen3.6 represent model families, not complete product stacks. Their repositories show active open-weight release lines, but a model page does not answer your production questions by itself.

For teams specifically evaluating open coding models rather than general-purpose model families, the Xiaomi MiMo-V2.5-Pro guide is a narrower next read because it focuses on agentic coding benchmarks, MoE serving requirements, and API access tradeoffs.

Before adopting an open model, run a small eval on your actual workload:

Select 30 to 50 representative prompts from non-confidential or synthetic data.
Define pass/fail rubrics before looking at model output.
Test latency on the hardware you will actually use.
Log failure cases, not only average quality.
Check the model license against your commercial use and redistribution plan.

If you cannot do that evaluation, keep the article-level claim modest: "candidate model family," not "best model."

When to Use / When to Skip

Use this open source stack when you need data control, local experimentation, composability, or the ability to inspect and modify the toolchain. It is especially useful for internal tools, prototypes, regulated data paths, and engineering teams that can own deployment details.

Skip this path when your team needs a managed service-level agreement, vendor support, compliance paperwork, centralized administration, or top frontier-model quality without running infrastructure. In those cases, proprietary hosted tools may be the more honest default.

Also skip any tool whose license you have not checked. "Open source" in casual writing can hide materially different license obligations. MIT and Apache-2.0 are not the same as fair-code or custom source-available licenses.

What Effloow Added

Effloow added three things beyond a normal roundup.

First, this page separates jobs instead of forcing all tools into one ranking. Coding agents, workflow builders, inference runtimes, interfaces, and model weights solve different problems.

Second, the main table uses a dated source snapshot and includes [DATA NOT AVAILABLE] behavior rather than inventing popularity, download, performance, or adoption claims. The exact GitHub star counts are from the GitHub API on 2026-06-15 and should be treated as volatile metadata.

Third, the worked example turns the comparison into an implementation starting point: Ollama for local runtime, Open WebUI for a staff-facing UI, n8n for automation, Qwen models for local evaluation, and llama.cpp as the lower-level escape hatch.

For Your Engineers

Primary sources checked:

Aider GitHub repository: https://github.com/aider-ai/aider
Cline GitHub repository: https://github.com/cline/cline
Dify GitHub repository: https://github.com/langgenius/dify
n8n GitHub repository: https://github.com/n8n-io/n8n
Ollama GitHub repository: https://github.com/ollama/ollama
llama.cpp GitHub repository: https://github.com/ggml-org/llama.cpp
Open WebUI GitHub repository and docs: https://github.com/open-webui/open-webui, https://docs.openwebui.com/
Qwen3 GitHub repository: https://github.com/QwenLM/Qwen3
Qwen3.6 GitHub repository: https://github.com/QwenLM/Qwen3.6

This article does not report Effloow hands-on testing, usage duration, benchmark results, revenue, traffic impact, affiliate conversion, or private customer adoption. Those would require a separate lab run or customer evidence note.

FAQ

What is the best open source AI tool for developers?

There is no single best tool. For code editing, start with Aider or Cline. For local inference, start with Ollama. For a self-hosted chat interface, add Open WebUI. For AI application workflows, evaluate Dify. For business automation, evaluate n8n.

Are GitHub stars a quality score?

No. Stars are a public attention signal, not a reliability score. They help estimate community size and documentation surface, but they do not replace a workload-specific evaluation.

Can this stack replace ChatGPT or Claude?

For some internal workflows, yes. For the strongest general reasoning and coding quality, [DATA NOT AVAILABLE] without a direct eval against your tasks. The honest path is to run a small evaluation before replacing a hosted frontier model.

What should I test first?

Pick one job. For example, ask Aider or Cline to fix a known bug in a throwaway repository, or ask Ollama plus Open WebUI to summarize synthetic support tickets. Keep the first test small enough that a human can review every output.

Best AI DevOps Tools 2026: From CI/CD to Deployment Automation

Jangwook Kim — Sun, 09 Aug 2026 11:35:47 +0000

DevOps teams in 2026 face a paradox: software ships faster than ever, but the infrastructure supporting it has become harder to manage. Microservices multiply. Deployment targets span multiple clouds. Incident alerts pile up at 3 AM. Traditional CI/CD pipelines still work, but they require constant manual tuning — and that tuning increasingly looks like the bottleneck.

AI-powered DevOps tools attack this problem directly. Instead of replacing your pipeline, the best ones embed intelligence into existing workflows: auto-diagnosing failed builds, generating infrastructure-as-code from plain English, triaging incidents before your on-call engineer finishes reading the alert, and predicting deployment risks before you push to production.

This guide compares the leading AI DevOps tools available in 2026, focusing on what actually matters for engineering teams: where the AI adds real value, what it costs, and how it fits into your existing stack. No affiliate links — just a practical comparison based on officially documented features.

For related tooling comparisons, see our guides to the best AI code review tools in 2026 and the best AI coding agents in 2026.

What Changed in AI DevOps for 2026

Before diving into individual tools, it helps to understand three shifts that define the current landscape:

From Copilots to Autonomous Agents

In 2024, AI in DevOps mostly meant chatbots that could answer questions about your pipeline configuration. In 2026, the tools have moved toward autonomous agents — AI systems that can investigate incidents end-to-end, validate their own findings, and take remediation actions with appropriate guardrails. Datadog's Bits AI SRE agent and GitLab's Duo Agent Platform are prime examples of this shift.

Natural Language Infrastructure Management

Generating Terraform from a prompt was a party trick in 2024. In 2026, tools like Spacelift Intent handle the full loop: you describe what you want in plain English, the AI generates the infrastructure code, plans the change, applies it within policy guardrails, and monitors for drift. The key difference is that these systems understand your existing infrastructure state, not just the syntax of IaC languages.

AI-Native Observability

Observability platforms no longer just collect data — they reason about it. When an alert fires, AI agents correlate metrics, logs, and traces automatically, generate root cause hypotheses, and validate them against your runbooks. The on-call engineer gets a conclusion, not a dashboard to stare at.

The Contenders

Tool	Primary Strength	AI Agent	Free Tier	Published Price (verified)
Harness AIDA	CI/CD pipeline intelligence	Yes	Yes (AIDA + 2,000 cloud credits/mo)	Paid plans: contact sales
Amazon Q Developer	AWS-native DevOps assistant	Yes	Yes (perpetual)	$19/user/mo (Pro)
Spacelift Intent	IaC orchestration with AI	Yes	Yes (2 users, no time limit)	Annual contract; quoted by resource count
Datadog Bits AI	Incident response & observability	Yes	Trial only	AI Credits (separate from core usage)
GitLab Duo	Full DevSecOps AI layer	Yes	Limited	$19/user/mo (Duo Pro)
GitHub Actions + Copilot	CI/CD workflow generation	Partial	Yes (Actions free for public repos)	$19/user/mo (Copilot Business)

Harness AIDA: AI-Native CI/CD Platform

Harness was one of the first DevOps platforms to embed AI deeply into its product rather than bolting it on as a chatbot. AIDA (AI Development Assistant) is integrated across every Harness module — CI, CD, Feature Flags, Cloud Cost Management — which means the AI has context about your entire delivery pipeline, not just the step that failed.

What AIDA Actually Does

Pipeline failure analysis. When a build or deployment fails, AIDA analyzes the logs and suggests specific fixes. This goes beyond pattern matching — the AI understands the pipeline configuration, the code changes in the PR, and the execution context to provide targeted remediation advice.

Natural language pipeline generation. You can describe a pipeline in plain English, and AIDA generates the YAML configuration. More usefully, you can ask it to modify existing pipelines — "add a canary deployment stage with 10% traffic split" — and it generates the correct configuration within your existing pipeline structure.

Deployment risk assessment. Harness evaluates deployment risk in real time, recommending canary rollouts or automatic rollback before a problematic release reaches production. This combines change analysis with historical deployment data to flag risky changes.

Policy generation. AIDA generates OPA Rego policies from natural language descriptions, which is genuinely useful for teams that need compliance guardrails but find Rego's syntax intimidating.

Test intelligence. AIDA analyzes code changes in a PR to predict which tests are relevant and identifies flaky tests that have been quietly degrading your pipeline reliability.

Pricing

Harness offers a free tier that includes AIDA and 2,000 CI cloud credits per month (Harness CI subscription docs). Beyond the free tier, Harness moved to a consolidated "Developer 360" model with Free, Essentials, and Enterprise plans, and it does not publish a flat per-developer rate — paid plans are quoted through sales (Harness pricing). Treat any specific seat price you see quoted second-hand as unverified until Harness confirms it for your seat count.

Best For

Teams already using (or evaluating) Harness as their CI/CD platform. AIDA's strength is deep integration across the Harness ecosystem — if you use a different CI/CD platform, you will not get the same level of contextual AI assistance.

Amazon Q Developer: AWS-Native DevOps Intelligence

Amazon Q Developer (the evolution of CodeWhisperer) is AWS's answer to AI-assisted development and operations. While it started as a code completion tool, it has expanded significantly into DevOps territory — particularly for teams running workloads on AWS.

What Q Developer Actually Does

Autonomous development agents. Q Developer can carry out multi-step tasks: implementing features, refactoring code, upgrading dependencies. For DevOps specifically, it generates IaC templates (CloudFormation, CDK) and DevOps scripts from natural language descriptions.

Code transformation at scale. The transformation capability handles large-scale migrations — upgrading Java 8 to 17 or porting .NET Framework apps to .NET 8. It rewrites code, runs tests, and produces a working result. This is particularly relevant for DevOps teams managing legacy codebases that block infrastructure modernization.

Security scanning. Built-in scanning covers OWASP Top 10 vulnerabilities with suggested code fixes — effectively shifting security left into the development workflow rather than catching issues in the deployment pipeline.

AWS console integration. In the AWS Console, you can ask Q to list Lambda functions, generate CLI commands, explain CloudWatch metrics, or troubleshoot deployment issues. It will not execute commands without confirmation, but it eliminates the constant context-switching between documentation and terminal.

Cost insights. Q Developer now includes AWS pricing capabilities — you can ask questions about service costs and get instant estimates, which is useful when designing infrastructure or evaluating architecture changes.

Pricing

The free tier is perpetual and covers individual use with basic code suggestions and security scanning. The Pro tier costs $19 per user per month and includes higher usage limits, organizational features, and administrative controls (Amazon Q Developer pricing). One lifecycle caveat worth checking before you standardize on it: AWS has published an end-of-support timeline for the Q Developer IDE plugins, so confirm the current status on the official product page for your intended use.

Best For

Teams heavily invested in AWS. Q Developer's DevOps value comes primarily from its deep AWS integration — understanding your console, your services, and your infrastructure. If you run a multi-cloud setup, the AWS-specific features become less compelling.

Spacelift Intelligence: AI for Infrastructure Teams

Spacelift occupies a specific niche — infrastructure-as-code orchestration — and its AI features are tightly focused on that domain. The Spacelift Intelligence suite, announced on March 18, 2026, adds AI capabilities to infrastructure management rather than trying to be a general-purpose DevOps AI.

What Spacelift Intelligence Actually Does

Spacelift Intent. This is the headline feature: natural language infrastructure provisioning. You describe what you want — "create a staging environment matching production but with smaller instance sizes" — and Intent carries out the change within your existing policy guardrails. One detail teams often get wrong: by default Intent provisions directly through provider APIs rather than emitting Terraform, with exporting to IaC offered as an optional path (Spacelift Intent docs). The key differentiator is that Intent understands your current infrastructure state, your stacks, and your policies, so it does not act in a vacuum.

Infra Assistant. A conversational AI embedded directly in the Spacelift dashboard that understands your stacks, state files, runs, and configuration. You can ask questions like "which stacks drifted this week" or "why did this run fail" and get contextual answers. It also handles policy creation, drift management, and troubleshooting through conversation.

AI-generated diagnostics. When a run fails, Spacelift Intelligence analyzes the error in context — considering your state, your provider configuration, and your policy rules — to provide targeted diagnostic information. This reduces the time spent reading Terraform error messages, which can be notoriously unhelpful.

Pricing

Spacelift's free "Cloud" tier covers a small team (2 users, no time limit) so you can evaluate the workflow before committing (Spacelift pricing). Paid plans are sold on annual contracts and quoted largely by the number of managed resources rather than a simple per-seat rate — the official Starter+ tier is listed as an annual subscription, and third-party marketplaces report higher tiers running into the low thousands of dollars per month. Because the published figures and reseller estimates do not always agree, get a current quote scoped to your resource count before budgeting.

Best For

Platform engineering teams managing complex infrastructure across multiple providers. Spacelift Intelligence shines when you have a significant IaC footprint and need AI that understands your specific infrastructure context, not just Terraform syntax in general.

Datadog Bits AI: Agentic Observability

Datadog has taken the most aggressive approach to autonomous AI agents in the DevOps space. Bits AI is not a chatbot — it is a suite of three specialized agents (SRE, Dev, Security Analyst) that operate autonomously within your observability data.

What Bits AI Actually Does

Bits AI SRE. This is the standout capability. When an alert fires, the SRE agent autonomously investigates: it analyzes runbooks, correlates telemetry across metrics, logs, and traces, generates root cause hypotheses, validates its own findings, and delivers a conclusion to your collaboration tools (Datadog documents integrations including Slack, Jira, ServiceNow, and GitHub). Datadog markets the agent as completing investigations in minutes and cites customer MTTR reductions (Bits AI SRE); read those as vendor figures, since the baselines behind the percentages are not published.

Bits AI Dev Agent. On the development side, Bits AI can investigate issues flagged in monitoring, suggest code fixes based on the telemetry context, and connect observability insights to the codebase. This bridges the gap between "something is slow in production" and "here is the specific code path causing the latency."

Bits AI Security Analyst. Reviews security signals with full operational context — understanding not just that a vulnerability exists, but whether it is actually exploitable given your infrastructure configuration and traffic patterns.

Natural language querying. You can ask questions about your observability data in plain English — "show me the p99 latency for the checkout service over the last 24 hours" — and get results without writing query syntax.

Pricing

Datadog uses consumption-based pricing across its core platform (per host, per GB of logs, per analyzed span). Bits AI agents, however, are metered separately on an AI Credits model — Datadog lists a credit bundle (on the order of $500/month for 500 credits) with on-demand credits priced per unit above that (Datadog pricing). In practice you need an existing Datadog subscription plus AI Credits to run the agents; there is no standalone free tier for the AI features.

Best For

Teams already using Datadog for observability who want to reduce mean time to resolution (MTTR) for incidents. Bits AI's strength is its access to your full telemetry dataset — metrics, logs, traces, and security signals — which gives it the context needed for accurate root cause analysis. If you use a different observability platform, Bits AI is not available to you. Teams instrumenting LLM-based services that want a self-hostable path instead can start with our self-hosted LLM observability guide.

GitLab Duo: AI Across the DevSecOps Lifecycle

GitLab Duo takes the platform approach: rather than excelling at one specific DevOps function, it embeds AI across the entire DevSecOps lifecycle — from planning to monitoring. The Duo Agent Platform became generally available in January 2026, marking GitLab's shift from individual AI features to a cohesive agent-based system.

What GitLab Duo Actually Does

Agentic chat. Multi-step reasoning using full GitLab context — issues, merge requests, pipelines, and security findings. You can ask Duo to generate code, refactor existing code, write tests, create documentation, or summarize project status, and it draws on the full project context to provide relevant responses.

AI-powered code review. Duo reviews merge requests for bugs, security issues, and code quality, providing inline suggestions. Agentic code review is metered at $0.25 per review; it first shipped in the GitLab 18.8.x series, and GitLab 18.10 opened it to Free-tier teams through the GitLab Credits model.

Pipeline failure root cause analysis. When CI jobs fail, Duo analyzes the failure context and suggests fixes, similar to Harness AIDA but within the GitLab ecosystem.

Vulnerability explanation and resolution. Duo explains security vulnerabilities found by GitLab's scanning tools and suggests specific code fixes, reducing the context-switching between security dashboards and code editors.

MCP client integration. The Duo Agent Platform connects to external systems via Model Context Protocol servers (GitLab's docs cite examples such as Jira, Slack, and Confluence), enabling cross-tool workflows without leaving GitLab. The Duo Agent Platform reached general availability on January 15, 2026.

Pricing

GitLab Duo Pro is available as a $19 per user per month add-on, and Duo Enterprise costs $39 per user per month, adding root cause analysis and vulnerability resolution (GitLab pricing). GitLab also runs an AI credits model: as of 2026, Premium includes $12 in credits per user per month and Ultimate $24, with extra credits listed at $1 each. The included-credit allowance is described as a promotion, so confirm the current allotment before you plan around it.

Best For

Teams using GitLab as their primary DevSecOps platform. Duo's value proposition is breadth — AI assistance at every stage of the pipeline — rather than depth in any single area. If your team already uses GitLab for source control, CI/CD, and security scanning, Duo adds AI capabilities without introducing another tool.

GitHub Actions + Copilot: The Ecosystem Play

GitHub does not have a single "AI DevOps product" in the way Harness or Datadog do. Instead, AI capabilities come from the combination of GitHub Actions (CI/CD) and GitHub Copilot (AI assistant), with Copilot increasingly understanding Actions workflow context.

What the Combination Actually Does

Workflow generation. Copilot can generate GitHub Actions workflow YAML from natural language descriptions. You can describe what your CI pipeline should do, and Copilot produces a working workflow file.

Failed run debugging. When Actions runs fail, Copilot can analyze the failure logs and suggest fixes, including modifications to the workflow configuration or the application code.

Code review with CI context. Copilot's code review capabilities consider CI results when reviewing pull requests, connecting test failures and build issues to the code changes that caused them.

Pricing

GitHub Actions is free for standard public repositories; private repositories on the Free plan include 2,000 minutes per month (Actions billing docs). GitHub Copilot pricing starts at $19 per user per month for Copilot Business, and the Enterprise tier is $39 per user per month (Copilot features).

Best For

Teams using GitHub as their primary development platform who want incremental AI assistance without switching to a dedicated AI DevOps platform. The GitHub combination is less powerful than purpose-built tools like Harness or Datadog for specific use cases, but it requires zero additional tooling if you are already in the GitHub ecosystem.

How to Choose: Decision Framework

The right tool depends on where your DevOps pain is concentrated:

Choose Harness AIDA if...

You need AI deeply integrated into CI/CD pipeline management. Harness excels at pipeline failure analysis, deployment risk assessment, and test intelligence. The free tier makes it accessible for evaluation.

Choose Amazon Q Developer if...

Your infrastructure runs primarily on AWS. Q Developer's value comes from deep AWS integration — console assistance, CloudFormation generation, service-specific troubleshooting. Multi-cloud teams will find the AWS focus limiting.

Choose Spacelift Intelligence if...

Infrastructure-as-code management is your primary bottleneck. Spacelift Intent's natural language provisioning and the Infra Assistant's contextual understanding of your infrastructure state are unmatched for IaC-heavy teams.

Choose Datadog Bits AI if...

Incident response and observability are your biggest time sinks. Bits AI SRE's autonomous investigation capabilities directly reduce MTTR. You need an existing Datadog subscription to use it.

Choose GitLab Duo if...

You want AI assistance across the entire DevSecOps lifecycle without adding new tools. Duo's breadth across planning, coding, CI/CD, security, and monitoring makes it the natural choice for GitLab-native teams.

Choose GitHub Actions + Copilot if...

You are already in the GitHub ecosystem and want to add AI incrementally. This combination is the lowest-friction option, though less powerful than purpose-built alternatives for specific DevOps functions.

Cost Comparison for a 20-Person Team

To make the comparison concrete, here is what each tool costs for a team of 20 developers (where pricing is publicly available):

Tool	Monthly Cost (est.)	Notes
Amazon Q Developer (Pro)	$380	$19/user/mo × 20 (verified list price)
GitLab Duo Pro	$380	$19/user/mo × 20 (add-on to GitLab subscription)
GitLab Duo Enterprise	$780	$39/user/mo × 20 (add-on)
GitHub Copilot Business	$380	$19/user/mo × 20 (Actions usage billed separately)
Harness AIDA	Quote required	Free tier covers evaluation; paid plans priced by sales
Spacelift Intent	Quote required	Annual contract, priced by managed-resource count
Datadog Bits AI	Usage + AI Credits	Core usage (hosts/logs/spans) plus a separate AI Credits bundle

Only the top four rows have list prices you can confirm on a public pricing page today. The bottom three are deliberately left as "quote required" because Harness, Spacelift, and Datadog do not publish a flat per-seat number you can multiply by 20 — anyone who hands you a precise figure for those is estimating. Note also that these are tool-specific costs: most teams combine several (for example, Datadog for observability plus GitLab Duo for CI/CD and code review), so total DevOps tooling spend will exceed any single line item.

What Is Not Ready Yet

Honesty matters more than hype. Here is where AI DevOps tools still fall short in 2026:

Autonomous remediation remains limited. Most tools can diagnose problems and suggest fixes, but actually applying fixes in production without human approval is still uncommon — and for good reason. The "suggest, then confirm" pattern dominates.

Cross-platform intelligence is weak. Each tool's AI works best within its own ecosystem. If you use Harness for CI/CD, Datadog for monitoring, and Spacelift for IaC, no single AI has full context across all three. This is the next frontier.

Small team overhead. For teams under 10 developers, the cost and configuration overhead of these tools may not be justified. A well-maintained GitHub Actions workflow with Copilot assistance covers most small team needs.

When to Use AI DevOps Tools — and When to Skip

The comparison above answers "which tool," but the prior question is whether AI DevOps tooling earns its place on your team at all.

Use it when:

A specific, repetitive failure mode eats real hours — flaky-test triage, reading unhelpful Terraform errors, or 3 AM incident investigation. Match the tool to that one pain point (Harness or GitLab for pipeline failures, Datadog for incident response, Spacelift for IaC) rather than buying breadth you won't use.
You are already standardized on the host platform. Every tool here is strongest inside its own ecosystem, so the value is highest when it sits on top of infrastructure you already run.
You can keep a human in the approval loop. These tools shine at "diagnose and suggest"; treat that as the design point, not a limitation.

Skip it (for now) when:

Your team is under ~10 engineers and your pipeline is stable. The per-seat or usage cost rarely clears the bar versus a tidy GitHub Actions setup with Copilot.
Your stack is genuinely multi-vendor with no dominant platform. No single AI here has cross-tool context, so you would be paying for partial coverage in several places.
You need autonomous production remediation. Unattended fixes are still rare and, for most teams, undesirable. If that is the requirement, none of these fully meet it today.

What Effloow Added

Vendor pages each describe their own tool in isolation. We normalized six of them into one capability matrix and one cost table, and split the comparison into prices you can confirm on a public pricing page today (Amazon Q, GitLab Duo, GitHub Copilot) versus the three that are quote-only by design (Harness, Spacelift, Datadog). Every pricing and feature claim above links to the vendor's own pricing, docs, or announcement page, so you can re-check each number against the source before you buy.

Conclusion

AI DevOps tools in 2026 are genuinely useful — not just demos or marketing promises. Harness AIDA reduces the time engineers spend debugging failed pipelines. Datadog Bits AI SRE investigates incidents autonomously while your on-call engineer is still waking up. Spacelift Intent lets platform teams provision infrastructure from natural language without sacrificing governance.

The key decision is not "should we use AI in DevOps" — it is "where does AI add the most value for our specific bottlenecks." Start with your biggest pain point, pick the tool that addresses it most directly, and evaluate from there.

For more AI tool comparisons, explore our guides to the best AI code review tools in 2026, the best AI coding agents in 2026, and GitHub Copilot agent mode in JetBrains.

Best AI Coding Agents 2026: Comprehensive Comparison & Rankings

Jangwook Kim — Sun, 09 Aug 2026 11:35:11 +0000

AI coding assistants were the story of 2024 and 2025. AI coding agents are the story of 2026.

The distinction matters. An assistant suggests code when you ask. An agent takes a task — "fix this bug," "implement this feature," "migrate this database" — and works through it autonomously: reading files, running tests, creating commits, and opening pull requests. Some agents run in your terminal. Others run in the cloud while you sleep. A few do both.

In February 2026, every major player shipped multi-agent capabilities in the same two-week window: Cursor launched background agents, Windsurf added parallel agent sessions, Claude Code introduced Agent Teams, OpenAI released Codex CLI with Agents SDK support, and Devin enabled parallel sessions. Running multiple agents simultaneously on different parts of a codebase is now table stakes.

But the landscape is noisy. Marketing claims outpace real-world performance, pricing structures range from straightforward to opaque, and benchmark scores only tell part of the story. This guide cuts through the noise with a practical comparison of every major AI coding agent available today — ranked by what actually matters: capabilities, cost efficiency, and how well they handle real development work.

If you are evaluating AI coding assistants (code completion, inline suggestions, chat) rather than autonomous agents, see our comparison of Cursor, Windsurf, and GitHub Copilot. For Copilot's agent mode inside JetBrains IDEs specifically, see our GitHub Copilot agent mode guide.

What Makes a Coding Agent Different from a Coding Assistant?

Before diving into the rankings, it helps to understand what separates an agent from an assistant:

Trait	Coding Assistant	Coding Agent
Interaction	You prompt, it responds	You assign a task, it executes
Scope	Single file or snippet	Multi-file, multi-step workflows
Autonomy	Suggests; you decide	Plans, executes, and iterates
Tool use	Code completion and chat	Terminal commands, file I/O, web browsing, git operations
Feedback loop	Manual	Runs tests, reads errors, self-corrects

The best agents in 2026 combine all of these traits. They read your codebase, form a plan, write code across multiple files, run your test suite, fix failures, and submit the result — all from a single task description.

That said, agent scaffolding matters as much as the underlying model. In a February 2026 benchmark, three different frameworks running identical models scored 17 issues apart on 731 problems. The architecture around the model — how it manages context, when it decides to run tests, how it handles errors — makes a measurable difference.

Quick Comparison: All Major AI Coding Agents

Agent	Type	Best For	SWE-bench Verified	Pricing	Open Source
Claude Code	Terminal agent	Deep reasoning, complex bugs	79.6% (Sonnet 4.6) — 80.8% (Opus 4.6)	$20–$200/mo or API	No
Devin	Cloud agent	Fully autonomous task execution	Not publicly disclosed for current release	$20/mo + ACUs	No
GitHub Copilot	IDE extension + cloud agent	Teams on GitHub workflows	Varies by model	$10–$39/mo	No
Codex CLI	Terminal agent	OpenAI ecosystem, parallel tasks	Varies by model	Included with ChatGPT plans	Yes
Cursor	IDE agent	Visual editing + background agents	Varies by model	$20/mo	No
Windsurf	IDE agent	Parallel multi-agent sessions	Varies by model	$15/mo	No
OpenHands	Cloud/self-hosted agent	Self-hosted, model-agnostic	53.0% (v0.38)	Free (open source) + API costs	Yes
Aider	Terminal agent	Git-native CLI workflows	Varies by model	Free (open source) + API costs	Yes
Kiro	IDE agent	Spec-driven development, AWS	Varies by model	Free preview	No
Google Antigravity	IDE agent	Multi-agent orchestration	76.2%	Free preview	No
Cline	VS Code extension	Flexible, open-source IDE agent	Varies by model	Free (open source) + API costs	Yes

When to Use an AI Coding Agent — and When to Skip

An agent is not always the right tool. It shines when a task is describable, verifiable, and larger than a single edit. It struggles when the work is ambiguous, when correctness is hard to check, or when the cost of a wrong-but-plausible answer is high. Run through this checklist before you assign a job to one.

Reach for an agent when:

The task has a clear finish line — a failing test to make pass, a bug with a reproduction, a feature with acceptance criteria.
The work spans several files or steps that would be tedious to do by hand but are mechanical once scoped.
You have a test suite or a linter the agent can run, so it can check its own work instead of guessing.
You can review the diff before it merges. Agents are fastest when a human still owns the merge button.

Skip the agent (or keep a human in the loop the whole way) when:

The requirements are still fuzzy. An agent will confidently build the wrong thing; write the spec first, or use a spec-driven tool like Kiro.
Correctness is expensive to verify — security-sensitive code, payment logic, schema migrations on production data. Here the review cost can exceed the time the agent saves.
The change is a one-line edit you already understand. Prompting, waiting, and reviewing is slower than just typing it.
The codebase is undocumented and the task needs deep domain judgment the agent cannot infer from the files alone.

A quick way to decide: if you could hand the task to a competent contractor with a paragraph of instructions and a way to check the result, an agent will probably do well. If you'd need a week of onboarding first, it won't.

Situation	Good agent fit	Why
Bug with a failing test	Claude Code, Devin, Copilot agent	Clear success signal the agent can verify
Large mechanical refactor	Aider, Claude Code	Multi-file, but the shape is known
Repeatable backlog tasks	Devin, Copilot coding agent	Well-scoped, runs unattended
Ambiguous "make it better" ask	None — scope it first	No finish line to aim at
Security or payments logic	Human-led, agent-assisted at most	Verification cost is high
Self-hosted / no-vendor-lock need	OpenHands, Cline, Aider	Model-agnostic, run your own infra

Detailed Rankings

1. Claude Code — Best for Deep Reasoning and Complex Tasks

What it is: Anthropic's CLI-based coding agent that operates directly in your terminal with full access to your project files, git history, and shell commands.

Why it ranks first: Claude Code holds a SWE-bench Verified score of 80.8% with the Opus 4.6 model — among the highest of any agent-model combination as of April 2026. But benchmarks aside, Claude Code's real advantage is reasoning depth. It excels at tasks that require understanding large codebases, tracing complex bugs across multiple files, and making architectural decisions.

Key capabilities:

Full agentic loop: reads files, writes code, runs commands, iterates on errors
Agent Teams for multi-agent parallel workflows (each agent gets its own context window)
Available as CLI, desktop app (Mac/Windows), web interface, and IDE extensions (VS Code, JetBrains)
MCP (Model Context Protocol) support for connecting to external tools and data sources
CLAUDE.md configuration files for project-specific instructions

Pricing:

Pro: $20/month (Sonnet 4.6 + limited Opus 4.6 access)
Max: $100/month (5x usage) or $200/month (20x usage) — recommended for Agent Teams
API: Pay-per-token (most flexible, but requires managing costs)

Best for: Senior developers tackling complex debugging, large refactors, and multi-file feature implementation. If the task requires genuine reasoning — not just pattern matching — Claude Code is the current leader.

Limitations: Agent Teams consume tokens fast (roughly 7x a single-agent session for a 3-agent team). The Pro plan can feel limiting for heavy daily use.

For a deep dive on getting the most out of Claude Code, see our complete Claude Code guide.

2. Devin — Most Autonomous Cloud Agent

What it is: Built by Cognition, Devin is a fully autonomous cloud-based coding agent with its own IDE, terminal, and browser. You assign tasks via Slack, a web interface, or integrations with Jira and Linear, and Devin works independently in a cloud environment.

Why it ranks here: Devin is the most "hands-off" option available. It can be given a task description and work entirely without human intervention — reading documentation, writing code, running tests, and opening PRs. Cognition reports a 67% PR merge rate on clearly defined tasks.

Key capabilities:

Interactive Planning: collaboratively scope tasks before execution
Parallel sessions: spin up multiple Devins working on different tasks simultaneously
Devin Wiki: automatically indexes repositories and generates architecture documentation
Integrations with Slack, Jira, Linear, and GitHub
Cloud-based IDE where you can step in and guide at any time

Pricing:

Core: $20/month + ACUs at $2.25 each (~15 minutes of active work per ACU)
Team: $500/month with 250 ACUs included (additional ACUs at $2.00 each)
Enterprise: Custom pricing with VPC deployment and SAML SSO

Best for: Teams with well-defined, repeatable tasks — bug backlogs, migration work, documentation maintenance, and routine feature implementation. Devin works best when success criteria are clear and verifiable.

Limitations: Cost-per-task can escalate quickly on complex work. The ACU billing model means you pay for the agent's compute time regardless of whether the task succeeds. Performance on ambiguous or open-ended tasks is weaker than on clearly scoped ones.

3. GitHub Copilot Coding Agent — Best for GitHub-Native Teams

What it is: GitHub Copilot now includes both an IDE-based agent mode (for interactive coding) and a fully autonomous coding agent (for background task execution). The coding agent takes a GitHub issue, works on it independently, and opens a PR.

Why it ranks here: The coding agent integrates directly into the workflow most teams already use — GitHub Issues and Pull Requests. You assign an issue to Copilot, and it creates a draft PR with the proposed changes. Combined with agentic code review, GitHub is building an end-to-end AI development pipeline.

Key capabilities:

Agent mode in VS Code and JetBrains for interactive multi-step coding
Autonomous coding agent that picks up GitHub issues and creates PRs
Agentic code review that analyzes PRs automatically
GitHub Spark for building apps from natural language (Pro+ and Enterprise)
Semantic code search across repositories
Multi-model support (GPT-4o, Claude, Gemini models)

Pricing:

Free: 2,000 completions + 50 premium requests/month
Pro: $10/month (300 premium requests)
Pro+: $39/month (1,500 premium requests + GitHub Spark)
Business: $19/user/month
Enterprise: $39/user/month

Best for: Teams already embedded in the GitHub ecosystem who want a coding agent that works within their existing issue-tracking and PR workflow. The free tier is generous enough for experimentation.

Limitations: The coding agent works best on well-scoped issues. Complex, multi-step tasks that require deep codebase understanding are better handled by Claude Code or Devin. Premium request limits on lower tiers can be constraining.

4. Codex CLI — Best Terminal Agent in the OpenAI Ecosystem

What it is: OpenAI's open-source terminal-based coding agent, built in Rust. Codex CLI runs locally and brings models like o3 and o4-mini into your terminal workflow. OpenAI also offers a cloud-based Codex service for parallel task execution.

Key capabilities:

Local terminal agent with full file system and git access
Cloud-based Codex for autonomous parallel task execution
Built-in worktrees and cloud environments for isolated agent work
Automations: Codex can work unprompted on routine tasks (issue triage, CI/CD, monitoring)
Open-source CLI with Agents SDK integration
Supports o3, o4-mini, and other OpenAI models

Pricing: Included with ChatGPT Plus, Pro, Business, Edu, and Enterprise plans. API token costs apply for heavy usage.

Best for: Developers already in the OpenAI ecosystem who want a fast, lightweight terminal agent. The Rust-based CLI is notably fast, and the cloud Codex service handles parallel workloads well.

Limitations: Limited to OpenAI models (unlike Aider or OpenHands, which are model-agnostic). Cloud Codex is still in research preview as of April 2026.

For a hands-on setup walkthrough covering safety modes, the sandboxing architecture, and MCP integration, see our Codex CLI deep-dive guide.

5. Cursor — Best IDE Agent with Background Execution

What it is: A standalone AI-powered editor (VS Code fork) that has evolved from a code completion tool into a full agentic development environment. Cursor's Background Agents run tasks autonomously in git worktrees while you continue working.

Key capabilities:

Background Agents: run up to eight agents in parallel on separate git worktrees
Agent mode with multi-file editing and terminal access
Tab completion with context-aware suggestions
Multi-model support (Claude, GPT-4o, Gemini, custom models)
Codebase-aware context using indexing

Pricing:

Hobby: Free (limited; no Background Agents)
Pro: $20/month (Background Agents included)
Business: $40/user/month

Best for: Developers who want agentic capabilities within a visual editor. Background Agents let you delegate tasks while continuing to work in the same IDE — a workflow that terminal-only agents cannot match.

Limitations: Standalone editor means leaving your existing VS Code setup (extensions may not all transfer). Background Agents require Pro tier. Cursor's parent Anysphere crossed $500M ARR in June 2025 and has raised at a multi-billion-dollar valuation since, so the product is well funded — but that scale of spend on model costs leaves the long-term pricing trajectory unclear.

For a detailed comparison of Cursor against other AI IDEs, see our Cursor vs Windsurf vs GitHub Copilot guide.

6. Windsurf — Best for Parallel Multi-Agent Sessions

What it is: Another VS Code fork (formerly Codeium, now part of Cognition AI after acquisition) with deep agentic capabilities through its Cascade system.

Key capabilities:

Cascade: fully agentic workflow engine with multi-step execution
Parallel Multi-Agent Sessions: run up to five agents simultaneously
Arena Mode: blind-test model quality to find the best model for your tasks
Plan Mode: separate planning from code generation for better control
In-IDE local preview for frontend development

Pricing: Starting at $15/month for Pro.

Best for: Developers who want IDE-based agentic coding with an emphasis on parallel execution and model experimentation. Arena Mode is a unique feature for teams evaluating which model works best for their codebase.

Limitations: The Cognition acquisition creates strategic questions — Windsurf and Devin are now under the same parent company. Feature trajectory may shift.

7. OpenHands — Best Open-Source Agent Platform

What it is: An MIT-licensed open-source platform for autonomous coding agents. OpenHands provides a full agentic loop — code writing, terminal commands, web browsing, and GitHub PR creation — all running in sandboxed Docker environments.

Key capabilities:

Model-agnostic: works with Claude, GPT-4o, Gemini, or local models via OpenRouter
Sandboxed Docker execution for safe autonomous operation
GitHub integration: point at an issue, get a PR
Kubernetes support (v1.6.0, March 2026)
Planning Mode beta
Self-hosted or cloud deployment options
Fine-grained access control for enterprise use

Pricing: Free and open-source. You pay only for the LLM API tokens you consume.

Best for: Teams that want full control over their agent infrastructure — self-hosted deployment, model flexibility, and no vendor lock-in. OpenHands is also excellent for contributors who want to understand and modify agent behavior at the code level.

Limitations: Requires more setup than commercial alternatives. Performance depends heavily on the model you choose. SWE-bench score of 53.0% (v0.38) is competitive for open-source but trails commercial leaders.

8. Aider — Best Git-Native Terminal Agent

What it is: An open-source terminal-based AI coding agent with best-in-class git integration. Aider maps your entire codebase, edits files, and commits changes with descriptive messages — all within your existing git workflow.

Key capabilities:

Supports 100+ programming languages
Works with any LLM: Claude, GPT-4o, Gemini, DeepSeek, local models
Automatic repo mapping for codebase understanding
Git-native: stages, commits, and manages changes automatically
Built-in linting and test execution with automatic error fixing
Voice input and in-code annotations for task description

Pricing: Free and open-source. API costs typically run $30–$60/month depending on usage and model choice.

Best for: Senior engineers who live in the terminal and want an agent that fits into existing CLI and git workflows without requiring a new editor or environment.

Limitations: No visual interface — terminal only. The lack of a GUI means less discoverability for newer developers. No built-in background execution (you watch it work in real-time).

9. Kiro — Best for Spec-Driven Development

What it is: Amazon's AI coding IDE that emphasizes specification-driven development. Before writing any code, Kiro generates a detailed spec covering requirements, data models, API endpoints, and task breakdown.

Key capabilities:

Spec-driven workflow: requirements → design → implementation
Agent Hooks: event-driven automations triggered on file save, create, or delete
Powered by Anthropic's Claude Sonnet 4 with Sonnet 3.7 fallback
Native AWS integration (Lambda, CDK, CloudFormation, CodeCatalyst)
MCP Server support
Steering files for project-level coding standards

Pricing: Free during preview. General-availability pricing has not been announced.

Best for: Teams building on AWS who want structured, specification-first development. Kiro's approach suits enterprise workflows where documentation and requirements clarity are as important as the code itself.

Limitations: Still in preview — feature set and pricing may change. Heavily oriented toward AWS; less useful if you are not in the AWS ecosystem.

10. Google Antigravity — Best for Multi-Agent Orchestration

What it is: Google's agent-first IDE (VS Code fork) designed to deploy autonomous agents that plan, execute, and verify tasks across your editor, terminal, and browser.

Key capabilities:

Multi-agent orchestration from day one
Powered by Gemini 3.1 Pro and Gemini 3 Flash
Multi-model support: Claude Sonnet 4.6, Claude Opus 4.6, GPT-OSS-120B
SWE-bench score of 76.2%
Cross-platform: macOS, Windows, Linux

Pricing: Free during public preview with generous Gemini 3 Pro rate limits.

Best for: Developers who want to experiment with multi-agent workflows without immediate cost commitment. The free preview with strong model support makes it an accessible entry point.

Limitations: Still in preview. The long-term pricing model and feature roadmap are not yet finalized. Google has a history of deprecating developer tools — longevity is a legitimate concern.

11. Cline — Best Open-Source VS Code Agent

What it is: An open-source VS Code extension that turns your editor into an agentic coding environment. Cline has passed 5 million installs across VS Code and other editors, and it continues to grow as a flexible, model-agnostic agent.

Key capabilities:

Works within VS Code (no editor switch required)
Model-agnostic: connect any LLM via API
Full agent loop: file editing, terminal commands, browser interaction
Active community and extension ecosystem

Pricing: Free and open-source. Pay only for API tokens.

Best for: VS Code users who want agentic capabilities without switching editors. Cline is particularly popular with developers who want to choose their own model and control costs.

Limitations: As a VS Code extension, it is constrained by VS Code's architecture. Less powerful than standalone agents like Claude Code for complex, multi-step tasks.

How to Choose: Decision Framework

The right agent depends on your workflow, team size, and the type of work you do. Here is a practical decision framework:

By Workflow Type

Terminal-first developers: Claude Code, Codex CLI, or Aider
IDE-first developers: Cursor, Windsurf, Cline, or Kiro
Hands-off delegation: Devin or GitHub Copilot coding agent
Self-hosted / open-source: OpenHands, Aider, or Cline

By Task Complexity

Complex debugging and refactoring: Claude Code (Opus 4.6) — highest reasoning capability
Well-defined, repeatable tasks: Devin — purpose-built for autonomous execution
Quick fixes and feature additions: GitHub Copilot agent mode or Cursor
Spec-driven enterprise work: Kiro — requirements-first approach

By Budget

$0/month: OpenHands, Aider, Cline (open-source; pay only for API tokens), GitHub Copilot Free tier, Google Antigravity preview, Kiro preview
$10–$20/month: GitHub Copilot Pro, Claude Code Pro, Cursor Pro, Windsurf Pro
$39–$200/month: GitHub Copilot Pro+, Claude Code Max
$500+/month: Devin Team, enterprise plans

By Language and Ecosystem

Most agents perform well across mainstream languages (Python, JavaScript/TypeScript, Go, Rust, Java). However:

AWS-heavy projects: Kiro has native AWS service integration
GitHub-centric teams: GitHub Copilot coding agent integrates directly with Issues and PRs
Multi-language polyglot projects: Aider (100+ languages) or Claude Code (strong cross-language reasoning)

The Multi-Agent Reality

The most common setup among experienced developers in 2026 is not one agent — it is two:

An IDE agent for daily work (Cursor, Windsurf, or Copilot) — fast feedback, inline editing, visual context
A terminal or cloud agent for hard problems (Claude Code, Devin, or Codex) — deep reasoning, autonomous execution, complex multi-file changes

This hybrid approach is not about choosing the "best" agent. It is about matching the right tool to the right task. Quick UI fixes do not need Opus 4.6. Complex system refactors do not belong in an inline completion.

For more on how AI is changing the way developers write code, see our guide to vibe coding and our MCP explainer to understand the protocol layer that connects these agents to external tools.

What to Watch in 2026

The AI coding agent space is moving fast. Several developments are worth tracking:

Cost per task is dropping. Devin's price cut from $500/month to $20/month entry was a market signal. Expect continued downward pressure on pricing as competition intensifies.
Multi-agent orchestration is maturing. Running 5–8 agents in parallel is technically possible today, but coordination — avoiding merge conflicts, sharing context, dividing work intelligently — is still primitive. The frameworks that solve orchestration well will pull ahead.
Benchmarks are becoming less reliable. As agents train on SWE-bench-style tasks, scores inflate. Real-world performance on novel, production codebases is a better signal than benchmark numbers alone.
Open-source agents are closing the gap. OpenHands, Aider, and Cline give developers full control and model flexibility. As frontier models become cheaper, the cost advantage of commercial agents narrows.

Final Verdict

If you want the highest raw capability and you are comfortable in the terminal, Claude Code with Opus 4.6 is the current leader in reasoning depth and benchmark performance.

If you want true hands-off autonomy for well-defined tasks, Devin is purpose-built for the job.

If your team lives in GitHub and you want minimal workflow disruption, GitHub Copilot's coding agent fits naturally.

If you want open-source flexibility and self-hosted control, OpenHands is the strongest option.

And if you want the most practical daily driver, the combination of Cursor or Windsurf (for IDE work) plus Claude Code (for hard problems) is what many experienced developers are converging on in 2026.

The best AI coding agent is the one that fits how you actually work. Start with one, measure the impact, and expand from there.

What Effloow Added

A ranking is only as useful as the criteria under it. Here is what we contributed beyond restating each vendor's marketing:

Benchmark claims point to the leaderboard. The SWE-bench Verified scores link to swebench.com, and each agent links to its official source, so you can check a number instead of trusting our order.
The ranking criteria are explicit — capability, cost-per-task, and real-development fit — and where a public number does not exist, we say so (Devin's current-release benchmark) rather than inventing one.
A when-to-use / when-to-skip checklist and a task-to-agent table turn the list into a decision, including the cases where no agent is the right answer.
We flag that benchmarks are getting less reliable as agents train on SWE-bench-style tasks, and push you toward a workflow-keyed choice instead of a single leaderboard position.

Looking for AI tools that help with code review specifically? See our guide to the best AI code review tools. For a deeper look at the difference between agents and assistants, read our AI Agents vs AI Assistants developer guide. And for hands-on tutorials, check out building agents with LangGraph or multi-agent systems with CrewAI.

AI Agent Frameworks Compared 2026: LangGraph vs CrewAI vs OpenAI SDK

Jangwook Kim — Sun, 09 Aug 2026 11:33:57 +0000

Building AI agents is no longer the hard part. Choosing the right framework to build them with — that is where most teams lose weeks.

The landscape shifted dramatically between late 2025 and early 2026. Microsoft merged Semantic Kernel and AutoGen into a single Agent Framework. OpenAI graduated its experimental Swarm project into a production-ready Agents SDK. Google shipped ADK across four languages. LangGraph hit 1.0 and then 1.1. CrewAI reported crossing 2 billion agentic executions. And the original AutoGen community forked into AG2, now positioning itself as a universal agent runtime.

Every framework promises multi-agent orchestration, tool use, and memory. The differences are in the details: how you define agent coordination, what happens when things fail, how much control you get over execution flow, and how painful it is to move from prototype to production.

The "what happens when things fail" part is the one most framework comparisons skip, and it is not a framework property alone — the model underneath has to notice the fault in the first place. We sent three synthetic tool faults drawn from the ToolMisuseBench fault catalog (schema drift, rate limiting, and an error message dressed up as success) to a live OpenAI model and recorded how it reacted, in our ToolMisuseBench recovery reproduction. Read it alongside whatever retry and checkpointing story your chosen framework offers.

This guide compares the six frameworks that matter most right now. No fabricated benchmarks. No synthetic scores. Just architecture, trade-offs, code patterns, and honest recommendations based on what each framework actually does well.

If you are building with these frameworks to create coding agents specifically, see our comparison of terminal AI coding agents in 2026 and our rundown of the best AI coding agents.

Quick Comparison Table

Before diving into each framework, here is how they compare across the dimensions that matter most for production agent systems:

Framework	Primary Language	Multi-Agent	Orchestration Style	State Management	MCP Support	License
LangGraph	Python, TypeScript	Yes	Graph-based (explicit)	Checkpointing, durable execution	Yes	MIT
CrewAI	Python	Yes	Role-based crews + event-driven flows	Session state in flows	Yes	MIT
OpenAI Agents SDK	Python, TypeScript	Yes	Handoffs + agents-as-tools	Sessions (built-in)	Yes	MIT
Microsoft Agent Framework	Python, C#/.NET	Yes	Workflows + chat patterns	Session-based, checkpointing	Yes	MIT
Google ADK	Python, Java, Go, TypeScript	Yes	Workflow agents + LLM routing	Session-based	Yes	Apache 2.0
AG2	Python	Yes	GroupChat + event-driven	Stateful runtime	Yes	Apache 2.0

LangGraph (by LangChain)

LangGraph is the agent orchestration layer built on top of LangChain. While LangChain provides the building blocks — model abstractions, tool definitions, retrieval chains — LangGraph adds the execution graph that determines how agents coordinate, retry, and persist state.

Architecture

LangGraph models agent workflows as directed graphs. Nodes are functions (often wrapping LLM calls or tool executions). Edges define transitions between nodes, and conditional edges let you route execution based on the current state. This is explicit orchestration: you draw the flow, and LangGraph executes it.

from langgraph.graph import StateGraph, START, END
from typing import TypedDict

class AgentState(TypedDict):
    messages: list
    next_agent: str

graph = StateGraph(AgentState)
graph.add_node("researcher", research_node)
graph.add_node("writer", writer_node)
graph.add_node("reviewer", reviewer_node)

graph.add_edge(START, "researcher")
graph.add_conditional_edges(
    "researcher",
    route_after_research,
    {"needs_writing": "writer", "done": END}
)
graph.add_edge("writer", "reviewer")
graph.add_conditional_edges(
    "reviewer",
    route_after_review,
    {"approved": END, "revision_needed": "writer"}
)

app = graph.compile()

What LangGraph Does Well

Durable execution. LangGraph persists agent state at every node transition. If a process crashes midway through a ten-step workflow, it resumes from the last checkpoint — not from scratch. This matters for long-running agents that interact with external systems.

Human-in-the-loop. The interrupt mechanism lets you pause execution at any node, present the state to a human, collect input, and resume. This is built into the graph execution model, not bolted on.

Visibility. The graph structure makes it straightforward to visualize and debug agent workflows. You can see exactly which node executed, what state was passed, and where things diverged.

Type safety (v1.1). The March 2026 v1.1 release added type-safe streaming and invoke with Pydantic model coercion, catching state shape errors at development time rather than production.

Where LangGraph Falls Short

Verbosity. Simple agent patterns require significantly more code in LangGraph than in frameworks designed for rapid prototyping. A two-agent handoff that takes 15 lines in OpenAI Agents SDK can take 50+ lines in LangGraph.

LangChain coupling. While you can use LangGraph without the full LangChain stack, the documentation and ecosystem strongly assume you are using LangChain's model abstractions, prompt templates, and tool definitions. Breaking away from that path requires more effort than the docs suggest.

Learning curve. Thinking in graphs is not natural for developers coming from imperative programming. The mental model of state flowing through nodes with conditional edges takes time to internalize.

Best For

Teams building production agent systems where reliability, observability, and fine-grained control over execution flow are the primary concerns. Particularly strong for workflows that need durable execution and human oversight — compliance workflows, multi-step data pipelines, and enterprise automation. For a hands-on multi-agent walkthrough in code, see our multi-agent Python tutorial with the OpenAI Agents SDK.

CrewAI

CrewAI takes a different approach: instead of graphs, you think in terms of crews — teams of role-playing AI agents that collaborate on tasks. The mental model is closer to how human teams work: you define agents with roles, assign them tasks, and let the framework handle coordination.

Architecture

CrewAI has two layers. Crews are teams of autonomous agents that can delegate to each other and make dynamic decisions. Flows are the enterprise orchestration layer — event-driven workflows that can contain Crews as steps, with explicit state management and conditional branching.

from crewai import Agent, Task, Crew, Process

researcher = Agent(
    role="Senior Research Analyst",
    goal="Find comprehensive data on AI framework adoption",
    backstory="You are an expert at analyzing technology trends...",
    tools=[web_search, arxiv_search],
    llm="anthropic/claude-sonnet-4-6"
)

writer = Agent(
    role="Technical Writer",
    goal="Create clear, accurate technical comparisons",
    backstory="You specialize in developer-facing content...",
    llm="anthropic/claude-sonnet-4-6"
)

research_task = Task(
    description="Research the top AI agent frameworks...",
    expected_output="A structured report with findings",
    agent=researcher
)

writing_task = Task(
    description="Write a comparison article based on research",
    expected_output="A polished article draft",
    agent=writer,
    context=[research_task]
)

crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, writing_task],
    process=Process.sequential
)

result = crew.kickoff()

What CrewAI Does Well

Speed to prototype. CrewAI has the fastest time-to-working-demo of any framework in this comparison. The role-based abstraction is intuitive — you describe what each agent does, and the framework handles the coordination plumbing.

Task delegation. Agents can autonomously decide to delegate subtasks to other agents in the crew. This emergent collaboration is useful for complex research and analysis workflows where the optimal task decomposition is not known in advance.

NVIDIA integration. The 2026 NemoClaw partnership adds infrastructure-level policy enforcement for enterprise deployments — rate limiting, content filtering, and audit logging at the framework level.

Adoption. CrewAI reports over 2 billion agentic executions and adoption across a large share of the Fortune 500, backed by a big community and an extensive ecosystem of pre-built tools and integrations. Those are the vendor's own figures, so read them as adoption signal rather than an independent benchmark.

Where CrewAI Falls Short

Limited control over execution. The role-playing abstraction that makes CrewAI fast to prototype can become a liability in production. When agents make autonomous delegation decisions, the execution path becomes less predictable. Debugging why an agent delegated a task incorrectly is harder than tracing a graph edge.

Python only. No official TypeScript or JavaScript SDK. If your stack is Node.js-based, CrewAI is not an option without running a Python service alongside your application.

Flows complexity. The Flows layer adds enterprise-grade control but introduces a second mental model on top of Crews. Teams often end up needing both, which increases the cognitive overhead.

Best For

Teams that want to prototype multi-agent workflows quickly and value intuitive abstractions over fine-grained control. Strong fit for research pipelines, content generation workflows, and analysis tasks where agent autonomy is a feature rather than a risk. Our CrewAI multi-agent tutorial walks through building a complete system from scratch.

OpenAI Agents SDK

The OpenAI Agents SDK is the production evolution of the experimental Swarm framework. It takes a deliberately minimalist approach: three core primitives — agents, handoffs, and guardrails — composed using plain Python.

Architecture

Agents SDK is built around the idea that agent orchestration should feel like writing normal Python code. Agents are objects with instructions and tools. Handoffs transfer control between agents. Guardrails validate inputs and outputs. Everything else is just Python.

from agents import Agent, Runner, handoff, tool

@tool
def search_docs(query: str) -> str:
    """Search internal documentation."""
    return doc_store.search(query)

@tool
def execute_sql(query: str) -> str:
    """Run a read-only SQL query."""
    return db.execute_readonly(query)

triage_agent = Agent(
    name="Triage",
    instructions="Route the user to the right specialist agent.",
    handoffs=[
        handoff(docs_agent, "Questions about documentation"),
        handoff(data_agent, "Questions about data or metrics"),
    ]
)

docs_agent = Agent(
    name="Docs Specialist",
    instructions="Answer questions using internal documentation.",
    tools=[search_docs]
)

data_agent = Agent(
    name="Data Analyst",
    instructions="Answer data questions using SQL.",
    tools=[execute_sql]
)

result = Runner.run(triage_agent, "What were our Q1 conversion rates?")

What OpenAI Agents SDK Does Well

Simplicity. The three-primitive model (agents, handoffs, guardrails) is the simplest mental model in this comparison. There are no graphs to draw, no roles to define, no workflow engines to configure. You compose agents with Python.

Tool integration. Any Python function becomes a tool with the @tool decorator. Schema generation and Pydantic validation are automatic. MCP server integration is built in.

Tracing. Built-in tracing for every agent run — visualize handoff chains, tool calls, and guardrail checks. The traces feed directly into OpenAI's evaluation and fine-tuning tools.

Guardrails. Input and output validation is a first-class concept, not an afterthought. You can define guardrails that reject, modify, or flag agent inputs and outputs before they reach the user.

Where OpenAI Agents SDK Falls Short

OpenAI-centric. While technically model-agnostic (you can plug in other providers), the SDK is optimized for OpenAI models. Tracing, evaluation, and fine-tuning tools all assume the OpenAI ecosystem. Using it with Anthropic or open-source models means losing some of the tighter integrations.

Limited orchestration patterns. The handoff model is elegant for linear delegation chains but becomes awkward for complex multi-agent patterns like parallel execution, voting, or iterative refinement loops. You end up building those patterns yourself in Python.

No durable execution. Unlike LangGraph, there is no built-in checkpointing or crash recovery. If a long-running agent process dies, you restart from the beginning. The Sessions feature adds persistence within a run but does not survive process crashes.

Best For

Teams that value simplicity and are building within the OpenAI ecosystem. Excellent for customer-facing agent applications where the triage-and-handoff pattern dominates — support bots, internal tools, and conversational agents that route to specialists. See our OpenAI Agents SDK multi-agent tutorial for a working example.

Microsoft Agent Framework

Microsoft Agent Framework is the result of merging Semantic Kernel and AutoGen into a single SDK. It combines Semantic Kernel's enterprise features — type safety, middleware, telemetry, extensive model support — with AutoGen's multi-agent conversation patterns.

Architecture

The framework offers two primary orchestration patterns. Chat-based patterns (inherited from AutoGen) put multiple agents in a conversation where a selector determines who speaks next. Workflows (inherited from Semantic Kernel) provide explicit, typed execution paths with checkpointing and human-in-the-loop support.

// C# example — Microsoft Agent Framework
using Microsoft.AgentFramework;

var researcher = new ChatCompletionAgent
{
    Name = "Researcher",
    Instructions = "Research topics thoroughly using available tools.",
    Kernel = kernel,
    Tools = [searchTool, webScrapeTool]
};

var analyst = new ChatCompletionAgent
{
    Name = "Analyst",
    Instructions = "Analyze research findings and produce insights.",
    Kernel = kernel
};

var groupChat = new GroupChat([researcher, analyst])
{
    SelectionStrategy = new RoundRobinSelectionStrategy(),
    TerminationStrategy = new MaxMessageTerminationStrategy(10)
};

await foreach (var message in groupChat.InvokeAsync())
{
    Console.WriteLine($"{message.AuthorName}: {message.Content}");
}

What Microsoft Agent Framework Does Well

Enterprise integration. Deep integration with Azure, Microsoft 365, and the broader Microsoft ecosystem. If your organization runs on Azure, the deployment story is seamless — Azure AI Foundry, Cosmos DB for state, Application Insights for telemetry.

C#/.NET first-class support. The only major framework in this comparison with production-quality C#/.NET support. Python support is at parity for GA features, but the C# experience is where the framework shines.

Typed workflows. The workflow system provides compile-time type checking for agent state, catching errors that other frameworks only surface at runtime.

Checkpointing. Long-running workflows can checkpoint state and resume after process restarts, similar to LangGraph's durable execution but with tighter integration into Azure's persistence layer.

Where Microsoft Agent Framework Falls Short

Complexity from merger. The merge of Semantic Kernel and AutoGen happened quickly, and while the framework reached 1.0 GA in February 2026, some teams report that the two lineages still surface as overlapping patterns and APIs. Chat patterns, workflows, or a hybrid? The decision tree for choosing the right approach is not always clear.

Migration overhead. Teams already using Semantic Kernel or AutoGen face a migration path that is well-documented but non-trivial. The mental models are similar but the APIs differ enough to require real engineering time.

Best For

Teams building enterprise agent systems in the Microsoft/Azure ecosystem, especially those using C#/.NET. Strong fit for organizations that need deep integration with Microsoft 365, Azure AI services, and enterprise identity/compliance infrastructure.

Google ADK (Agent Development Kit)

Google ADK is the newest entrant, designed to make agent development feel like software development. It reached 1.0 across Python, TypeScript, Java, and Go in 2026 — the broadest language support of any framework.

Architecture

ADK uses a tree-based agent hierarchy. A root agent delegates to sub-agents, which can be either LLM-powered agents or workflow agents (Sequential, Parallel, Loop). The framework supports both explicit workflow orchestration and LLM-driven dynamic routing.

from google.adk import Agent, SequentialAgent, ParallelAgent

research_agent = Agent(
    name="researcher",
    model="gemini-2.5-pro",
    instruction="Research the given topic thoroughly.",
    tools=[google_search, url_context]
)

fact_check_agent = Agent(
    name="fact_checker",
    model="gemini-2.5-pro",
    instruction="Verify claims against authoritative sources.",
    tools=[google_search]
)

writing_agent = Agent(
    name="writer",
    model="gemini-2.5-pro",
    instruction="Write a clear, accurate article from verified research.",
)

pipeline = SequentialAgent(
    name="content_pipeline",
    sub_agents=[
        ParallelAgent(
            name="research_phase",
            sub_agents=[research_agent, fact_check_agent]
        ),
        writing_agent
    ]
)

What Google ADK Does Well

Language breadth. Python, TypeScript, Java, and Go — all at 1.0. No other framework covers this range. Java and Go support opens the door for backend teams that would otherwise be excluded from the agent framework ecosystem.

Workflow agents as primitives. Sequential, Parallel, and Loop agents are first-class constructs, not library utilities. Composing them into complex workflows is natural and type-safe.

Deployment flexibility. Agents can run locally, deploy to Vertex AI Agent Engine for managed scaling, or containerize with Docker/Cloud Run. The same agent definition works across all deployment targets.

Built-in tools. Google Maps, URL context fetching, code execution (container-based and Vertex AI-based) — practical tools that other frameworks leave to third-party integrations.

Where Google ADK Falls Short

Google ecosystem bias. While officially model-agnostic, ADK is optimized for Gemini. Using non-Google models requires more configuration and loses some of the tighter integrations (evaluation, deployment to Agent Engine).

Newer ecosystem. The community and third-party tooling are smaller than LangGraph or CrewAI. Finding solutions to edge cases often means reading source code rather than Stack Overflow answers.

Documentation gaps. As a 1.0 framework across four languages, the documentation is comprehensive for Python but thinner for Java, Go, and TypeScript.

Best For

Teams that need multi-language support (especially Java or Go), are building on Google Cloud, or want a structured approach to agent orchestration with explicit workflow primitives. Good fit for organizations with polyglot backend teams.

AG2 (Formerly AutoGen)

AG2 is the community-driven fork of Microsoft's original AutoGen, led by the original creators after they departed Microsoft in late 2024. While Microsoft took AutoGen's ideas into Agent Framework, AG2 rearchitected the core with an event-driven, async-first design.

Architecture

AG2's primary coordination pattern is GroupChat: multiple agents in a shared conversation where a selector (round-robin, LLM-based, or custom) determines who speaks next. The framework also supports direct agent-to-agent messaging and nested conversations.

from ag2 import ConversableAgent, GroupChat, GroupChatManager

coder = ConversableAgent(
    name="Coder",
    system_message="Write clean Python code to solve problems.",
    llm_config=llm_config
)

reviewer = ConversableAgent(
    name="Reviewer",
    system_message="Review code for bugs and improvements.",
    llm_config=llm_config
)

executor = ConversableAgent(
    name="Executor",
    system_message="Execute code and report results.",
    code_execution_config={"work_dir": "workspace"}
)

group_chat = GroupChat(
    agents=[coder, reviewer, executor],
    messages=[],
    max_round=10,
    speaker_selection_method="auto"
)

manager = GroupChatManager(groupchat=group_chat, llm_config=llm_config)
coder.initiate_chat(manager, message="Build a web scraper for...")

What AG2 Does Well

Framework interoperability. AG2's standout feature is its ability to connect agents from different frameworks — AG2, Google ADK, OpenAI, and LangChain agents — into a single team. This is valuable for organizations that have already invested in multiple frameworks.

Conversational patterns. The GroupChat model is natural for problems that benefit from multi-perspective discussion — code review, brainstorming, and iterative refinement. Agents genuinely build on each other's outputs in ways that pipeline architectures cannot replicate.

Code execution. Built-in sandboxed code execution with Docker support. Agents can write, execute, and iterate on code within the conversation flow.

Community-driven. Rapid iteration without enterprise bureaucracy. New features and integrations ship faster than in corporate-backed alternatives.

Where AG2 Falls Short

Enterprise readiness. AG2 lacks the enterprise features (compliance, audit logging, managed deployment) that Microsoft moved into Agent Framework. Organizations with strict compliance requirements may find gaps.

Conversation management. GroupChat conversations can become long and expensive as agents discuss back and forth. Token costs scale with conversation length, and managing context windows across many agents requires careful configuration.

Branding confusion. The AutoGen/AG2/Microsoft Agent Framework split creates confusion. Documentation from 2024-2025 may reference APIs that exist in neither the current AG2 nor the Microsoft fork.

Best For

Teams doing research, rapid experimentation, or building systems that need to combine agents from multiple frameworks. Strong fit for code generation workflows where iterative discussion between agents (write, review, execute, refine) produces better results than single-pass pipelines.

How to Choose: Decision Framework

The right framework depends on your constraints, not on which one is "best" in the abstract. Use these questions to narrow down:

What language does your team use?

C#/.NET → Microsoft Agent Framework (only serious option)
Java or Go → Google ADK (only option with 1.0 support)
Python → All six frameworks are available
TypeScript → LangGraph, OpenAI Agents SDK, or Google ADK

How much control do you need over execution?

Maximum control → LangGraph (explicit graphs, durable execution)
Moderate control → Google ADK (workflow agents) or Microsoft Agent Framework (typed workflows)
Minimal control / maximum speed → CrewAI (role-based) or OpenAI Agents SDK (handoffs)

What is your cloud platform?

Azure → Microsoft Agent Framework (deep integration)
Google Cloud → Google ADK (Vertex AI Agent Engine)
AWS or cloud-agnostic → LangGraph, CrewAI, or OpenAI Agents SDK

What orchestration pattern fits your use case?

Linear delegation (triage → specialist) → OpenAI Agents SDK
Explicit multi-step pipelines → LangGraph or Google ADK
Autonomous team collaboration → CrewAI
Multi-perspective discussion → AG2
Enterprise workflows with checkpointing → LangGraph or Microsoft Agent Framework

Do you need to combine multiple frameworks?

Yes → AG2 (universal agent interoperability)
No → Choose the best single framework for your constraints

When to skip an agent framework entirely

Not every problem needs one of these. Reach for a framework only when the coordination itself is the hard part. Skip it when:

Your task is a single LLM call with one or two tools. A plain SDK call plus a function wrapper is less code, fewer dependencies, and easier to debug than a graph or a crew. Frameworks earn their weight at three or more coordinated steps, not at one.
The work is a fixed pipeline with no branching. If step A always feeds step B with no runtime routing, a short script beats an orchestration layer that exists to make branching manageable.
Governance and audit are the real requirement. If you need to assign work, enforce policy, and keep an audit trail across many agents, that is an orchestration-platform concern, not a code-library one. We describe that split in the "What We Use at Effloow" section below.

The failure mode to avoid is adopting a framework for the label rather than the coordination problem. A one-agent task wrapped in a multi-agent framework carries all the abstraction cost and none of the benefit.

Framework Maturity and Ecosystem

Understanding where each framework sits in its lifecycle helps set expectations. Star counts move daily and reveal little about production fitness, so the more useful signals are release maturity and how the vendor positions the project. Check the linked repository for the current version and adoption before you commit:

Framework	First Release	Release Maturity	License	Repository
LangGraph	Jan 2024	1.x stable	MIT	langchain-ai/langgraph
CrewAI	Nov 2023	1.x stable	MIT	crewAIInc/crewAI
OpenAI Agents SDK	Mar 2025	Production, pre-1.0	MIT	openai/openai-agents-python
Microsoft Agent Framework	Oct 2025	1.0 GA	MIT	microsoft/agent-framework
Google ADK	Apr 2025	1.0 GA	Apache 2.0	google/adk-python
AG2	Late 2024	Production, pre-1.0	Apache 2.0	ag2ai/ag2

A pre-1.0 label here (OpenAI Agents SDK, AG2) is not a warning — both vendors describe their SDK as production-ready — but it does mean the public API can still change between minor releases, so pin your version.

MCP Support Across Frameworks

Model Context Protocol (MCP) has become the standard for connecting agents to external tools and data sources. All six frameworks now support MCP, but the depth of integration varies. For a worked example of how tool access changes an agent's behavior in practice, see our proof run on agentic web search and context control.

LangGraph: MCP tools integrate as standard LangChain tools via the MCP adapter. Full support for both stdio and Streamable HTTP transports.
CrewAI: MCP server tools can be assigned to agents alongside native CrewAI tools.
OpenAI Agents SDK: Built-in MCP server tool integration — declare MCP servers in agent configuration and tools are automatically discovered.
Microsoft Agent Framework: MCP is a first-class feature — MCPStdioTool and MCPStreamableHTTPTool connect to MCP servers natively, and agents can themselves be exposed as MCP servers.
Google ADK: MCP tool integration available across all language SDKs.
AG2: MCP support via the interoperability layer.

What We Use at Effloow

At Effloow, our agent infrastructure uses a different approach entirely — we built Paperclip, a governance-first agent orchestration platform that coordinates multiple AI agents (including the one writing this article) through a task-based system rather than a framework-based one. This is a fundamentally different architecture from any of the frameworks compared above: instead of agents calling other agents through code, a control plane assigns work, manages state, and enforces policies.

We mention this not as a recommendation but for transparency. The frameworks in this guide are the right tools when you are building agent logic in application code. Paperclip solves a different problem — orchestrating agents as workers in a company-like structure.

Conclusion

The AI agent framework landscape in 2026 is mature enough that there is no wrong choice among the top options — only choices that fit your constraints better or worse.

If you are starting fresh with Python and want maximum control, LangGraph is the most battle-tested option. If you want to move fast and value intuitive abstractions, CrewAI gets you to a working prototype fastest. If simplicity is your priority and you are in the OpenAI ecosystem, the Agents SDK is hard to beat. If your team is C#/.NET or deep in Azure, Microsoft Agent Framework is the clear choice now that it has reached GA. If you need multi-language support, Google ADK is the only framework that covers Python, TypeScript, Java, and Go at 1.0. And if you need to stitch together agents from multiple frameworks, AG2 is uniquely positioned for that.

The one thing that does not work: choosing a framework based on hype or GitHub stars alone. Match the framework to your team's language, your cloud platform, your orchestration needs, and the level of control you need. Everything else is noise.

For more on the AI tools and agents we cover, see our comparison of Cursor, Windsurf, and GitHub Copilot, our guide to the best AI code review tools, and our guide to what vibe coding actually means.

What Effloow Added

Each of these six frameworks has its own docs that explain it in isolation. None tells you which one to pick. The added value here is the cross-framework decision:

A single comparison axis across all six — orchestration model, language support, license, and MCP support — with each framework's repository linked so the claims are checkable, not asserted.
A constraint-driven decision framework that maps a real situation (your language, your cloud, your control needs) to a specific framework, instead of declaring a single winner.
An explicit skip test — when a plain SDK call, a fixed script, or a governance platform is the better answer than any framework here.
No star-count theater. We deliberately left GitHub stars out of the maturity table, because they move daily and say nothing about production fitness; release maturity and license are the signals that actually inform a build decision.

The value is the side-by-side judgment with sources attached, not six feature lists reprinted.