DEV Community: directree

Next.js leads our crawl of 10,242 software websites

Henrik Åberg — Tue, 15 Sep 2026 08:00:54 +0000

Originally published on directree Research. Data as of 15 September 2026.

We detected Next.js on 3,863 public homepages in directree's September 2026 dataset.

That is 76.3% of the 5,060 websites where our scanner identified at least one framework, and 37.7% of all 10,242 websites with a recorded stack check.

We found Next.js on more than half of the checked websites in both the AI Image & Video Generation and Video & Audio categories. Across the full sample, we could not identify a framework on roughly half of the websites.

What we checked

We examined the latest stored technology observations for published directree listings, collected between 6 and 15 September 2026. Each of the 10,242 listings in this study has a distinct hostname. The directory covers software broadly, including AI products, productivity tools, developer tools and other categories.

Our scanner looks for recognizable clues in public website responses, such as headers, cookies and HTML. For Next.js, those clues include its response header and characteristic page data or asset paths. These are observations about the website visitors arrive at; the application behind a login may use a different stack.

Coverage	Websites	Share of checked websites
At least one framework signal detected	5,060	49.4%
No recognized framework signal	5,182	50.6%
Total checked	10,242	100.0%

The 368 published listings without a recorded stack check are excluded. An absent signal in the checked sample means the scanner did not identify a supported framework. It does not establish that the website uses none.

Next.js is the most common signal by a wide margin

Detected technology	Websites	Share of 5,060 with a detected framework	Share of all 10,242 checked
Next.js	3,863	76.3%	37.7%
React, directly detected	830	16.4%	8.1%
Laravel	235	4.6%	2.3%
Vue, directly detected	198	3.9%	1.9%
Nuxt	184	3.6%	1.8%
Django	88	1.7%	0.9%
SvelteKit	81	1.6%	0.8%
Ruby on Rails	74	1.5%	0.7%
Angular	30	0.6%	0.3%

These rows overlap. We found multiple framework signals on 481 websites, so the percentages should not be added into a market-share total.

Next.js is itself a React framework. Our stored stack data omits signals inferred solely from another detected technology. The 830 therefore counts websites with an independently matched React signature; it is not the total number using React. There were 574 websites with a React signal and no Next.js signal. The same distinction applies to Vue and Nuxt.

Some signatures are more distinctive than others. A framework name mentioned in page content can match a broad detection rule, while an application can hide the signals our scanner expects. We did not manually audit every site's source code.

Next.js by product category

We compared categories with at least 200 checked websites. The chart uses all checked websites in each category as its denominator.

Category	Websites checked	Next.js detected	Share of all checked	Share where a framework was detected
AI Image & Video Generation	488	298	61.1%	77.8%
Video & Audio	384	202	52.6%	82.1%
Directory & Listings	309	158	51.1%	81.9%
Design & UI	368	181	49.2%	83.0%
Marketing & SEO	277	129	46.6%	79.1%
Productivity	797	295	37.0%	77.0%
Developer Tools	302	93	30.8%	66.9%

Selected categories from the 16 groups meeting the 200-website threshold; the complete aggregate table accompanies this study.

We detected Next.js on 298 of the 488 checked AI Image & Video Generation websites. Within the subset where any framework was detected, Design & UI has the highest Next.js share among the qualifying groups, at 83.0%.

We detected Next.js on 93 of the 302 Developer Tools websites. We did not ask those teams why they chose their stacks. Category differences may reflect website architecture, templates, the products listed, or how easily a framework can be detected. Our observations do not distinguish those explanations.

For teams building integrations, templates or documentation, these 3,863 listed hostnames provide a starting point for investigating a Next.js audience. We did not measure performance, revenue or developer satisfaction, so the counts cannot tell a team which framework to choose.

How we measured this

Snapshot: queried on 15 September 2026. Latest stored stack observations span 6–15 September; 8,944 of the 10,242 observations date to 6 September. We assembled the snapshot from those stored observations; we did not recrawl every site on publication day.
Population: published directree listings with a non-null stack observation timestamp. The study includes 10,242 distinct hostnames across 10,178 registered domains. The sample follows directory membership and covers software broadly, including AI tools. We did not sample randomly from the web.
Detection: directree's public-page scanner compares response evidence with its supported signatures. It does not run a full browser session or inspect private application code. The framework group includes libraries such as React and Vue; unsupported technologies and unmatched implementations remain undetected. Next.js also allows sites to disable its identifying response header, although other signals may still be present.
Counting: each listing counts once per detected technology. The headline denominator is the 5,060 listings with at least one framework signal. The broader denominator is all 10,242 checked listings. Percentages are rounded to one decimal place.
Sensitivity check: grouping by registered domain and counting a domain when any of its listed hostnames shows a signal gives 3,854 Next.js domains out of 5,040 domains with a detected framework, or 76.5%. That is close to the 76.3% hostname result.
Refresh: repeat the stored queries quarterly. Changes in directory membership or detection rules must be accounted for before interpreting a future percentage as a change in adoption.

Download the framework counts, complete category table and counting queries. We have retained the underlying observation snapshot for this edition. To inspect the signals on your own public website, use directree's tech stack detector.

GPTBot in robots.txt: the hosting toggle developers need to check

Henrik Åberg — Mon, 07 Sep 2026 09:35:38 +0000

Your robots.txt may express an AI policy you did not write.

We checked the homepage and robots.txt of 9,037 live AI tools listed on directree on 6 and 7 September 2026. Of those, 945 explicitly disallow OpenAI’s GPTBot in its own user-agent group: 10.5% of the sample.

Treat AI crawler rules as deployment configuration. Review them when you change hosting, enable a CDN feature, adopt a starter template, or hand site operations to someone else.

Read the full research and methodology.

GPTBot, search, and user browsing are separate

A common configuration blocks model training while keeping a site available in AI-assisted search and browsing:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

These are separate crawlers with separate purposes. In our sample, 839 of the 945 sites that block GPTBot, or 88.8%, still allow OAI-SearchBot. That is a deliberate and useful distinction if your goal is to opt out of training while remaining eligible to be cited in ChatGPT search.

The same pattern appears across AI labs. ClaudeBot is explicitly blocked by 10.1% of the 9,037 tools, while Claude-SearchBot is blocked by just 0.1%. Google-Extended is blocked by 9.9%, but its purpose is also distinct from ordinary Google Search crawling.

Do not assume a broad-looking rule has the result you want. Check the actual crawler names and decide which capabilities you want to permit.

A safe way to review your file

Start by opening the public URL:

https://your-domain.example/robots.txt

Then look for three things:

A named crawler group, such as User-agent: GPTBot.
A Disallow: / directly inside that group.
A wildcard group, User-agent: *, that could affect all crawlers.

Our measurement only counts a site as blocking GPTBot when the named GPTBot group itself contains Disallow: /. This matters because ordinary technical exclusions are widespread. Only 31 sites in the 9,037-site sample, or 0.3%, block every crawler outright. Meanwhile, 44% have a path-level Disallow rule in a wildcard group, often for an admin area or API route. That is normal site hygiene, not necessarily an AI policy.

Keep sensitive paths protected through authentication and application controls too. robots.txt is a crawler instruction, not access control.

Why hosting provider settings deserve attention

Hosting correlated strongly with whether a tool blocked GPTBot. Of 3,749 Cloudflare-hosted sites, 22.4% explicitly blocked it. On Vercel, 5.2% of 2,556 sites did. Netlify was 4.0% across 273 sites, Fastly 3.9% across 361, and CloudFront 3.4% across 328.

That makes Cloudflare-hosted tools more than four times as likely to block GPTBot as Vercel-hosted tools in this sample.

The likely explanation is operational rather than philosophical. A hosting control that offers to block AI bots can write or manage crawler policy at scale. That is convenient, but it can make a consequential content policy feel like a harmless performance or security toggle.

Whenever you enable one of these settings, verify the deployed result rather than relying on the dashboard label. Fetch robots.txt after deployment and include it in your release checklist.

Make the choice explicit

A small policy file is easier to maintain than an inherited one. If you want to block training but preserve discoverability, state that clearly with named groups. If you want to allow everything, avoid leaving old template rules behind. If you want a complete block, understand that wildcard rules can affect conventional search crawlers too.

The broader data shows this is becoming more common: 13.6% of 2,900 domains registered in 2026 block GPTBot, compared with 6.9% of 145 domains registered in 2023. That makes it even more important to know whether your policy is intentional.

For the complete crawler breakdown, hosting results, and methodology, see How many AI tools block GPTBot.