DEV Community: Gustavo Garcia

How to make your Next.js site appear in ChatGPT (and any LLM)

Gustavo Garcia — Mon, 03 Aug 2026 02:11:43 +0000

You blocked GPTBot in robots.txt to keep your content out of training runs — and then wondered why ChatGPT Search never cites your docs. Those are different systems. OpenAI’s own crawler docs say each bot is independent: allowing OAI-SearchBot keeps you eligible for ChatGPT search answers while disallowing GPTBot opts you out of foundation-model training.

This article is a Next.js App Router playbook for Generative Engine Optimization (GEO): how answer engines discover pages, which user-agents actually matter, how to configure robots.ts and sitemaps, how to keep HTML crawlable, and what llms.txt does — and does not — guarantee.

How LLMs find your site

Treat “appearing in an LLM” as three separate pipelines:

Pipeline	What it does	Typical bots / tokens
Training crawl	Collects public pages that may enter future model training	`GPTBot`, `ClaudeBot`, `Google-Extended` (token), Common Crawl’s `CCBot`
Search / answer index	Builds or refreshes retrieval so answers can cite your URLs	`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, classic `Googlebot` / `Bingbot` (and partners)
User-triggered fetch	Downloads a specific URL because a human asked for it (or pasted a link)	`ChatGPT-User`, `Claude-User`, `Perplexity-User`

Blocking the training bot does not automatically block the search bot. OpenAI states this explicitly for GPTBot vs OAI-SearchBot. Anthropic documents the same split for ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity documents PerplexityBot for search indexing and Perplexity-User for live fetches.

ChatGPT Search can also partner with third-party search providers. OpenAI’s help center documents that rewritten queries may be sent to partners such as Bing (and others listed in that article). Independently, OpenAI recommends allowing OAI-SearchBot if you want to appear in ChatGPT search answers. Practical implication: keep search bots allowed and stay healthy in major web indexes — do not optimize for a single rumor about which partner is “really” used this month.

OpenAI: the bots that decide ChatGPT visibility

From OpenAI’s crawler overview:

OAI-SearchBot — citations in ChatGPT Search

Used to surface websites in ChatGPT’s search features. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as plain navigational links. OpenAI recommends allowing it in robots.txt and permitting its published IP ranges. Changes can take about 24 hours to propagate.

GPTBot — training, not Search opt-out

Crawls content that may be used to train generative foundation models. Disallowing GPTBot signals that content should not be used for that training. It does not control ChatGPT Search eligibility.

ChatGPT-User — live, user-initiated fetches

Used when ChatGPT or Custom GPTs fetch a page because of a user action. OpenAI notes it is not used for automatic web crawling, not used to decide Search inclusion, and that robots.txt rules may not apply because the fetch is user-initiated. Manage Search with OAI-SearchBot; treat ChatGPT-User as a separate live-fetch channel.

OAI-AdsBot

Only visits pages submitted as ads on ChatGPT; not used to train foundation models. Relevant if you run ChatGPT ads — ignore it for organic GEO.

A common, defensible policy for content sites that want citations but not training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Next.js: configure `app/robots.ts`

App Router can generate /robots.txt from a typed file. Official docs: robots.txt file convention.

Example that keeps search/citation bots allowed, optionally opts out of training, fences private routes, and advertises the sitemap:

// app/robots.ts
import type { MetadataRoute } from 'next'

const SITE = 'https://example.com'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/admin/', '/drafts/'],
      },
      // ChatGPT Search + live fetch
      { userAgent: 'OAI-SearchBot', allow: '/' },
      { userAgent: 'ChatGPT-User', allow: '/' },
      // Training opt-out (optional — remove this rule to allow training)
      { userAgent: 'GPTBot', disallow: '/' },
      // Claude
      { userAgent: 'Claude-SearchBot', allow: '/' },
      { userAgent: 'Claude-User', allow: '/' },
      { userAgent: 'ClaudeBot', disallow: '/' },
      // Perplexity
      { userAgent: 'PerplexityBot', allow: '/' },
      { userAgent: 'Perplexity-User', allow: '/' },
      // Gemini grounding/training token (allow if you want Gemini apps to use you)
      { userAgent: 'Google-Extended', allow: '/' },
    ],
    sitemap: `${SITE}/sitemap.xml`,
    host: SITE,
  }
}

After deploy, open https://your-domain/robots.txt and confirm the groups look right. Spoofed user-agents exist — for verification, OpenAI and Perplexity publish IP range JSON files; Anthropic currently points publishers at robots.txt rather than relying on IP blocks.

Sitemap, Bing, Google, and freshness

Crawlers need URLs to discover. Next.js can generate /sitemap.xml from app/sitemap.ts (docs):

// app/sitemap.ts
import type { MetadataRoute } from 'next'

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getPublishedPosts() // your data layer

  return [
    { url: 'https://example.com', lastModified: new Date(), priority: 1 },
    ...posts.map((p) => ({
      url: `https://example.com/blog/${p.slug}`,
      lastModified: p.updatedAt ?? p.publishedAt,
      changeFrequency: 'monthly' as const,
      priority: 0.7,
    })),
  ]
}

Then:

Google Search Console — verify the property, submit the sitemap, fix crawl errors.
Bing Webmaster Tools — verify (you can often import from GSC), submit the same sitemap. Microsoft’s February 2026 AI Performance preview in Bing Webmaster Tools reports citations across Copilot, Bing AI summaries, and select partner integrations — useful GEO telemetry for Microsoft surfaces; do not treat it as a complete ChatGPT citation dashboard unless Microsoft says so.
IndexNow — ping participating engines (Bing, Yandex, and others listed on the site) when URLs are added, updated, or deleted so fresh content is prioritized sooner than passive crawl alone.

If ChatGPT Search partners with Bing for some queries, a healthy Bing index is still cheap insurance. A healthy Google index still matters for Google Search, AI Overviews / AI Mode, and anything that grounds on Google’s index.

Make the HTML worth crawling (Next.js-specific)

Answer engines and classic crawlers are more reliable when the important prose is in the first HTML response. App Router Server Components and SSR help; shipping an empty shell that only fills after client JavaScript is a classic way to look invisible.

Practical checklist for Next.js:

Prefer Server Components (or SSR) for article bodies, docs, and product copy.
Put real <title>, meta description, and Open Graph tags via the Metadata API — they travel with the document.
Avoid gating the main answer behind client-only fetches, infinite scroll without crawlable URLs, or auth walls for content you want cited.
Keep canonical URLs stable; use alternates.languages in the sitemap when you ship locales.
Return fast, stable 200 responses for public pages — timeouts and soft 404s waste crawl budget.

Structured data (JSON-LD for Article, FAQPage, Organization, etc.) does not replace good prose, but clear headings, short definitional paragraphs, tables, and FAQ sections match how Bing’s own GEO guidance describes content that is easier to cite accurately.

Content shape that answer engines can quote

Technical access is necessary but not sufficient. Pages that get cited tend to:

Answer the query early in plain language (then go deep)
Use descriptive ## / ### headings that match how people ask questions
Separate facts from opinion; link primary sources
Stay updated — stale version numbers get skipped or contradicted
Avoid thin listicles that every other site already paraphrased

You do not need a new CMS. You need pages that are the best extractable answer for a specific intent.

`llms.txt`: useful map, not a ranking switch

Jeremy Howard’s llms.txt proposal suggests a Markdown file at /llms.txt: site name as # heading, a short > summary, then ## sections with curated absolute links. Optional companion files (for example full concatenated context) exist in the ecosystem. Docs platforms (and many agent workflows) already use this pattern.

What it is not: a formal standard enforced by OpenAI, Google, or Anthropic for citation ranking. It does not replace robots.txt. It cannot block crawlers. Treat it as a curated table of contents for agents and humans who fetch /llms.txt on demand — especially documentation sites — not as a guaranteed ChatGPT ranking lever.

In Next.js you can start with public/llms.txt, or generate it from a Route Handler:

// app/llms.txt/route.ts
export function GET() {
  const body = `# Acme Docs
> Official documentation for the Acme API and SDKs.

## Docs
- [Quick start](https://example.com/docs/quickstart.md): Install and make your first request
- [Auth](https://example.com/docs/auth.md): API keys and OAuth

## Optional
- [Changelog](https://example.com/changelog): Release history
`

  return new Response(body, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=3600',
    },
  })
}

If you also serve Markdown mirrors of key pages (the proposal’s .md suffix idea), agents get cleaner context than parsing your marketing HTML.

Common mistakes

Disallowing GPTBot and assuming you left ChatGPT Search — you need OAI-SearchBot allowed for Search answers.
Correct robots.txt, hostile WAF — Cloudflare / AWS WAF “block AI bots” rules can drop search crawlers before they read your allow rules. Whitelist by user-agent and published IPs where vendors provide them.
Client-only content — if curl does not show the answer text, many crawlers will not either.
No sitemap / never submitted to Bing or Google — discovery stalls.
Treating llms.txt as access control — it guides; it does not enforce.
Blocking Google-Extended while expecting Gemini grounding — that token covers training and Gemini grounding, not Search ranking.
Expecting overnight miracles — OpenAI notes ~24h for Search robots adjustments; indexes and reputation still take time and quality.

When this matters (and when it does not)

Invest in GEO when you publish public expertise (docs, tutorials, comparisons, research) and want referral traffic or brand citation from ChatGPT, Claude, Perplexity, Copilot, or Gemini.

De-prioritize (or fully opt out) when the product is private, paywalled, legally sensitive, or you deliberately do not want model/training or answer-engine reuse — then disallow the relevant bots and accept lower AI visibility.

FAQ

Does blocking GPTBot hide me from ChatGPT?

No. Per OpenAI, Search visibility is governed by OAI-SearchBot. GPTBot is the training-oriented crawler. You can disallow GPTBot and still allow OAI-SearchBot.

Is Bing enough to appear in ChatGPT?

OpenAI documents third-party search partners (including Bing) and its own OAI-SearchBot. Do both: allow OAI-SearchBot and keep a healthy presence in major indexes. Do not rely on a single secondary blog’s claim about exclusive Bing or Google dependency.

Will `llms.txt` make ChatGPT cite me?

There is no public commitment from major AI labs that llms.txt controls ChatGPT citation ranking. It is still worth shipping for docs and agent UX. Citations still depend on crawl access, indexability, and content quality.

Should I allow every AI bot?

Not automatically. Decide per pipeline: search/citation vs training vs user fetch. Many content sites allow search bots, allow or disallow training consciously, and keep sensitive paths disallowed for everyone.

Does App Router hurt SEO or AI visibility?

No — Server Components and the Metadata / sitemap / robots file conventions are well suited to crawlable HTML. Problems come from client-only rendering patterns, not from App Router itself.

Conclusion

To show up in ChatGPT Search and other LLM answers: allow the search crawlers, do not let your WAF undo robots.txt, ship crawlable HTML from Next.js, submit sitemaps (and IndexNow where it helps), and write pages that answer specific questions clearly. Optionally add llms.txt as a curated map. Optionally disallow training bots if that is your policy — without confusing them for search bots.

Ship the OAI-SearchBot allow rule this week if it is missing. Everything else compounds on top of being fetchable.

References

Overview of OpenAI Crawlers — OpenAI Developers (accessed Aug 2026)
ChatGPT Search — OpenAI Help Center
Does Anthropic crawl data from the web… — Anthropic / Claude Help Center (Apr 2026)
Perplexity Crawlers — Perplexity Docs
Google’s common crawlers (incl. Google-Extended) — Google for Developers
The /llms.txt file — Jeremy Howard / Answer.AI proposal
Next.js robots.txt — Next.js Docs
Next.js sitemap — Next.js Docs
Introducing AI Performance in Bing Webmaster Tools — Bing Webmaster Blog (Feb 2026)
IndexNow — IndexNow protocol
OAI-SearchBot IP ranges — OpenAI
GPTBot IP ranges — OpenAI
How to make your Next.js site appear in ChatGPT - Personal Portfolio Article

6 problems your API gateway is already suffering from

Gustavo Garcia — Mon, 03 Aug 2026 02:04:21 +0000

An API gateway looks like a thin proxy until the night it becomes the outage.

Centralization is the point: one place for TLS termination, authentication, rate limiting, routing, and policy enforcement.

Centralization is also the risk.

Every request shares the same event-loop budget, connection pools, memory, and blast radius.

Microsoft's Gateway Offloading guidance says it plainly: keep the gateway highly available, ensure it never becomes the bottleneck, and don't put business logic inside it.

Canva's public incident report from November 2024 demonstrated what happens when that shared front door saturates: approximately 1.5 million requests per second—around 3× normal peak traffic—combined with a telemetry lock that blocked Netty's event loop until Linux began OOM-killing gateway instances.

This article walks through six production problems API gateways commonly suffer from—and how to prevent them before the next traffic spike.

What an API Gateway Is (and Isn't)

An API gateway is the edge facade sitting in front of backend services.

Typical responsibilities include:

TLS termination
Authentication
Rate limiting
Routing
Metrics and tracing
Header or path transformation

It is not responsible for:

Business authorization
UI composition
Unlimited retries
Product-specific business rules

Treating the gateway as "just a proxy" hides its failure modes.

Treating it like an application server creates new ones.

1. Shared Blast Radius

Symptom

Everything appears offline even though backend services remain healthy.

Autoscaling launches new gateway instances that immediately fail.

Why it happens

One gateway cluster fronts every public request.

A traffic spike, blocking plugin, or memory leak impacts the entire platform simultaneously.

Microsoft explicitly warns against allowing the gateway to become a single bottleneck.

Canva provides a textbook example.

A delayed CDN asset caused over 270,000 clients to reconnect simultaneously.

Once the asset became available, every client resumed requests at nearly the same moment.

The gateway fleet saturated.

Off-heap memory exploded.

Linux OOM-killed containers faster than autoscaling could replace them.

Recovery only became possible after traffic was blocked at the CDN.

How to fix it

Keep baseline capacity well above average load
Load test the gateway as a Tier-0 service
Add load shedding (HTTP 503) before queues consume the fleet
Isolate listener pools or gateways by product area
Practice traffic blocking at the CDN instead of relying only on cluster scaling

Saturation isn't simply "high latency."

It's when the gateway itself becomes the system bottleneck.

2. Broken Timeout Hierarchies

Symptom

Clients receive HTTP 504 while backend requests continue executing.

Connection pools fill.

Latency increases until the entire gateway stalls.

Why it happens

Timeouts are configured independently instead of forming a hierarchy.

If the gateway waits longer than the client, it continues consuming resources after users have already disconnected.

Slow upstreams then exhaust connection pools.

Queues continue growing.

New requests begin failing.

AWS documents this behavior in API Gateway timeout guidance.

Increasing timeout values without redesigning timeout relationships simply keeps abandoned requests alive longer.

Correct timeout ordering

Client timeout
      >
Gateway → Backend timeout
      >
Per-retry timeout

Recommendations

Gateway timeouts should always be shorter than client timeouts
Propagate deadlines downstream
Cap pending request queues
Prefer early HTTP 503 over extremely long waits
Long-running work belongs in asynchronous workflows, not synchronous gateway paths

3. Retry Storms

Symptom

A minor upstream failure becomes a platform-wide outage.

CPU utilization spikes.

Request volume multiplies.

Recovery takes longer than the original incident.

Why it happens

Retries amplify traffic.

AWS Well-Architected explicitly warns against retries without:

exponential backoff
jitter
retry limits
centralized retry ownership

Multiple retry layers compound.

For example:

Client retries × Gateway retries × Service Mesh retries

A single failed request can become dozens.

Envoy recommends using retry budgets rather than unlimited retry counts.

Recommendations

Retry only:

connection failures
transient timeouts
selected HTTP 5xx
HTTP 429

Never retry:

validation errors
authentication failures
authorization failures

Also:

use exponential backoff
add jitter
cap retry attempts
configure retry budgets
retry in one layer only

4. The God Gateway

Symptom

Every product change requires modifying gateway configuration.

Gateway plugins begin parsing JSON.

Business rules appear inside Lua or JavaScript extensions.

Latency slowly increases.

Why it happens

Cross-cutting concerns gradually become business logic.

Microsoft's Gateway Offloading documentation is unambiguous:

Business logic should never be offloaded to the gateway.

Good gateway responsibilities:

TLS
authentication
routing
API versioning
correlation IDs

Poor gateway responsibilities:

entitlement rules
product workflows
screen composition
response shaping
domain validation

Rule of thumb

If a Product Manager is approving gateway changes more often than an SRE, the gateway is doing too much.

5. Blind Observability

Symptom

Monitoring shows "API latency."

Nobody knows whether the delay comes from:

TLS
authentication
plugins
gateway overhead
backend services

Or worse:

Everything works until load increases.

Then a seemingly harmless metrics library blocks the event loop.

Why it happens

Observability itself becomes part of the request path.

Canva's postmortem identified a telemetry lock that reduced Netty throughput precisely when the traffic spike arrived.

Recommendations

Measure separately:

Gateway processing time
Upstream latency
Pending queue depth
Active connections
Retry overflow
Circuit breaker events

Prefer:

local JWT verification
cached JWKS
asynchronous logging
non-blocking plugins

If you can't answer

"Is the gateway slow, or is Payments slow?"

within one minute, observability needs improvement.

6. Weak Edge Hardening

Symptom

Rate limits work inconsistently.

Backends remain publicly accessible.

Administrative APIs are internet-exposed.

Why it happens

Three common mistakes:

Per-node rate limiting

Each gateway replica enforces its own counters.

Four replicas effectively quadruple the intended limit.

Backend bypass

Clients can reach backend services directly.

Gateway security policies become optional.

Exposed control plane

Administrative APIs remain publicly reachable.

For products like Kong, exposing the Admin API effectively grants full control over the gateway.

Recommendations

Keep backend services private
Enforce authentication at the gateway
Use shared rate limiting for cluster-wide quotas
Protect administrative APIs with private networking and RBAC
Continuously scan for routes bypassing the gateway

How These Problems Reinforce Each Other

Traffic spike
      │
      ▼
Gateway saturation
      │
      ├── Timeout inversion
      │         │
      │         ▼
      │   Pool exhaustion
      │
      ├── Retry storms
      │         │
      │         ▼
      │   Traffic amplification
      │
      ├── Business logic
      │         │
      │         ▼
      │   Reduced capacity
      │
      ▼
Observability failures
      │
      ▼
Security and control-plane gaps

Fixing only retries while telemetry still blocks the event loop doesn't solve the outage.

These issues reinforce one another.

FAQ

Is an API Gateway always a single point of failure?

Not necessarily.

Multiple gateway instances remove a single process as a SPOF.

However, shared configuration, shared telemetry, or synchronized traffic spikes can still create a shared failure domain.

Should the gateway perform retries?

Sometimes.

Only for idempotent operations and only within a defined retry budget.

Avoid retries simultaneously in the client, gateway, and service mesh.

Gateway vs BFF vs Service Mesh

Component	Responsibility
API Gateway	North–south traffic, TLS, authentication, routing
BFF	Product-specific aggregation and UI contracts
Service Mesh	East–west resilience, mTLS, retries, service communication

How do I know my timeout hierarchy is wrong?

Typical indicators include:

backend work continues after clients disconnect
growing pending queues
frequent HTTP 504 responses
connection pool exhaustion

Instrument client timeout, gateway timeout, and upstream duration in the same trace.

Final Thoughts

Production gateways rarely fail because routing is difficult.

They fail because every concern eventually accumulates at the edge:

shared saturation
inverted timeout budgets
retry amplification
business logic
opaque observability
weak control-plane security

Each problem has well-established architectural guidance.

The goal isn't building a smarter gateway.

It's building one that is boring, observable, predictable, and capable of failing in small pieces instead of taking the entire platform down.

References

Canva incident report: API Gateway outage — Canva Engineering (Dec 2024)
Gateway Offloading pattern — Microsoft Azure Architecture Center
API gateways in microservices — Microsoft Azure Architecture Center
Gateway Routing pattern — Microsoft Azure Architecture Center
Circuit breaking — Envoy Proxy documentation
Outlier detection — Envoy Proxy documentation
Circuit breakers (proto) — retry_budget — Envoy Proxy API
REL05-BP03 Control and limit retry calls — AWS Well-Architected
REL05-BP05 Set client timeouts — AWS Well-Architected
Troubleshoot API Gateway HTTP 504 timeout errors — AWS re:Post
Amazon API Gateway integration timeout limit increase beyond 29 seconds — AWS News (Jun 2024)
Secure the Admin API — Kong Gateway documentation
Kong API Gateway Misconfigurations: An API Gateway Security Case Study — Trend Micro
Timeouts, retries, and backoff with jitter — Amazon Builders’ Library
Exponential Backoff And Jitter — AWS Architecture Blog
6 problems your API gateway is already suffering from - Personal Portfolio Article

DEV Community: Gustavo Garcia

How to make your Next.js site appear in ChatGPT (and any LLM)

How LLMs find your site

OpenAI: the bots that decide ChatGPT visibility

OAI-SearchBot — citations in ChatGPT Search

GPTBot — training, not Search opt-out

ChatGPT-User — live, user-initiated fetches

OAI-AdsBot

Other answer engines (same idea, different names)

Anthropic (Claude)

Perplexity

Google Gemini vs Google Search

Next.js: configure app/robots.ts

Sitemap, Bing, Google, and freshness

Make the HTML worth crawling (Next.js-specific)

Content shape that answer engines can quote

llms.txt: useful map, not a ranking switch

Common mistakes

When this matters (and when it does not)

FAQ

Does blocking GPTBot hide me from ChatGPT?

Is Bing enough to appear in ChatGPT?

Will llms.txt make ChatGPT cite me?

Should I allow every AI bot?

Does App Router hurt SEO or AI visibility?

Conclusion

References

6 problems your API gateway is already suffering from

What an API Gateway Is (and Isn't)

1. Shared Blast Radius

Symptom

Why it happens

How to fix it

2. Broken Timeout Hierarchies

Symptom

Why it happens

Correct timeout ordering

Recommendations

3. Retry Storms

Symptom

Why it happens

Recommendations

4. The God Gateway

Symptom

Why it happens

Rule of thumb

5. Blind Observability

Symptom

Why it happens

Recommendations

6. Weak Edge Hardening

Symptom

Why it happens

Per-node rate limiting

Backend bypass

Exposed control plane

Recommendations

How These Problems Reinforce Each Other

FAQ

Is an API Gateway always a single point of failure?

Should the gateway perform retries?

Gateway vs BFF vs Service Mesh

How do I know my timeout hierarchy is wrong?

Final Thoughts

References

Next.js: configure `app/robots.ts`

`llms.txt`: useful map, not a ranking switch

Will `llms.txt` make ChatGPT cite me?