DEV Community: Muhammad Ahmad

How to Add llms.txt, llms-full.txt, and W3C tdmrep.json to a Next.js App Router Site

Muhammad Ahmad — Wed, 11 Nov 2026 04:37:53 +0000

If you maintain developer documentation, an API platform, or a SaaS web app in Next.js (App Router), adding machine-readable endpoints for AI coding assistants (Cursor, Windsurf, GitHub Copilot, Claude Code) and AI search engines (OAI-SearchBot, PerplexityBot) takes less than 10 minutes.

In this tutorial, we will look at the exact structure required by the llmstxt.org specification, the 3 most common RFC 9309 robots.txt mistakes that break AI crawlers, and how to serve /llms.txt and /.well-known/tdmrep.json using Next.js static route handlers.

1. The Strict `llmstxt.org` Markdown Grammar

Unlike sitemap.xml, /llms.txt is designed to be read by both deterministic parsers and LLM tokenizers. The specification enforces a strict ordering:

# Project or Platform Name

> A concise, factual 1-to-2 sentence summary of what the platform does, its core architecture, and target audience (zero marketing superlatives).

Optional concise operational notes (e.g., authentication model, rate limits, or API base URL).

## Core Product & Tools

- [JSON Formatter & Validator](https://example.com/developer/json-formatter): Strict RFC 8259 JSON validator with 64-bit BigInt truncation detection and JSONPath evaluation.
- [HAR File Analyzer & Sanitizer](https://example.com/developer/har-analyzer): Client-side 7-phase HTTP waterfall inspector and 14-surface secret scrubber.

## Optional

- [Privacy Policy](https://example.com/privacy): Details on client-side data handling and GDPR/CCPA compliance.

Three Linter Rules to Watch Out For:

Exactly One # H1 at Line 1: Parsers expect the document to begin with a single H1 heading followed immediately by the > summary blockquote.
No Vague Link Anchors: Never write - [Click here](...) or - [Read more](...). The anchor text and colon-separated description are what the LLM uses to decide whether to fetch the underlying URL.
Use the ## Optional Section for Secondary Pages: According to the spec, links under ## Optional tell context-constrained models which URLs can be skipped when token budgets are tight.

2. Serving `/llms.txt` in Next.js App Router (`app/llms.txt/route.ts`)

In Next.js App Router, you can serve /llms.txt at the edge with zero runtime overhead using force-static and Cache-Control headers:

// app/llms.txt/route.ts
export const dynamic = 'force-static';

const LLMS_TXT_CONTENT = `# YourPlatformName

> Factual summary of your platform and technical architecture.

## Docs & Core Tools
- [API Reference](https://example.com/docs/api): Complete REST & GraphQL endpoint specifications.
`;

export async function GET() {
  return new Response(LLMS_TXT_CONTENT, {
    status: 200,
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=3600, s-maxage=86400, stale-while-revalidate=604800',
      'X-Robots-Tag': 'all',
    },
  });
}

3. Auditing Your `robots.txt` for RFC 9309 Prefix Leaks

Under IETF RFC 9309, Disallow: /admin matches /admin, /admin/, AND /admin-guide.html (because Disallow is a prefix match!). Conversely, if you write Disallow: /private/, a request to /private (without the trailing slash) is still allowed.

And if you use Cloudflare in front of Next.js, verify that Cloudflare's Managed robots.txt ("Block AI training bots") isn't prepending its own rules above your app/robots.ts output—which frequently causes inconsistent AI bot policies across Amazonbot, Claude-Web, and Meta-ExternalAgent.

If you want to auto-convert your existing sitemap.xml or HTML docs into llms.txt, llms-full.txt, W3C tdmrep.json, and Next.js route code—while profiling token usage across GPT-4o (128k), Claude 3.7 Sonnet (200k), and DeepSeek R1 (64k)—you can use the client-side KivaroTools llms.txt & AI Crawler Generator.

Turning a Production .HAR Capture into a 7-Phase Latency Audit, Playwright Test, and k6 Load Script

Muhammad Ahmad — Wed, 11 Nov 2026 00:15:58 +0000

When a web app feels sluggish in production or fails intermittently during a multi-step checkout flow, a browser HTTP Archive (.har) capture contains the ground truth. Yet most developers only open .har files to glance at status codes, missing two massive productivity wins:

Automated 7-Phase Network Bottleneck Diagnostics (distinguishing DNS/TLS handshakes from HTTP/1.1 connection pool starvation and backend TTFB stalls).
Instant Replay Code Generation (converting the exact filtered network sequence into a runnable Playwright E2E test, a k6 load test script, or an OpenAPI 3.1 schema).

Let's look at how the HAR 1.2 timing specification works under the hood and how to transform raw network traces into executable tests.

1. Reading the 7 Timing Phases in `entry.timings`

Per the HAR 1.2 spec, every network entry records seven timing deltas (in milliseconds, where -1 indicates the phase did not apply, such as ssl on a reused keep-alive socket):

export interface HarTimingPhases {
  blocked: number; // Waiting for an available TCP socket (6-connection browser limit!)
  dns: number;     // Domain Name System lookup time
  connect: number; // TCP 3-way handshake (SYN / SYN-ACK / ACK)
  ssl: number;     // TLS 1.2 / 1.3 handshake negotiation
  send: number;    // Time spent transmitting request headers & body
  wait: number;    // Time to First Byte (TTFB) — server processing + network RTT
  receive: number; // Time spent downloading the response payload
  total: number;
}

When analyzing a waterfall across hundreds of requests, three patterns immediately pinpoint the bottleneck:

High blocked (> 100 ms) with connect: -1: The browser is hitting the 6-connections-per-origin limit on HTTP/1.1, or main-thread JavaScript is starving the network queue. Upgrading the origin to HTTP/2 or HTTP/3 (multiplexing over a single connection) eliminates the blocked queue.
High wait (TTFB > 500 ms) with low receive: The bottleneck is strictly server-side (database query latency, cold-start serverless execution, or upstream microservice contention).
High receive on JSON/JS responses: Check response.headers for Content-Encoding: br or gzip. If response.bodySize equals response.content.size on a 500 KB payload, compression is missing at the CDN edge.

2. Generating Playwright E2E and k6 Load Scripts Directly from Filtered HAR Entries

Instead of manually writing page.request.post(...) or k6 http.batch(...) calls from scratch to reproduce a bug, you can filter a HAR capture down to the XHR/Fetch API calls (resourceType === 'fetch' || resourceType === 'xhr'), sanitize any live credentials into environment variable placeholders ({{BEARER_TOKEN_1}}), and compile the trace directly into test code:

// Auto-generated k6 load script from a sanitized HAR sequence:
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 10,
  duration: '30s',
  thresholds: {
    http_req_duration: ['p(95)<500'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const params = {
    headers: {
      'Accept': 'application/json',
      'Authorization': __ENV.BEARER_TOKEN_1 || 'Bearer REDACTED',
    },
  };
  const res = http.get('https://api.example.com/v1/orders?limit=25', params);
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}

If you want to inspect 7-phase waterfalls, compare two HAR captures ("Before vs. After Deployment" regression diffing), or export sanitized traces to k6, Playwright, OpenAPI 3.1, Postman v2.1, or cURL/Fetch/Python without uploading your HAR file to any server, you can run the KivaroTools Client-Side HAR Analyzer & Sanitizer locally in your browser.

How to Render and Query a 25 MB / 60,000-Node JSON Tree in the Browser at 60 FPS

Muhammad Ahmad — Tue, 10 Nov 2026 23:10:20 +0000

How to Render and Query a 25 MB / 60,000-Node JSON Tree in the Browser at 60 FPS

If you have ever pasted a 10 MB JSON response into a browser-based JSON viewer and watched the tab freeze with an "Aw, Snap! Out of Memory" crash, the culprit is almost never JSON.parse() itself.

In V8, parsing 10 MB of JSON takes ~60 milliseconds. What kills the browser tab is DOM node explosion and main-thread blocking:

A 10 MB JSON payload easily contains 50,000 to 150,000 keys and array elements. Creating 3 <div> and <span> elements per node injects 300,000+ DOM nodes into the document tree, exhausting layout and style recalculation budgets.
Running syntax repair, key sorting, and search indexing on the UI thread blocks requestAnimationFrame for seconds at a time.

Here is how we architected a client-side JSON workbench that formats, repairs, queries, and renders up to 25 MB payloads at 60 FPS using Web Workers, flat tree virtualization, and a 4-stage dirty-JSON repair pipeline.

Step 1: Offloading Payloads > 256 KB to an Inline Web Worker

For small payloads under 256 KB, synchronous parsing on the main thread takes less than 4 ms. Once a payload crosses 256 KB (up to our 25 MB browser safety ceiling), every heavy operation—strict parsing, lexical diagnostics, recursive key sorting, and stats calculation—should run off the main thread:

const MAX_INPUT_BYTES = 25 * 1024 * 1024; // 25 MB hard cap
const WORKER_THRESHOLD = 256 * 1024;      // 256 KB triggers Web Worker

By serializing the worker script into a Blob URL (URL.createObjectURL(new Blob([workerSource]))), the worker runs 100% locally in the browser without requiring external network fetches, keeping the main UI thread responsive while typing or scrolling.

Step 2: Flattening the Hierarchy for 28px Virtualized Row Recycling

Instead of recursively rendering nested <ul><li> components in React, we flatten only the currently expanded tree branches into a contiguous 1D array of VisibleTreeRow descriptors:

export interface VisibleTreeRow {
  path: string;          // e.g., "$.users[0].profile"
  key: string | null;
  value: JsonValue;
  depth: number;
  isExpandable: boolean;
  childCount: number;
  summary: string;       // e.g., "{ 4 keys }" or "[ 120 items ]"
}

With a fixed row height (ROW_HEIGHT = 28 px) and an overscan buffer (OVERSCAN = 12 rows above and below the scroll viewport), a tree with 60,000 expanded rows only mounts ~45 actual DOM nodes at any given millisecond:

const startIndex = Math.max(0, Math.floor(scrollTop / ROW_HEIGHT) - OVERSCAN);
const endIndex = Math.min(
  visibleRows.length,
  Math.ceil((scrollTop + viewportHeight) / ROW_HEIGHT) + OVERSCAN
);
const windowedRows = visibleRows.slice(startIndex, endIndex);

Step 3: Repairing "Dirty" JavaScript & Python Logs into Valid RFC 8259 JSON

Developers frequently copy broken JSON from terminal logs, Python dicts, or JavaScript source files containing:

Single-line (//) and block (/* ... */) comments
Single-quoted strings ('status': 'active')
Unquoted object keys ({ userId: 42, active: true })
Python/JS non-JSON literals (None, True, False, undefined, NaN, Infinity)
Trailing commas before } or ]

A naive .replace(/'/g, '"') corrupts strings that contain apostrophes ("Don't"). Instead, a deterministic 4-pass character lexer (repairJson) tracks string context so it only converts outer single quotes while escaping internal double quotes (\"), strips comments outside string literals, quotes bare identifier keys, and strips trailing commas.

You can inspect this virtualized tree architecture, test the 4-pass auto-repair engine, or run live JSONPath filter queries ($.items[?(@.price < 50)]) directly in your browser with the KivaroTools JSON Formatter, Validator & Tree Viewer.

A Browser-Network Audit of Seven Developer Utilities

Muhammad Ahmad — Wed, 21 Oct 2026 19:07:35 +0000

Web-based developer tools can make everyday tasks—formatting JSON, converting data, encoding text, or testing regular expressions—quick and convenient. But when a tool processes input on a remote server, that input may leave the browser. For developers working with sensitive material, it is useful to understand what network activity occurs while a tool is in use.

This case study examines seven utilities in the KivaroTools developer tools directory. The site describes its tools as processing inputs locally. To observe their behavior in a browser, I ran an automated test using Headless Google Chrome and the Chrome DevTools Protocol (CDP).

Audit method

The audit used Headless Google Chrome v155.0.8059.80 on October 21, 2026. Each utility was given a harmless, unique test input. The test monitored captured browser network events—including Fetch, XHR, beacons, WebSockets, and asset requests—during page load and tool interaction. It also checked whether the unique input strings appeared in captured request data.

The initial page loads generated requests for resources such as HTML, JavaScript, CSS, and icons. The audit report also recorded sitewide analytics or telemetry activity during page load. The interaction results below refer to the subsequent tool actions, not to the initial page loads.

Results

Utility	Test input	Observation during interaction
JSON Formatter	`{"audit_live": "test_json_kivaro_99"}`	No new outgoing requests observed; test string not found in captured traffic.
SVG Optimizer	`<svg id="audit_live_svg_kivaro_88"></svg>`	No new outgoing requests observed; test string not found in captured traffic.
Password Generator	A generated test password	No new outgoing requests observed. The generated value is not reproduced here.
JSON Converter	`[{"audit_live": "csv_kivaro_77"}]`	No new outgoing requests observed; test string not found in captured traffic.
Base64 Encoder	`audit_live_base64_kivaro_66`	No new outgoing requests observed; test string not found in captured traffic.
SQL Formatter	`SELECT * FROM audit_live_sql_kivaro_55`	No new outgoing requests observed; test string not found in captured traffic.
Regex Tester	`audit_live_regex_kivaro_44`	No new outgoing requests observed; test string not found in captured traffic.

Across the tested interactions, the report recorded no new outgoing network requests. It also found no test inputs in the captured request data. The report described the operations as running in the browser—for example, using client-side code for formatting and conversion, and the browser’s JavaScript engine for regular-expression matching.

What these results show—and what they do not

In this test session, the observed network activity occurred during initial page loads. No new outgoing requests were observed while the seven tools processed the test inputs. This is consistent with client-side processing under the tested conditions.

The finding is limited to this test session, browser build, and set of inputs. It does not establish that the tools will behave identically in every browser, configuration, or future version. Nor does the absence of a recognizable input string in captured traffic rule out every other possibility: data could be transformed before transmission, and this network test does not determine whether scripts or browser extensions can access page content locally. It is not a comprehensive security audit or a guarantee that sensitive information is safe to enter.

Reproducing the check

Open a tool in Chrome and launch Developer Tools.
In the Network tab, enable Preserve log and select All.
Record the initial page-load activity. Then clear the log or mark where the interaction test begins.
Enter a harmless, unique test string and run the tool.
Check for new requests and search captured request data for the test string.
Repeat for each tool, noting the browser version, date, and test inputs.

Use synthetic inputs for this check—not actual passwords, API keys, customer data, or other secrets. A browser network inspection can reveal some data transfers, but it cannot by itself establish that a website is secure.

Takeaway

The automated test reported no new outgoing network requests during interaction with these seven utilities, and the test strings were not found in captured traffic. That is a useful, bounded observation about their behavior in the tested session. Developers should treat it as one piece of evidence—not as a substitute for evaluating the tool, its implementation, and their own data-handling requirements.

DEV Community: Muhammad Ahmad

How to Add llms.txt, llms-full.txt, and W3C tdmrep.json to a Next.js App Router Site

1. The Strict llmstxt.org Markdown Grammar

Three Linter Rules to Watch Out For:

2. Serving /llms.txt in Next.js App Router (app/llms.txt/route.ts)

3. Auditing Your robots.txt for RFC 9309 Prefix Leaks

Turning a Production .HAR Capture into a 7-Phase Latency Audit, Playwright Test, and k6 Load Script

1. Reading the 7 Timing Phases in entry.timings

2. Generating Playwright E2E and k6 Load Scripts Directly from Filtered HAR Entries

How to Render and Query a 25 MB / 60,000-Node JSON Tree in the Browser at 60 FPS

Step 1: Offloading Payloads > 256 KB to an Inline Web Worker

Step 2: Flattening the Hierarchy for 28px Virtualized Row Recycling

Step 3: Repairing "Dirty" JavaScript & Python Logs into Valid RFC 8259 JSON

A Browser-Network Audit of Seven Developer Utilities

Audit method

Results

What these results show—and what they do not

Reproducing the check

Takeaway

Disclosure: The author is affiliated with KivaroTools, whose utilities are examined in this case study.

1. The Strict `llmstxt.org` Markdown Grammar

2. Serving `/llms.txt` in Next.js App Router (`app/llms.txt/route.ts`)

3. Auditing Your `robots.txt` for RFC 9309 Prefix Leaks

1. Reading the 7 Timing Phases in `entry.timings`