A brand hands us a domain, and most of the crawler is about which five pages not to read

Setting up a campaign in Nakodo starts with one field: your domain. From that we read your site and write the brief the rest of the product works from.

The naive version of this is six lines: fetch the homepage, strip the tags, send the text to a model. We shipped something closer to 370 lines, and almost none of the extra is about fetching. It is about choosing what not to send, and about what to say when a site will not open.

Three ways to ask for a homepage

const urls = [`https://${domain}/`, `https://www.${domain}/`, `http://${domain}/`];

Plenty of sites answer on exactly one of those. The ordering is not interesting; what the failures mean is:

for (const url of urls) {
  if (url.startsWith("http:") && failures.some((f) => f.problem !== "error" && f.problem !== "tls")) break;
  ...
}
const worth = failures.find((f) => f.problem !== "not_found") ?? failures[0];
throw new CrawlError(crawlErrorMessage(domain, worth.problem, worth.detail), worth.problem);

Two rules fell out of watching real domains fail:

Plain HTTP is a last resort, not a third try. If https://acme.com returned a 403 and https://www.acme.com timed out, trying port 80 tells you nothing you want to act on: the site is up and refusing you. The only failures worth downgrading for are a connection error and a TLS error, because those are what a site that genuinely has no working HTTPS looks like.

"Not found" is only true if every attempt said so. ENOTFOUND on the apex while www resolves and then refuses is a configured site, not a typo. Reporting "we couldn't find acme.com. Check the spelling" there sends the customer off to check a spelling that is correct.

Eight failure kinds, because "we couldn't read your site" is useless

export type CrawlProblem = "invalid" | "not_found" | "blocked" | "private" | "timeout" | "tls" | "empty" | "error";

Each one is a sentence someone can do something with. A few of them:

case "private":
  return `${domain} points to a private network address, so we can't read it.`;
case "tls":
  return `${domain} has a security certificate problem, so we couldn't load it safely.`;
case "empty":
  return `${domain} has almost no text we can read. It may need JavaScript to show its content.`;

The mapping from the underlying failure is one function with no clever parts, and the HTTP statuses are the opinionated bit:

if (failure.status !== undefined) {
  return [401, 403, 429, 451, 503].includes(failure.status) ? "blocked" : "error";
}

401, 403, 429, 451 and 503 all mean "a server is there and chose not to serve you", which is a different conversation with the customer than a 500. "private" comes from our own fetch wrapper refusing to resolve to a private network address, which is its own post.

"empty" is the one that earns its keep most often:

const MIN_SITE_TEXT = 300; // less readable text than this and the brief would be guesswork

A client rendered marketing site serves about four words of HTML. Without this check the model receives those four words, writes a confident brief about a company it knows nothing about, and the customer finds out three screens later. Three hundred characters is low enough to pass a genuinely terse landing page and high enough to catch an empty shell.

Which pages, in which order

Six regexes, in priority order, each contributing at most one page:

const PAGE_KINDS: RegExp[] = [
  /\/(about|about-us|our-story|story|who-we-are|mission)(\/|$|\.)/,
  /\/(products?|shop|store|collections?|catalog|menu)(\/|$)/,
  /\/(pricing|plans|price)(\/|$)/,
  /\/(features?|how-it-works|solutions?|services?|use-cases?|platform)(\/|$)/,
  /\/(faq|faqs|help)(\/|$)/,
  /\/(shipping|delivery|international-shipping)(\/|$|-)/,
];

Candidates come from the homepage's own links first and the sitemap second, and when several match a kind, the tie breaks are:

.sort(([pa, a], [pb, b]) =>
  a.depth - b.depth || Number(b.onHome) - Number(a.onHome) || pa.length - pb.length)

Shallowest wins, then linked from the homepage, then shortest path. That is three heuristics standing in for "the one a human would click", and in practice the depth term does most of the work: /pricing beats /en/business/pricing-2024.

Five pages is the cap. Not because of a token budget, but because page six is reliably a careers page or a cookie policy, and the marginal sentence it adds about the company is worth less than the risk of the model latching onto it.

One Set stops the navigation being read five times

Every page on a site repeats the header, the nav, the footer and the cookie banner. Five pages of raw text means five copies of the menu, and a model given five copies of a menu will tell you the menu is important.

function pageText(lines: string[], seen: Set<string>, maxChars: number): string {
  const out: string[] = [];
  let length = 0;
  for (const line of lines) {
    if (seen.has(line)) continue;
    seen.add(line);
    if (length + line.length > maxChars) break;
    out.push(line);
    length += line.length + 1;
  }
  return out.join("\n");
}

The seen Set lives across the whole crawl, and the homepage goes first, so the nav is read exactly once, as part of the homepage, where it is genuinely informative. Every later page gets only the lines that are new on it.

The budgets are 8,000 characters for the homepage and 5,000 for each key page. There is a second order effect that is easy to miss: this text is stored and sent again every time anything is written from the research, so a page of duplicated chrome is not a one off cost, it is a cost per generation forever.

A page whose unique text comes to under 100 characters is dropped entirely. That is usually a "/products" page that is all images and a repeated nav.

The sitemap is structure, not content

We also look for a sitemap, because the shape of a site says something its prose does not. Candidates come from robots.txt first:

const robots = await tryFetch(`${origin}/robots.txt`);
if (robots?.status === 200) {
  for (const m of robots.body.matchAll(/^\s*sitemap:\s*(\S+)/gim)) candidates.push(m[1]);
}
candidates.push(`${origin}/sitemap.xml`, `${origin}/sitemap_index.xml`);

A sitemap index gets its children sorted by how likely they are to describe the business:

const sitemapPreference = (url: string) =>
  /page|product|collection|categor|service/i.test(url) ? 0
    : /post|blog|news|image|video|author|tag/i.test(url) ? 2 : 1;

Three children at most. A shop's product sitemap tells you what they sell; their news sitemap tells you they have a blog.

Then the URLs are summarised rather than listed. Group by first path segment, show the 40 biggest sections with their counts, and sample up to 150 paths round robin across sections, one from each in turn:

for (let i = 0; sample.length < SITEMAP_PATH_SAMPLE && lists.some((l) => i < l.length); i++) {
  for (const list of lists) if (i < list.length && sample.length < SITEMAP_PATH_SAMPLE) sample.push(list[i]);
}

Round robin rather than first 150, because a 4,000 URL sitemap sorted by section would otherwise spend the whole sample inside /blog/.

The 40 section cap has a specific origin, recorded in the comment:

// The biggest sections only: a site with its posts at the top level
// (/post-slug/) has a "section" per URL, thousands of lines that say
// nothing and are sent again with every proposal written from the research.

A WordPress site with pretty permalinks at the root produces one "section" per post. Without the cap the summary becomes thousands of lines of /some-post-title: 1 URLs, which is both noise and, again, a cost paid on every later generation.

The nice to haves are allowed to fail

The sitemap and the brand's logo both run under a timeout, and both resolve to null rather than throwing:

const logo = withTimeout(findLogo(home.icons), LOGO_MS, () => new Error("Logo timed out")).catch(() => null);
const sitemap = await withTimeout(readSitemap(origin), SITEMAP_MS, () => new Error("Sitemap timed out")).catch(() => null);

Eight seconds each. A research run that fails because a sitemap was slow is a worse product than one that quietly has no sitemap summary, and the only way to make sure that stays true is to make the failure path the cheap one.

The logo promise is also deliberately not awaited there. It is started before the sitemap, awaited at the very end, and the key pages are fetched in between, so the logo fetch overlaps everything else rather than adding to the total.

What it is all for

The output is a list of pages with their text plus a sitemap summary, and that is the entire input to the brief. Everything above is in service of one idea: the quality of what a model writes about a business is decided almost entirely before the model is called, by what you chose to put in front of it.

If you want to see what comes out the other end, the brief section of how Nakodo works describes it in product terms, and our brief template guide is the version you would write by hand for a creator. Research runs are metered per plan, 3 a month on Free up to 150 on Business, which the pricing table lists alongside everything else that is counted, because a crawl of somebody else's website is not a free operation for either of us.