DEV Community

Daniel Pertu
Daniel Pertu

Posted on

One page on our site holds 461 of its 516 URLs, 154 KB of HTML, and its biggest section is the condition we shipped last

Munchable's website is 516 URLs, and 461 of them are one page each answering one question about one ingredient for one gut condition. Pages like that have a structural problem: they are only ever going to be entered from search, each one is thin on its own, and a crawler has no path through them unless you build one.

So there is one page whose entire job is to be that path. I measured what it actually is, in production:

content-length: 154,582 bytes
hrefs:          486 (485 unique)
answer links:   461, every one unique
sections:       8
Enter fullscreen mode Exit fullscreen mode

461 of the site's 516 URLs are reachable in one hop from that single document. It is the largest page on the site by HTML size, ahead of the homepage at 98 KB, the recipe index at 92 KB and the condition index at 83 KB. An individual answer page is 41 KB.

No pagination, no search, no filter

The page is eight <section> elements, one per condition, each a heading linking to that condition's guide, a sentence, and a <ul>. That is the whole component.

The received wisdom says a list of 461 items wants pagination, or a client-side filter, or at minimum a search box. I built none of it, and the reasoning is that the two audiences for this page both want the opposite of those things.

A crawler wants one fetch. Paginate into 20 pages and you have turned one hop into two, plus 19 more URLs to get crawled, plus a decision about whether those pages should be indexed, plus rel=next markup to maintain. A filter is worse: links behind client-side state are links that are not in the HTML.

A reader has one condition, not eight. They do not scan 461 questions, they find their heading and read 60 lines. The section for their condition is their shortlist, and the sections above it are scroll. A search box would be a worse version of Ctrl+F, which their browser already ships and already works with no JavaScript.

The one thing that would break is a list of 461 items that is not grouped. Grouping is what makes "find your heading" possible, and the grouping is by condition rather than alphabetically across the lot because nobody arrives wanting to know about onions in general.

The section sizes are a map of the product

Here is every section on the page, with its link count, in the page's own order:

Section Links
IBS and low FODMAP 64
GERD and reflux 44
Lactose intolerance 52
Bile acid malabsorption 67
Gastroparesis 34
SIBO 64
IBD, Crohn's and colitis 48
Coeliac disease 88

Sum: 461, with no page appearing twice, because each page is one ingredient and condition pair and therefore belongs to exactly one section by construction.

The order is not by size and it is not alphabetical: it comes from the registry of condition guides, so this page's running order and the condition index can never disagree. Which is how the newest condition ends up last on the page while being the biggest section on it, by a margin of 21 links over the next one.

That last row is the part I did not predict. Coeliac disease was the eighth condition added to this product, it is the most recent, and it produced more answer pages than any of the seven before it. There is a mundane reason, which is that the vocabulary of a single grain on a label is unusually large, and the words are unusually indirect. I ended up giving that vocabulary its own reference page because 88 individual question pages are the long tail and a person who wants the whole list wants one page.

Sections also cannot be empty. The build filters them:

.filter((g) => g.questions.length > 0)
Enter fullscreen mode Exit fullscreen mode

A condition with nothing to say about any ingredient gets no heading at all, rather than a heading over an empty list. That matters more than it sounds: a heading promises content underneath it, and a page that keeps its promises is a page you can add a condition to without reviewing the layout.

The anchor text is generated by the thing that generates the URL

Every one of those 461 links reads as a full question: "Is onion low FODMAP?", "Does chocolate cause reflux?". Not the ingredient name, not a truncated label. The whole question, which is also the target page's <h1>, which is also its <title>, which is also its slug.

That is not discipline, it is a single function. One phrasing registry per condition owns the slug, the heading, the title tag and the way this index renders the link:

<Link href={`/${page.question}`}>
  {QUESTION_PHRASING[guide.condition].ask(lowerName(entry.name))}
</Link>
Enter fullscreen mode Exit fullscreen mode

Phrasing is per condition rather than one template across all of them, because what people type differs: "low FODMAP" for IBS, "high in lactose" for lactose intolerance, "cause reflux" for GERD. Across the 461 slugs that comes out as 417 beginning is- and 44 beginning does-.

The payoff for a site this shape is that anchor text, heading and title cannot drift apart, because there is no second place where any of them is written down. The usual failure here is an index that was hand-maintained once and now describes a page that has been reworded twice.

What the page is actually holding up

Each answer page carries a block of related links to its siblings, capped at twelve, chosen alphabetically. When I counted the live link graph, that cap turned out to produce twelve hubs per condition and a long tail: 128 of the 461 pages have no inbound link from anywhere else on the site.

Which means this one page is the only internal link 128 pages have. Delete it and 128 URLs are reachable only from the sitemap, which is a hint rather than a path, and in my experience the difference between a page that gets crawled on a schedule and a page that gets crawled eventually.

For 154 KB, that is a good trade. It is also a dependency worth writing down, because "the index page" sounds like a convenience and is in fact load-bearing.

The one thing I would change

The page's <h1> is "Ingredient answers". Its <title> is "Is this ingredient OK for my gut condition?".

Both are deliberate: the title tag is the phrasing that matches a search, the heading is the phrasing that works as a label when you have arrived. But the child pages' breadcrumbs also say "Ingredient answers", and the sitemap knows it as /answers, so this page now has three names depending on where you are standing. I have just written about the same problem in our breadcrumbs, which suggests it is not a coincidence but a habit.

If you have a generated section on your site, the audit worth running is this one: fetch the index, count the links, and compare that count against the number of URLs in the sitemap for the same section. Mine matched exactly, 461 against 461, which is the answer you want and not the answer you should assume. Here is the page to try it on: munchable.app/answers.

Top comments (0)