DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Twelve links chosen alphabetically on all 461 of our generated pages, and the 128 pages nothing else points to

Munchable tells you whether a packaged food fits the gut conditions you manage. Alongside the app there is a set of generated pages that answer one ingredient question each, because "is malt extract gluten free" is a thing people type into a search box at 7pm in a supermarket. Those pages are at munchable.app/answers, the condition guides behind them are at munchable.app/conditions, and a typical one is munchable.app/is-soy-sauce-gluten-free.

There are 461 of those question pages. This week I fetched all 461 from production and counted what links to what. The answer was not what the code looked like it was doing.

The shape of the site

The live sitemap has 516 URLs:

417  /is-...          question pages
 44  /does-...        question pages
 38  /recipes/...
  8  /conditions/...
  9  fixed pages (/, /answers, /recipes, /conditions, /about, legal)
Enter fullscreen mode Exit fullscreen mode

The question pages are one dynamic segment at the site root, which has its own hazard: without dynamicParams = false the route answers every unmatched top-level path, and every typo on the domain becomes a soft 404 a crawler will happily index. With it, the generated slugs exist and everything else is a real 404.

Each page is built from one template, and the template puts three kinds of internal link on it: a breadcrumb back to the index, a link to the condition guide, and two "related" blocks.

The index exists because a sitemap is not enough

The index at /answers is explicit about its job:

/**
 * The index for the answer pages. Its job is reachability: pages that are only
 * linked from a sitemap get crawled slowly and ranked poorly, so they need a real
 * page linking to them from inside the site, grouped the way a reader thinks.
 */
Enter fullscreen mode Exit fullscreen mode

It groups by condition rather than alphabetically across the lot, for the reason written next to it: a reader has one condition, so the group is the shortlist of what to watch for with it. The live page emits 478 internal links in 154,570 bytes of HTML, and 461 of those links are the question pages. Every one of them is reachable in two clicks from the home page.

Each condition guide does the same for its own subset: 88 links from one, 67 from another, 64, 64, 52, 48, 44, 34. So the floor for any question page is two inbound links, the index and its guide.

The two related blocks

The interesting part is the per-page "related" links, because those are the ones that are supposed to distribute authority through the set rather than funnel it from one index.

The first block is the same ingredient under the site's other conditions, which is genuinely per-page and usually one to three links. The second block is other ingredients under the same condition:

export function alsoFlaggedFor(
  condition: ConditionId,
  exclude: string,
  limit = 12,
): IngredientEntry[] {
  return INGREDIENTS.filter(
    (e) => e.slug !== exclude && ingredientFacts(e).some((f) => f.condition === condition),
  ).slice(0, limit);
}
Enter fullscreen mode Exit fullscreen mode

Filter the ingredient list to the ones this condition has something to say about, drop the current page, take twelve.

Read that again as a link graph rather than as a list. INGREDIENTS is built once and the builder ends with a sort:

.sort((a, b) => a.name.localeCompare(b.name));
Enter fullscreen mode Exit fullscreen mode

So the list is in alphabetical order by display name, and .slice(0, limit) returns the same twelve ingredients for every page of a given condition, except on the twelve pages that have to exclude themselves.

What production actually looks like

Measured, not reasoned about. All 461 pages fetched, links extracted, counted alongside the index, the eight guides and one reference page:

outbound question links per page   min 12, max 15, mean 12.6
                                   248 pages have exactly 12

inbound question links per page    min 2, max 92, mean 14.8, median 3
                                   128 pages have exactly 2
                                   194 pages are linked by no sibling page at all
Enter fullscreen mode Exit fullscreen mode

The top of the inbound table is four pages with 90 or more: barley at 92, beer at 91, ale and baguette at 90. The bottom is 128 pages whose only two inbound links are the index and their own condition guide, and the mean of 14.8 against a median of 3 is the signature of exactly that: a handful of enormous hubs dragging the average up over a long flat tail.

Per condition, the number of distinct "other ingredients" blocks is thirteen. Not 88, not 64. Thirteen, every time, and for a satisfying reason: twelve pages are in their own top twelve so they each get a set with themselves swapped out for the thirteenth name, and every remaining page in that condition gets one identical block. For the coeliac set, 76 of the 88 pages carry the same twelve links, which begin: baguette, barley, barley flour, barley malt extract, barley malt vinegar, beer, biscuit, bread, breadcrumbs, bretzel.

That is the alphabet. The link graph of a 461-page content set was decided by the alphabet.

Why it is wrong, precisely

Every one of those links is correct. Every one is useful to a reader, who gets twelve other things worth knowing about under the condition they already care about. Nothing is broken, nothing 404s, and the pages pass every test in the suite.

What is wrong is the distribution. Internal links are the main way a crawler finds and weighs a page in a set this size, and the current shape says: these twelve pages per condition matter enormously, and the other several hundred are leaves hanging off one index entry. For a set where the whole point is the long tail, that is backwards. The pages with the thinnest search competition are exactly the ones with the thinnest internal support.

It is also a wasted slot. Twelve outbound links across 461 pages is 5,532 link slots, and they currently reach 104 distinct destinations: thirteen per condition, eight conditions.

The other block, the same ingredient under the site's other conditions, is the one quietly doing the real work. It is one to three links per page rather than twelve, it is genuinely different on every page, and it reaches 213 distinct destinations. A twentieth of the link budget, twice the coverage.

The fix, and the constraint that shapes it

Three options, in increasing order of how much thinking they need.

Rotate deterministically. Instead of slice(0, 12), start the window at an offset derived from the current page's own slug, modulo the list length, and wrap. Still pure, still static, still the same twelve-item block on screen, and the inbound links spread almost evenly across the set because every page starts from a different point in the list. It is a four-line change.

Pick neighbours. Choose the twelve by closeness to the current ingredient: same family, same label vocabulary, same aliases. Better for the reader, more work, and it needs an ordering notion the data does not currently carry.

Give every condition a reference page. One of the eight already has one: munchable.app/hidden-gluten-ingredients is a reference list that links out to 88 question pages with its own prose around them, and it is one reason that subset dominates the inbound table. A reference page is a real hub rather than a list of twelve repeated 88 times, and it earns its own search traffic rather than only passing traffic along.

The constraint on all three is a sentence in the template that I am not willing to break:

{/* Not billed as substitutes. The engine reasons about what is printed
    on a label, not about cooking, so it cannot honestly say one
    ingredient replaces another in a recipe. */}
Enter fullscreen mode Exit fullscreen mode

Whatever picks the twelve, the block cannot imply "use this instead". It can only say "these are other things worth checking", because that is the only claim the data supports. A naive "related ingredients" recommender would quietly turn a factual block into cooking advice nobody is qualified to give.

I am doing the rotation first, because it is the one that is objectively better than the alphabet at no cost, and then the reference pages one condition at a time.

Two things worth copying

First, measure the graph from production rather than from the source. Reading alsoFlaggedFor tells you it takes twelve things; fetching 461 pages tells you that .slice and a sorted list conspired to make twelve hubs. I had read that function several times and never seen it.

Second, the number to look at is the median inbound link count, not the mean. Mean 14.8 looks like a healthily interlinked site. Median 3 is the truth.

If you want to see the set, start at munchable.app/answers and click any question, then look at the two related blocks at the bottom of the page and open the same section on a sibling page. You will see the same twelve names. By the time you read this they may have started to differ, which would mean the rotation shipped.

Top comments (0)