DEV Community

Daniel Pertu
Daniel Pertu

Posted on

I counted our link graph from the source five days ago, then from the live HTML, and the worst page is a different one

Five days ago I published Fifteen pages, 45 sibling links, and the one page no sibling links to. That post counted the internal link graph of Notifio's marketing site by reading the typed catalogues in the repository: every per-site page names three siblings, the counting took a dozen lines of TypeScript, and it found one page, /alerts/facebook-marketplace, that no sibling points at.

That was a source-level audit, and a source-level audit can only see the links that come from the thing it reads. The nav does not come from that array. Neither does the footer, the hub grids, the breadcrumbs, the home page, or any sentence with a link typed into it.

So this morning I counted the other way: fetch all 38 URLs out of the sitemap, pull every internal href from the rendered HTML, and tally how many of the 38 pages link to each one. Forty lines of Python, no repository access needed, and anybody can run it against the live site.

It found a worse page, and it is not facebook-marketplace.

The shape of the graph is a cliff

in-degree distribution (how many of the 38 pages link to each page):

  38 inbound  ->  15 pages
   7 inbound  ->   3 pages
   6 inbound  ->   2 pages
   5 inbound  ->   7 pages
   4 inbound  ->   7 pages
   3 inbound  ->   3 pages
   1 inbound  ->   1 page

orphans (in-degree 0): none
Enter fullscreen mode Exit fullscreen mode

No orphans, which is the first thing I was checking and the thing I am relieved about. Every URL in the sitemap can be reached by following a link from some other page in the sitemap.

Everything else about that distribution is a cliff. Fifteen pages are linked from every single page on the site. Then nothing at all between 7 and 38. Then twenty-three pages clustered between 3 and 7.

The fifteen are not a mystery. They are the ten top-level routes, plus five of the fifteen per-site alert pages:

38 inbound:  /  /about  /alerts  /compare  /download  /guides  /help  /pricing  /privacy  /terms
             /alerts/funda  /alerts/kamernet  /alerts/pararius  /alerts/rightmove  /alerts/spareroom
Enter fullscreen mode Exit fullscreen mode

Those five are the footer's column of rental sites, which is hand-picked rather than imported from the catalogue because importing all fifteen pulled 87KB of page content into every route. I knew about that trade. What I had not looked at is what it does to the graph: being in the footer is worth 38 inbound links, and not being in the footer is worth between 4 and 7. There is no middle because nothing else on the site is sitewide.

The page with one inbound link

/compare/rentola  (in-degree 1)  <-  ['/compare']
Enter fullscreen mode Exit fullscreen mode

One. The comparison hub lists it, and that is the entire inbound link profile of that page on our own website. Its four siblings do not link it. No guide links it. No audience page links it.

For contrast, here is the page my source-level audit flagged last week:

/alerts/facebook-marketplace  (in-degree 4)  <-  ['/', '/alerts', '/for/amsterdam', '/guides/rental-scams-netherlands']
Enter fullscreen mode Exit fullscreen mode

Four. The source audit was right that no sibling points at it, and wrong, by omission, that this made it the worst-connected page. It gets picked up by the home page, the hub, an audience page and a guide. The rendered graph has link sources the typed related arrays know nothing about, and for that page they were enough.

/compare/rentola has none of those. And the reason I did not catch it is that my dozen-line audit counted the alerts catalogue and the articles catalogue, and never opened the comparison catalogue at all.

Five arrays, and every one of them omits the same slug

server/src/lib/compare/targets.ts has five targets, and each one hand-names three related slugs:

$ grep -n 'slug: "\|related: \[' server/src/lib/compare/targets.ts
 13:    slug: "rentbird",
 90:    related: ["rentslam", "stekkies", "visualping"],
 94:    slug: "rentslam",
171:    related: ["rentbird", "stekkies", "visualping"],
175:    slug: "stekkies",
252:    related: ["rentslam", "rentbird", "visualping"],
256:    slug: "visualping",
335:    related: ["rentbird", "rentslam", "stekkies"],
339:    slug: "rentola",
414:    related: ["rentbird", "rentslam", "stekkies"],
Enter fullscreen mode Exit fullscreen mode

Read the right-hand column. The word rentola does not appear in any of the five arrays, including its own, which is correct for its own. Rentola was the fifth target added, and adding it meant writing one new block, which I did, and editing four existing blocks, which I did not.

There is a second, quieter consequence in the same listing. rentola's array is ["rentbird", "rentslam", "stekkies"], which is character for character visualping's array one block above it. It was copied. So rentola does not link visualping either, and the sibling in-degrees come out like this:

Target Linked by siblings Should be
rentbird 4 4
rentslam 4 4
stekkies 4 4
visualping 3 4
rentola 0 4

Five pages, each showing three of the other four, maintained by hand. Twenty possible edges, seventeen present. One page gives three and receives nothing.

This is the one place that is not derived

What makes this worth a post rather than a commit is that the rest of this site is built specifically to prevent it. The sitemap file says so in its own header:

/**
 * The sitemap, derived rather than typed out.
 *
 * The per-site alert pages come straight from the catalogue in `lib/alerts`, so
 * adding a site is one edit in one file and it cannot be half-added: the page,
 * the hub grid, the footer column and this list all read the same array.
 */
Enter fullscreen mode Exit fullscreen mode

"It cannot be half-added." That is true of the page, the hub grid, the footer column and the sitemap, because all four read one array. It is not true of related, because related is the one field where a page states a relationship rather than being derived into one. And a relationship typed on one side is exactly the kind of thing that gets half-added.

The type does not help either, in the same way it did not help for the alerts catalogue:

related: string[];
Enter fullscreen mode Exit fullscreen mode

string[], not a union of the slugs that exist. A typo compiles and renders one fewer link. An omission compiles and renders nothing at all, because an omission is not a thing the compiler can see. There is no syntax for "this array is missing an element".

Three fixes, and only one of them is the right one

Derive it. With five targets and three slots, the arrays encode almost no information: four of the five are just "the other four, minus one, arbitrarily". COMPARE_TARGETS.filter((t) => t.slug !== target.slug) gives every page all four siblings, makes the sub-graph complete and symmetric, and deletes five lines of data that can only ever be wrong. The hand-written list buys the ability to order and curate them, which, looking at the five arrays above, nobody has ever actually used.

Or assert it. If the curation is worth keeping, the test is four lines: every slug must appear in at least one sibling's related, and the relation must be symmetric. That is the same test the alerts catalogue needs, which that post concluded with and which still does not exist. Writing it once for both catalogues is less work than the two posts I have now written about not having it.

Or add rentola to four arrays, which fixes today and schedules this post again for whenever a sixth target lands.

I am doing the first for /compare and the second for /alerts, where the three-of-fifteen curation genuinely is editorial.

What the two counts are each good for

The source-level count sees intent. It can tell you that an array is missing a slug, which is the actual defect, and it runs in milliseconds in CI.

The rendered count sees outcome. It cannot tell you why a page is poorly linked, but it is the only one of the two that knows the footer exists, and it is measuring the thing that actually matters, which is how many doors there are into a page.

I needed both, and I wrote the cheap one first and assumed it was the whole answer. If you want to check mine, every number above came from fetching the 38 URLs in our sitemap and counting href="/..." in the responses. The comparison hub is where the one-inbound-link page lives, and by the time you read this it may have four more.

Top comments (0)