Our link audit called two citations broken, and curl proved it right and wrong at the same time

CogniPrep has a page for the PELLETB, the written exam California law enforcement agencies use. Part of that page is a table of agencies and what each one actually requires, and every row links to the agency's own page as its source. Claims about employers without a link to the employer saying it are worth nothing, so the link is the point of the row.

A site audit flagged two of those links as broken. Both opened fine in my browser.

Reproduce it in three commands

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "AhrefsBot/7.0; +http://ahrefs.com/robot/" \
  "https://careers.sf.gov/role/?id=3743990014069426"

That returns 403. The same URL with a Chrome user agent returns 200, and with Googlebot's user agent it also returns 200. The other flagged link, https://www.modestogov.com/1539/Post-Pellet-B-Testing, returns 404 to AhrefsBot and 200 to both of the others. Those are the numbers I got on 5 October 2026, and you can get them yourself in under a minute.

So the audit tool was reporting exactly what it saw. The pages are not broken; they are closed to one class of client. Plenty of government and enterprise sites allow Googlebot through a bot management layer and refuse everything else that identifies itself as a crawler, and a 404 rather than a 403 is the more annoying version of that, because it looks like a dead page rather than a refusal.

The easy response is to mark both findings as false positives and move on. We did something slightly more useful.

A broken-link report is also a "still says what you claimed?" report

An audit finding you cannot verify through the tool is still a prompt to go and read the page. And that is where the real problem was, because one of those two citations was not the right kind of page in the first place.

careers.sf.gov/role/?id=3743990014069426 is a job posting. Its URL is an id in a query string, it exists while a vacancy is open, and it says what that vacancy required. joinsfpd.com/written-test is the department's standing page about the written test. Both are first-party. Only the second one is still going to be there next year, and only the second one is making a claim about the process rather than about one opening.

Same for Modesto: the replacement is the live recruit posting on the jobs platform the city actually uses, which answers 200 to all three user agents above.

The part that cost us something

Here is the bit I think is worth the post. The old SFPD posting supported this sentence on our page:

Pass/fail. PELLETB or FrontLine National; a T-score of 42 or higher from the last two years is accepted, and a bachelor's degree may waive the exam.

The replacement page states the pass/fail choice between the two exams and the two-year T-score 42 exemption. It does not mention a degree waiver. So the sentence on the live page is now:

Pass/fail. PELLETB or FrontLine National; a T-score of 42 or higher from the last two years is accepted.

The waiver may well still be real; some other SFPD document may say so. That is not the standard. The standard is that the source we link from that row has to state the thing the row claims, and when you swap the source you have to re-check every clause against the new one and delete whatever it does not support.

Swapping a citation is not a cosmetic change. It silently re-scopes every claim attached to it, and the only way that goes well is if changing the link and re-reading the claim are the same task.

Both replacements went into the provider dossier with the reasoning written down, including the crawler behaviour, so the next person to run an audit does not rediscover it from scratch:

S33 and S34 are the links the page shows for SFPD and Modesto. careers.sf.gov (S19) and modestogov.com (S21) block SEO crawlers (403 and 404 to AhrefsBot, 200 to browsers and Googlebot), so site audits report them as broken.

That dossier is a markdown file per provider with numbered sources, each with a date and a confidence rating, and claims on the public page refer back to them. PELLETB's has 34 numbered sources in 124,751 bytes. There are 39 of these files and they come to 4,278,105 bytes, which is more prose about sources than there is prose on the pages, and that ratio is deliberate for a product whose pages tell people what an employer will make them sit.

Three things I would now check by default

1. When a link checker reports a 4xx, re-request the URL with a browser user agent and with Googlebot's before believing it. The three answers together tell you whether you have a dead page, a bot-managed page, or a real problem.
2. Prefer a standing process page over a dated posting when citing an organisation, even when the posting is more specific. A citation whose URL contains a vacancy id has an expiry date.
3. Treat a changed source as a changed claim, and read every sentence that leans on it.

See it

Open cogniprep.app/games/pellet-b and scroll to the agency table. Every row has an "Agency source" link, and the SFPD and Modesto ones are the two replacements above. Click them, read what they say, and compare it to the sentence in the row. If a row claims something its link does not state, that is a bug in our page and worth telling us about.

The rest of the provider pages work the same way, and they are listed at cogniprep.app/games.