Disclosure: I work on Keyserp. This article was generated by an AI agent at the account owner’s request, using the design of our internal article-monitoring tool as context. It has not received independent human editorial review. It is a design note, not a claim about search rankings or a public product release.
The boolean that loses information
A published-page checker seems like a simple function: fetch a URL and return true or false. But a timeout, a login wall, and a missing article are three different observations. Mapping all three to false gives an operator the wrong next action.
Our internal workflow tracks whether an external article remains readable and whether its expected reference is present. The important design decision was to preserve uncertainty instead of hiding it behind a green or red badge.
Keep observations separate from conclusions
A useful result has at least these fields:
- requested URL and final URL after redirects
- checked-at timestamp
- transport outcome and HTTP status, if available
- whether the article body was identified
- whether an article-specific text marker matched
- whether the expected link was found inside that body
- a reason for any unknown result
Use null for a link that could not be checked. Reserve false for a completed inspection that did not find it. A missing observation should not silently become negative evidence.
A small decision table
| Observation | Classification | Next action |
|---|---|---|
| Timeout or access denial | Unknown | Retry later within the site's limits |
| Successful response showing a login page | Unknown | Check permitted access manually |
| Expected article body and expected link | Verified at this time | Schedule the next check |
| Expected article body, complete inspection, no expected link | Link not found | Review the article |
| 404 or 410 | Missing response observed | Recheck before escalating |
| Body selector no longer matches | Unknown | Review the site adapter |
These are illustrative fixtures, not measured incident counts. A repeated missing response may justify a removal-suspected state; it still does not prove why the publisher removed anything.
Link matching deserves its own contract
A domain appearing somewhere in the HTML is weak evidence. Navigation, comments, related articles, and advertising can all contain links outside the author's text.
Scope the inspection to the identified article body. Resolve relative URLs against the final page URL. Then compare the parsed destination with the explicitly configured target. Do not use substring matching: an unrelated hostname can contain your domain as text.
Decide deliberately whether trailing slashes, query parameters, or fragments are significant for each target. Do not strip every query parameter by default; some applications use them to identify the resource. If a publisher wraps links through a redirect service, report that wrapper as an observation and review it separately instead of declaring a direct match.
Store the observed anchor text and rel attribute as evidence. Finding a link does not establish that a search engine indexed it or assigned it value.
Tests that challenge the happy path
Before connecting a classifier to live pages, I would give it fixtures that deliberately look almost correct:
- A 200 login page with the expected domain in its footer.
- The correct article with the target link only in a reader comment.
- The correct article with a relative link that resolves to a different destination.
- An article shell whose body is rendered only after JavaScript runs.
- The expected article with no link after a complete body inspection.
- A transient missing response followed by a successful check.
The expected outcome for the first four is unknown or a narrowly described mismatch, depending on what the adapter actually inspected. It should never be verified merely because a string was found.
Fixture tests establish classifier behavior. They do not establish that a live publisher's current markup is supported. Record those two checks separately.
A checker should not become a publisher
An unknown or missing result should create a review task, not automatically repost the article. Otherwise a temporary outage can turn into duplicate content and unwanted community activity.
The same boundary helps with alerting: keep the last verified observation visible alongside the latest failed attempt. That tells an operator what was known, when it was known, and what changed.
Our tool is still internal, and long-term resilience to layout changes remains an open operational question. The practical lesson is to design the unknown state before automating the next action.
Affiliation: Keyserp. I have an operational interest in the service mentioned here.
Top comments (0)