Empty Is Not a State

Your poller returned None.
What happened?

Maybe the source published nothing new. Maybe the server went down. Maybe the endpoint returned HTML instead of RSS and the parser quietly produced zero items. Maybe the credentials expired. Maybe the API retired the endpoint. Or maybe you do not have enough evidence to know yet.

These are different facts. They require different retry policies, different alerts, and different state transitions. Yet many production systems collapse all of them into one value:

result = fetch(source)

if result is None:
    source.done = True

This is not mainly a logging problem. It is a control-flow problem.

The moment None is allowed to drive a state transition, the system destroys the distinction between absence, failure, and uncertainty. A temporary outage can permanently stop monitoring. A parser regression can look like a quiet feed. An expired credential can trigger endless retries against an endpoint that is perfectly healthy.

The fix is not complicated. Record what happened, interpret it separately, and only mutate durable state when the evidence supports the transition.

Observation. Adjudication. Verdict.

One empty result, at least five different facts

Consider a scheduled job polling an external feed.

The first possibility is genuinely boring: the request succeeded, the parser succeeded, and the source contains no item newer than the last one you processed. The correct action is to update the last successful check and poll again normally.

The second is temporary unavailability. DNS failed, the connection was refused, or the server returned 503. That needs backoff and eventually an alert. It is not evidence that the source is finished.

The third is a contract failure. The server returned 200 and some bytes, but the parser could no longer extract what it previously extracted. Perhaps the XML changed. Perhaps a CDN returned an HTML challenge page. Perhaps the API schema moved. Retrying the same parser every fifteen minutes will not repair it.

The fourth is an access or capacity limit. 401, 403, and 429 do not describe the content. They describe your ability to retrieve it. The response may require new credentials, a slower request rate, or a quota decision.

The fifth is the one production code tends to resist: you do not know. The response is ambiguous, the source has no history, or the available signals contradict each other. Undetermined is not an implementation failure. It is a legitimate state in the model.

All five can arrive at the caller as None. That is the design smell.

The fetcher should not decide what the fetch means

The boundary component should record what it observed without trying to convert it into business meaning.

from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class Observation:
    source_id: str
    observed_at: datetime
    http_status: int | None
    transport_error: str | None
    content_type: str | None
    bytes_received: int
    items_parsed: int | None
    parse_error: str | None
    newest_item_date: datetime | None
    etag: str | None

Every attempt produces an Observation, including the failed ones. A timeout is data. A 200 response containing 48 KB of HTML is data. A parser that never ran is different from a parser that ran successfully and found zero items.

This distinction is easy to erase with exception-driven code:

try:
    return parse(fetch(url))
except Exception:
    return None

That function has converted several independent failure domains into the same value. By the time the scheduler receives it, the useful information is already gone.

Empty is only readable against history

A single observation is often insufficient. A 200 response with zero parsed items can be normal for one source and a strong regression signal for another.

You need a baseline.

@dataclass(frozen=True)
class Baseline:
    last_success_at: datetime
    newest_item_date: datetime | None
    typical_item_count: int | None
    expected_content_type: str | None
    etag: str | None = None

Suppose a feed has returned between 15 and 25 items on every poll for six months. Today it returns 200, 42 KB of data, text/html, and zero parsed items. Calling that “no new content” would be an impressive act of optimism.

Now take the same response from a source you have never seen before. It may be broken, or it may simply not be a feed. You cannot infer a contract regression because you have no known contract to compare against.

Same response. Different evidence. Different verdict.

This is why history is not just useful metadata. It changes what can be concluded from the current observation.

A verdict is not another boolean

The adjudicator combines the observation with the baseline and produces a result the scheduler can act on.

from enum import StrEnum


class Outcome(StrEnum):
    ITEMS_FOUND = "items_found"
    NO_NEW_ITEMS = "no_new_items"
    UNAVAILABLE = "unavailable"
    CONTRACT_FAILURE = "contract_failure"
    ACCESS_LIMITED = "access_limited"
    ENDPOINT_GONE = "endpoint_gone"
    UNDETERMINED = "undetermined"


@dataclass(frozen=True)
class Verdict:
    outcome: Outcome
    observation: Observation
    baseline: Baseline | None
    reasons: tuple[str, ...]

    @property
    def is_settled(self) -> bool:
        return self.outcome in {
            Outcome.ITEMS_FOUND,
            Outcome.NO_NEW_ITEMS,
        }

I deliberately did not put confidence: 0.83 in this version.

Unless those scores are calibrated against labelled outcomes, confidence is usually just intuition wearing a decimal point. A deterministic classifier can still express uncertainty without inventing measurement. It can return UNDETERMINED, preserve its reasons, and wait for more evidence.

If you have historical data and can demonstrate that verdicts emitted at 0.8 are correct roughly 80% of the time, add calibrated confidence. Until then, evidence is more useful than decorative precision.

The adjudicator

The rules do not need to be clever. They need to preserve distinctions that affect downstream behaviour.

def adjudicate(
    obs: Observation,
    baseline: Baseline | None,
) -> Verdict:
    def verdict(outcome: Outcome, *reasons: str) -> Verdict:
        return Verdict(outcome, obs, baseline, reasons)

    if obs.transport_error:
        return verdict(
            Outcome.UNAVAILABLE,
            f"transport error: {obs.transport_error}",
        )

    if obs.http_status in {401, 403, 429}:
        return verdict(
            Outcome.ACCESS_LIMITED,
            f"HTTP {obs.http_status}",
        )

    if obs.http_status in {404, 410}:
        return verdict(
            Outcome.ENDPOINT_GONE,
            f"HTTP {obs.http_status}",
        )

    if obs.http_status is not None and obs.http_status >= 500:
        return verdict(
            Outcome.UNAVAILABLE,
            f"HTTP {obs.http_status}",
        )

    if obs.http_status == 304:
        return verdict(
            Outcome.NO_NEW_ITEMS,
            "server confirmed that the representation was not modified",
        )

    if obs.http_status != 200:
        return verdict(
            Outcome.UNDETERMINED,
            f"unhandled HTTP status: {obs.http_status}",
        )

    if obs.parse_error:
        return verdict(
            Outcome.CONTRACT_FAILURE,
            f"parser failed: {obs.parse_error}",
        )

    if obs.items_parsed == 0:
        if baseline is None:
            return verdict(
                Outcome.UNDETERMINED,
                "zero items with no historical baseline",
            )

        if baseline.typical_item_count and baseline.typical_item_count > 0:
            return verdict(
                Outcome.CONTRACT_FAILURE,
                "source historically contained items",
                "current parser returned zero",
            )

        return verdict(
            Outcome.NO_NEW_ITEMS,
            "zero items is consistent with the baseline",
        )

    if obs.items_parsed and baseline is None:
        return verdict(
            Outcome.ITEMS_FOUND,
            "first successful observation",
        )

    if (
        obs.newest_item_date
        and baseline
        and baseline.newest_item_date
        and obs.newest_item_date <= baseline.newest_item_date
    ):
        return verdict(
            Outcome.NO_NEW_ITEMS,
            "newest parsed item does not advance the watermark",
        )

    if obs.items_parsed:
        return verdict(
            Outcome.ITEMS_FOUND,
            f"parsed {obs.items_parsed} items",
        )

    return verdict(
        Outcome.UNDETERMINED,
        "observation did not match a known rule",
    )

There is nothing particularly advanced here. That is a feature.

The value is in making the failure taxonomy explicit. Once the system can name a failure mode, it can attach the correct policy to it. Before that, every retry strategy is guessing.

Different verdicts need different policies

The scheduler no longer reacts to “empty.” It reacts to a specific outcome.

def next_action(verdict: Verdict) -> str:
    return {
        Outcome.ITEMS_FOUND:
            "ingest and advance the baseline",
        Outcome.NO_NEW_ITEMS:
            "record success and poll normally",
        Outcome.UNAVAILABLE:
            "retry with exponential backoff; alert after a threshold",
        Outcome.CONTRACT_FAILURE:
            "quarantine the result; alert the parser owner",
        Outcome.ACCESS_LIMITED:
            "apply rate-limit or credential recovery policy",
        Outcome.ENDPOINT_GONE:
            "confirm across repeated observations; then disable",
        Outcome.UNDETERMINED:
            "preserve state and collect another observation",
    }[verdict.outcome]

Notice that even ENDPOINT_GONE does not immediately delete or permanently complete the source. A single 404 can come from a bad deployment, a routing error, or an eventually consistent configuration change. The verdict is strong enough to change behaviour, but destructive state transitions should still require repeated evidence or human confirmation.

This is the difference between classification and governance. The classifier says what the current evidence most strongly supports. The policy decides what the system is authorised to do about it.

Do not let uncertainty rewrite history

The baseline influences future decisions, so updating it is not an innocent write. A bad update changes how later observations will be interpreted.

Only settled content outcomes should advance it.

def advance(
    baseline: Baseline | None,
    verdict: Verdict,
) -> Baseline | None:
    if not verdict.is_settled:
        return baseline

    obs = verdict.observation

    return Baseline(
        last_success_at=obs.observed_at,
        newest_item_date=(
            obs.newest_item_date
            or (baseline.newest_item_date if baseline else None)
        ),
        typical_item_count=(
            obs.items_parsed
            if obs.items_parsed and obs.items_parsed > 0
            else (baseline.typical_item_count if baseline else None)
        ),
        expected_content_type=(
            obs.content_type
            or (baseline.expected_content_type if baseline else None)
        ),
        etag=obs.etag or (baseline.etag if baseline else None),
    )

An unavailable source does not become the new normal. A parser failure does not reset the typical item count to zero. An ambiguous first poll does not establish a baseline that will make the second ambiguous poll look valid.

That last failure is particularly unpleasant. Once an uncertain observation is written as truth, later logic uses the corrupted history as evidence. The system does not merely remain wrong. It manufactures confirmation for its own mistake.

What the cases look like

These fixture cases exercise the decisions that used to share one return value:

Case	Observation	History	Verdict	Action
First valid poll	200, 20 items	None	items_found	Ingest and establish baseline
No newer content	200, 20 known items	Existing watermark	no_new_items	Record success and continue
Explicitly unchanged	304	Existing ETag	no_new_items	Poll normally
HTML instead of a known feed	200, bytes, parser error	Previously valid	contract_failure	Quarantine and alert
Zero items from a known active feed	200, zero items	Typical count: 20	contract_failure	Preserve baseline and investigate
Zero items from an unknown source	200, zero items	None	undetermined	Preserve state and observe again
DNS failure	Transport error	Any	unavailable	Backoff and thresholded alert
Rate limited	429	Any	access_limited	Respect retry policy
Retired endpoint	410	Any	endpoint_gone	Confirm, then disable

Rows five and six are the important pair. The current response is effectively identical. History changes what the system is justified in concluding.

A boolean cannot express that difference. Neither can None.

Five rules worth keeping

The implementation can change. The constraints should not.

First, the fetcher records and the adjudicator interprets. Mixing the two makes transport behaviour, parsing behaviour, and business policy impossible to test independently.

Second, every attempt produces an observation. Exceptions should enrich the record, not erase it.

Third, historical claims require history. Without a baseline, “changed” is not a conclusion you are entitled to make.

Fourth, uncertainty must be representable. If the type system only permits success or failure, ambiguous evidence will be forced into one of them.

Fifth, an uncertain or failed observation must not rewrite the state used to judge future observations. Durable state transitions need stronger evidence than temporary scheduling decisions.

This is not really about feeds

Pollers make the problem easy to see, but the same mistake appears everywhere.

A search returning no documents may mean there were no matches, the index is stale, a tenant filter was wrong, or the retrieval service failed. A queue consumer receiving no messages may mean the queue is empty, visibility is delayed, permissions changed, or the broker is unavailable. A monitoring query returning no datapoints may mean zero traffic, broken instrumentation, or a dead collector.

In each case, absence is an observation. It is not yet an explanation.

This matters even more in AI systems, where probabilistic components are often allowed to collapse ambiguous evidence into confident language. The governance layer around them should do the opposite. It should preserve distinctions, expose uncertainty, and restrict which conclusions are allowed to mutate state.

The implementation here is small. A few dataclasses, one explicit taxonomy, and a deterministic policy layer. The difficult part is resisting the convenience of pretending that None means one thing.

It never did.

Comments

[Ospilak_Ocram_AI]

"Absence is an observation. It is not yet an explanation." Great point! AI need those kind of stuff!