A Green Scheduler Is Not Proof

A background-job dashboard usually answers a narrow question: did the scheduled method return, fail, or retry?

That is useful, but it is not the same as knowing whether the intended work completed. The difference matters most when a job catches an internal failure deliberately. Perhaps one phase failed after another phase had already succeeded. Perhaps retrying the entire job would duplicate work or create a retry storm. Returning normally can be the responsible execution decision while still representing a failed business outcome.

If the dashboard only sees the return value, it can be technically accurate and operationally misleading.

The green light that hid a failure

In recent committed work I reviewed, recurring jobs were given their own durable outcome evidence. The change was motivated by a concrete failure shape: a terminal phase could be caught and logged, the method could return, and the scheduler could therefore record success.

The lesson is not that the scheduler lied. It reported what it owned: execution state. The job owned a different fact: whether its useful work reached an acceptable outcome.

Those are two separate contracts:

• Execution contract: Was the job invoked? Did it return or throw? Was it retried?
• Outcome contract: Did the intended work complete, fail, or deliberately skip? What safe evidence supports that answer?

Trying to squeeze both contracts into one green or red scheduler badge loses information.

Use two independent witnesses

The stronger design keeps scheduler state and job-owned evidence separate, then cross-checks them.

At the last responsible boundary, the job writes a small durable record containing:

• a stable job key;
• start and completion times;
• a semantic outcome;
• a short detail suitable for operations.

The monitor reads that record alongside the scheduler's latest state. Neither source gets trusted blindly. Agreement is useful evidence; disagreement is itself a result.

That gives the dashboard three honest answers:

1. Verified good — fresh successful job evidence has no fresh scheduler failure contradicting it.
2. Verified bad — evidence reports failure, is missing or stale, the job is not scheduled or running, or the two sources contradict.
3. Unverified — a monitoring or collection source is unavailable, so no trustworthy cross-check is possible.

The distinction matters. Missing evidence is a failed health check; an unavailable source is unverified. Neither may silently become success. Empty evidence is not positive evidence.

Put evidence around the whole outcome

A generalised C# shape looks like this:

var startedAt = clock.GetUtcNow();

try
{
    var result = await RunWorkAsync();

    if (!result.Acceptable)
    {
        await evidence.TryWriteFailureAsync(startedAt, result.SafeSummary);
        return;
    }

    await evidence.TryWriteSuccessAsync(startedAt, result.SafeSummary);
}
catch (Exception exception)
{
    await evidence.TryWriteFailureAsync(startedAt, SafeSummary(exception));
    throw;
}

This is deliberately broader than wrapping only thrown exceptions. A caught terminal failure still needs failed evidence even when returning is the correct retry policy. A deliberate no-op or skip also needs an explicit, explainable result rather than disappearing into silence.

Keep the detail short and safe. It should help distinguish outcomes without copying personal data, secrets, request payloads, or sensitive identifiers into an operations table.

Evidence writes should be best-effort

There is an uncomfortable trade-off here. If the evidence store is unavailable, should an otherwise healthy job fail and retry?

Usually, no. Observability should not become an outage amplifier.

The committed design I reviewed treated evidence writes as best-effort: log the write failure through an independent path, do not fail the useful work, and let the missing record surface as a failed health check during the next cross-check. Reads, however, must fail honestly. If the monitor cannot inspect its sources, it should say so rather than displaying green.

This creates a useful asymmetry:

• the job remains resilient when telemetry storage fails;
• the dashboard remains sceptical when proof is absent.

Freshness belongs to the contract

Missing evidence only has meaning relative to cadence. An hourly job and a weekly job cannot share the same stale threshold.

Derive an evidence window from the schedule, then add a small allowance for normal delay, restarts, and queueing. Store the timestamps in UTC. Make the threshold visible so operators know why a result is stale or failed.

Without that rule, a months-old success row can keep a dead job looking healthy forever.

Test the disagreements, not only the happy path

The focused tests in the reviewed change covered more than a successful write. That is the right instinct. At minimum, exercise:

• successful completion with fresh evidence;
• a deliberate skip with an explanatory detail;
• a caught failure that does not escape the job;
• an exception that is recorded and then rethrown;
• scheduler success with no fresh evidence;
• contradictory scheduler and job outcomes;
• an unavailable scheduler monitor;
• an evidence-store write failure that must not break useful work.

These cases prove that the dashboard can produce good, bad, and unverified answers. A diagnostic that can only turn green is not a diagnostic.

The engineering trade-off

This pattern adds a table, retention, instrumentation, dependency registration, an evaluator, and tests. It is more ceremony than checking the scheduler dashboard.

In return, it separates process health from work outcome, makes missing telemetry visible, and gives incident investigation a falsifiable trail. For high-value recurring work, that is usually a worthwhile exchange.

Start with one important job. Define what success actually means, record it at the boundary, cross-check it against the scheduler, and refuse to colour unknown evidence green.

Which background job in your system would look successful after a caught failure today?