Short answer: poll logs and the errors API into a shadow decision process first, then alert only on distinct failed checkout operations; keep rollback behind a separately tested state check. For a property-management checkout, that separation matters more than whether the final notification goes to email or a Slack webhook. A Node.js service can emit the evidence while a small Python cron job evaluates it without sharing application memory or deployment fate.
Don't let an exception count become a rollback command.
The flow is deliberately plain. Each checkout carries an operation_id through payment and reservation commit. The poller reads new records from two configured API URLs, normalizes them, saves the source cursors, and evaluates a rolling window. A qualifying decision becomes a durable alert with an idempotency key. An operator or a separate controller then checks current reservation and payment state before taking action. Polling is detection; rollback is control.
API implementation: a runnable shadow poller for checkout evidence
Start with the smallest event contract that can answer an operational question. For this workflow, that means an evidence ID, operation ID, property ID, stage, outcome or status, observed timestamp, and source. Use an opaque reservation key if responders need to correlate a booking, but keep guest names, addresses, payment data, and access tokens out of the record. The errors API may contribute an exception class; it must not be the only place that carries the operation ID.
The example below treats the two source URLs as complete, configured endpoints. It makes no assumptions about vendor routes. Both APIs return an items array and a next_cursor; adapt only normalize() if the actual schemas differ. The decision threshold is a policy argument rather than a constant hidden inside the polling loop, which makes the evaluator easy to exercise in a notebook and then move unchanged into a scheduled process.
from __future__ import annotations
import hashlib
import json
import os
import sqlite3
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from urllib.parse import urlencode
from urllib.request import Request, urlopen
@dataclass(frozen=True)
class Evidence:
evidence_id: str
operation_id: str
property_id: str
stage: str
status: int
observed_at: datetime
source: str
def fetch_page(url: str, token: str, cursor: str) -> dict:
separator = "&" if "?" in url else "?"
page_url = f"{url}{separator}{urlencode({'after': cursor})}"
request = Request(page_url, headers={"Authorization": f"Bearer {token}"})
with urlopen(request, timeout=8) as response:
return json.load(response)
def normalize(item: dict, source: str) -> Evidence:
timestamp = item["timestamp"].replace("Z", "+00:00")
return Evidence(
evidence_id=f"{source}:{item['id']}",
operation_id=item["operation_id"],
property_id=item["property_id"],
stage=item["stage"],
status=int(item.get("status", 0)),
observed_at=datetime.fromisoformat(timestamp),
source=source,
)
def decisions(
evidence: list[Evidence],
now: datetime,
window: timedelta,
distinct_operations: int,
) -> list[dict]:
cutoff = now - window
grouped: dict[str, dict[str, list[str]]] = {}
for record in evidence:
if (
record.stage != "reservation_commit"
or record.status < 500
or record.observed_at < cutoff
):
continue
operations = grouped.setdefault(record.property_id, {})
operations.setdefault(record.operation_id, []).append(record.evidence_id)
alerts = []
for property_id, operations in grouped.items():
if len(operations) < distinct_operations:
continue
evidence_ids = sorted({item for ids in operations.values() for item in ids})
alerts.append(
{
"kind": "checkout-commit-failure",
"property_id": property_id,
"failed_operations": len(operations),
"evidence_ids": evidence_ids,
"next_action": "verify payment and reservation state before rollback",
}
)
return alerts
def send_webhook(url: str, payload: dict, idempotency_key: str) -> None:
body = json.dumps(payload, separators=(",", ":")).encode("utf-8")
request = Request(
url,
data=body,
method="POST",
headers={
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
},
)
with urlopen(request, timeout=8) as response:
response.read(1)
def cursor(db: sqlite3.Connection, source: str) -> str:
row = db.execute(
"SELECT value FROM poll_state WHERE source = ?", (source,)
).fetchone()
return row[0] if row else ""
def run_once(db_path: str, urls: dict[str, str], token: str, webhook: str) -> None:
with sqlite3.connect(db_path) as db:
db.execute(
"CREATE TABLE IF NOT EXISTS poll_state "
"(source TEXT PRIMARY KEY, value TEXT NOT NULL)"
)
db.execute(
"CREATE TABLE IF NOT EXISTS sent_alert "
"(alert_key TEXT PRIMARY KEY, sent_at TEXT NOT NULL)"
)
pages = {
source: fetch_page(url, token, cursor(db, source))
for source, url in urls.items()
}
records = [
normalize(item, source)
for source, page in pages.items()
for item in page["items"]
]
for alert in decisions(records, datetime.now(timezone.utc), timedelta(minutes=5), 3):
material = "|".join(alert["evidence_ids"]).encode("utf-8")
alert_key = hashlib.sha256(material).hexdigest()
already_sent = db.execute(
"SELECT 1 FROM sent_alert WHERE alert_key = ?", (alert_key,)
).fetchone()
if already_sent:
continue
send_webhook(webhook, alert, alert_key)
db.execute(
"INSERT INTO sent_alert(alert_key, sent_at) VALUES (?, ?)",
(alert_key, datetime.now(timezone.utc).isoformat()),
)
for source, page in pages.items():
db.execute(
"INSERT OR REPLACE INTO poll_state(source, value) VALUES (?, ?)",
(source, page["next_cursor"]),
)
if __name__ == "__main__":
run_once(
"checkout-alerts.db",
{"log": os.environ["LOGS_API_URL"], "error": os.environ["ERRORS_API_URL"]},
os.environ["OBSERVABILITY_TOKEN"],
os.environ["ALERT_WEBHOOK_URL"],
)
There is an intentional boundary in this sample: it sends a decision, not a command to release inventory or reverse payment. The webhook receiver should honor the idempotency key. If it cannot, put a durable queue between the poller and notification delivery, because a process can stop after the remote endpoint accepts a message but before SQLite records the send. Exactly-once delivery isn't something a local flag can promise across that network boundary.
Rollback reliability needs a state machine, not an alert side effect
A checkout failure is ambiguous. A 502 observed at reservation_commit says the caller did not receive a successful response; it does not, by itself, prove that no reservation exists. Likewise, an exception after payment authorization doesn't establish that capture completed. The safe response therefore has explicit states: detected, reconciled, paused, replayed, and resolved. Only reconciliation can move a case from detection toward a reversible action. This changes the alert payload: include the property, distinct operation count, evidence IDs, window boundaries, and a neutral next action, but do not include a button that blindly performs rollback. The reconciliation reader fetches the current payment and reservation records by operation ID, compares them with the intended checkout outcome, and records the observation used for its transition. If current state is unavailable, it waits; uncertainty is a state, not permission to undo customer work. A feature toggle can provide the coarse pause mechanism for new checkout attempts. Martin Fowler's discussion of feature toggles also shows why toggle configuration needs to remain manageable: toggles introduce decision points and carrying them indefinitely creates complexity. Here, the useful toggle is narrow and operational, such as pausing new reservation commits for one property while existing cases are reconciled. It should have an owner and a removal condition. The alert can recommend evaluating that toggle, but the poller should not flip it merely because three records crossed a threshold.
This is the core rule: detect broadly, mutate narrowly.
Direct in-process hooks are another valid design. Logback's appender documentation, for example, defines an extension point for sending logging events to a destination. An equivalent hook in the checkout service can reduce detection delay, but it also puts alert-delivery behavior on the application's execution path unless carefully isolated. A poller accepts bounded delay in exchange for a separate failure domain. Neither choice makes rollback safe on its own; the state check does.
Test the decision before testing the notification
I start this kind of work in a notebook because the policy is easier to inspect as a pure function than as a cron schedule. The evaluator above takes records, a clock, a window, and a threshold. Feed it a small synthetic table and inspect decisions before wiring email or a Slack webhook. That's eval-driven operations: the expected decision is the assertion, and transport comes later.
Use at least these cases in the harness. Three 5xx records for one operation must count as one failed checkout, while three distinct operation IDs at one property may cross the example threshold. A delayed error record must not become “new” merely because the poller saw it late. Evidence outside the five-minute policy window must not count. A successful reservation_commit must not be selected. Running the same page twice must produce the same alert key. Then test a mixed case in which the log record and exception record share an operation ID; both evidence IDs belong in one decision.
The numbers 3 and 5 minutes are sample policy inputs, not universal recommendations. Replay retained traffic, label cases that truly required intervention, and compare candidate policies on missed incidents, duplicate decisions, and time to a safe action. I'm not sure what delay a particular property portfolio can tolerate, and notification volume won't answer that. The deciding measurement is elapsed time from failed checkout evidence to reconciled state, split by property and checkout stage.
Prompt-cost awareness has a useful parallel here. Every extra field and every extra query has a processing and retention cost, but stripping context too aggressively makes the output impossible to evaluate. Keep the compact identifiers required to join evidence and explain a decision; fetch detailed records during reconciliation. That gives the routine poll a small, predictable payload without reducing the alert to an unauditable counter.
How should cron job thresholds turn 5xx logs into failed-job alerts?
Raw exception volume is usually the wrong denominator. One operation may retry several times, and a single library error may emit through both the application log and the errors service. Group by stable operation ID first, then by the operational boundary that can be paused safely, which is the property in this scenario. A global threshold could hide a concentrated checkout failure at a small building or overreact to unrelated failures spread across many properties.
Keep 5xx status and exception class as evidence, not identity. Deduplication should use stable source evidence IDs, while policy counting should use distinct checkout operations. The two concepts have different jobs. Evidence dedup prevents reprocessing; operation grouping estimates affected work.
The catch is that polling is not suitable when the source cannot provide a stable cursor, durable evidence IDs, or a correlation key shared across checkout stages. In that situation, instrument the application to publish a durable outbox event, or consume a queue that already preserves order and redelivery. Stick with streaming detection when the acceptable response budget is shorter than the polling interval plus source-ingestion delay. Keep polling as a reconciliation backstop if it still adds independent evidence.
Email may fit a low-urgency review queue; a Slack webhook may fit a staffed operational channel. Neither is incident state. The durable decision record, acknowledgement, reconciliation result, and recovery transition belong in storage that can be queried after the chat message scrolls away.
Cursor governance and evidence retention before production
Before production, walk one synthetic checkout from evidence emission to resolution. Verify that both cursors advance only after both pages are fetched and decisions are processed; confirm that a replay keeps the alert key stable; stop the process immediately after webhook delivery and observe the receiver's idempotent behavior; skew an event timestamp; paginate beyond one response; redact payloads; and expire old deduplication rows under a documented retention rule. Run the shadow evaluator without paging anyone until its decisions match reviewed outcomes, then enable one notification channel at a time.
Also test recovery. A quiet interval isn't proof that checkout recovered, so emit a recovery decision only after direct success evidence meets a separate policy. The operator should be able to answer three questions from stored data: what failed, which current state was reconciled, and why the chosen action was reversible. If the system can answer only “the exception count dropped,” it isn't ready to control rollback.
Keep the scheduler boring. Give it a single-instance guarantee or a lease, bounded HTTP timeouts, credentials limited to evidence reads and notification writes, and monitoring for cursor age. The application remains free to be Node.js; the Python evaluator remains a small control-plane component with pure policy logic that can be replayed in CI. That separation is what makes the notebook-to-production path credible.
References
- Martin Fowler, “Feature Toggles”: https://martinfowler.com/articles/feature-toggles.html
- Logback Manual, “Appenders”: https://logback.qos.ch/manual/appenders.html
Top comments (0)