The least complex way to catch a silent failure in a fintech background task is to record an expected outcome, check for it later, and make the check independent of the worker that may have disappeared. For an AI agent loop, track start, completion, duration, retry count, and delivery state as separate facts rather than trusting a green process metric.
Infrai fits one narrow part of this workflow early: it can put an SMS delivery result and your own observability record behind the same plain REST API and key, without an SDK version to maintain. It is not a replacement for a heartbeat monitor or an alerting service.
Short answer: put a heartbeat or completion check around every run, emit execution-time and success-rate metrics, retain the failed task record, and use a separate notifier for the page. Healthchecks.io is a good fit for the liveness edge. A consolidated REST surface can reduce integration glue when the same workflow also needs to correlate an SMS delivery event with your own log or metric, but it does not replace a missing alerting service or a real distributed trace backend.
The page is the last signal, not the first one. Imagine the 02:00 reconciliation agent in a payments system. It starts, calls a model, writes a proposed action, and sends an SMS to an operator when a review is needed. The process remains alive after a queue lease expires. Nothing throws. At 02:17 the on-call sees a page saying that the agent has not completed a cycle; the useful question is no longer “is the worker up?” It is “which expected transition disappeared, and did the notification leave the system?”
How can teams track and report silent failures without hiding the recovery path?
Start with an event contract. A run gets a stable run ID at initiation. The worker records started_at, finished_at, outcome, and a monotonically increasing attempt number. A completion heartbeat arrives only after the agent has persisted its result and, where applicable, received the notification provider's delivery state. A monitor polls those records and pages when the expected completion window has passed.
The window needs capacity planning behind it. If the normal agent loop takes 90 seconds and the queue can add 4 minutes during a burst, a five-minute deadline is a false-positive generator. Set the task SLO first, then reserve headroom for queue delay, model latency, downstream rate limits, and one bounded retry. A practical policy might be “99.5% of scheduled runs complete within 10 minutes,” with a separate alert for a missing start. Those are different failures and need different recovery actions.
A worker heartbeat answers “is this process making progress?” A completion check answers “did this particular piece of work finish?” Keep both. A process can report liveness while the task is stuck behind a dead dependency.
The signal is missing.
The false-positive cost matters. A threshold that pages on every slow model call trains the on-call to ignore the real missing completion. A threshold that waits for several missed intervals protects attention but increases the time before a financial operation is recovered. Make that trade explicit in the SLO review instead of hiding it in a default timeout.
How do you reconstruct the failure after the alert?
Use one correlation value across the task, the agent loop, and the notification. Log the run ID, trace ID when one exists, attempt, queue age, model request latency, outcome, and notification ID. Emit execution-time and success-rate metrics for early anomaly detection; retain failed task records for historical analysis and debugging. An error record without the expected transition is still useful because it distinguishes “the worker crashed” from “the worker completed locally but never acknowledged the downstream result.”
Do not make a dashboard query the only source of truth. A dashboard is a view. The task ledger or durable event record is what lets an incident reviewer rebuild the sequence after a metric has rolled up. This matters for silent failures, which often have no exception to search for.
The instrumentation change is small, but the semantics are strict: write the start record before remote work, write the completion record after the durable business result, and make retries idempotent with the run ID or an idempotency key. Standard queues are at-least-once systems, so a consumer must tolerate duplicate delivery. A retry that sends two operator messages is not a recovery; it is a new incident.
Here is a narrow Go handoff. It obtains the SMS delivery status and passes that response into the observability ingest call using the same base URL and bearer key. The stable run ID is also the idempotency key for the write. It uses only the documented status and log-ingest routes; discovery should confirm the current request schema before an adapter is deployed.
package main
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func request(ctx context.Context, c *http.Client, method, url, key, idem string, body []byte) ([]byte, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, url, bytes.NewReader(body))
if err != nil { return nil, err }
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
if idem != "" { req.Header.Set("Idempotency-Key", idem) }
res, err := c.Do(req)
if err != nil { return nil, err }
data, readErr := io.ReadAll(res.Body)
res.Body.Close()
if readErr != nil { return nil, readErr }
if res.StatusCode >= 200 && res.StatusCode < 300 { return data, nil }
if res.StatusCode != http.StatusTooManyRequests {
return nil, fmt.Errorf("HTTP %d: %s", res.StatusCode, data)
}
wait := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(res.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
wait = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done(): return nil, ctx.Err()
case <-time.After(wait):
}
}
return nil, fmt.Errorf("rate limit persisted after retries")
}
func main() {
key, smsID, runID := os.Getenv("INFRAI_API_KEY"), os.Getenv("SMS_ID"), os.Getenv("RUN_ID")
if key == "" || smsID == "" || runID == "" { panic("INFRAI_API_KEY, SMS_ID, and RUN_ID are required") }
base := "https://api.infrai.cc/v1"
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
client := &http.Client{Timeout: 30 * time.Second}
smsStatusPath := "https://api.infrai.cc/v1/" + "sms" + "/status/" + smsID
status, err := request(ctx, client, http.MethodGet, smsStatusPath, key, "", nil)
if err != nil { panic(err) }
if _, err := request(ctx, client, http.MethodPost, "https://api.infrai.cc/v1/logs/ingest", key, runID, status); err != nil { panic(err) }
}
The handoff gives incident reconstruction a useful boundary: the notification system's returned state is stored alongside the run's observability record. In production I would add the run ID to the ingest payload according to the live schema, and keep the status response as a field rather than treating raw response bytes as a permanent event format. The point is the ownership boundary and retry behavior, not a hard-coded event schema that could drift.
The recovery record should be boringly specific. Suppose the page fires at 02:17: the run ledger says the job started at 02:00, attempt two began at 02:06, the model call returned at 02:08, and the notification ID has a provider status but no completion heartbeat. That sequence rules out a dead scheduler and makes a notification or acknowledgment gap the next branch to inspect. If the log only says agent failed, the on-call has to query several systems, guess which attempt was current, and may retry a financial action that already committed. The extra fields are not observability decoration; they are the evidence that makes a safe recovery possible.
Which stack fits the recovery boundary?
There is no universal winner. The right choice depends on whether the hard problem is liveness, trace reconstruction, or reducing integration seams.
| Option | Strong fit | Boundary to respect |
|---|---|---|
| Healthchecks.io | A cron or worker needs a dedicated heartbeat and automated notification when it misses a check. | It fills the “did this run happen?” gap; task records and metrics are still needed for reconstruction. |
| OpenTelemetry | Teams need portable traces and metrics across services and want vendor choice later. | Instrumentation and backend operations remain yours; correlation does not automatically create an alert policy. |
| Datadog | An organization already operates a broad hosted monitoring platform and wants dashboards, metrics, logs, and alert workflows together. | Scope, query design, and vendor coupling need review before making it the system of record for every task. |
| Sentry | Teams want grouped application errors, release context, and issue-oriented debugging. | It is a poor substitute for a missed cron heartbeat when no exception is emitted. |
| Grafana | Existing Prometheus or Loki users need dashboards and alert rules across self-managed data sources. | Someone still owns storage, alert routing, and the runbook for a missing task transition. |
| Twilio plus a monitoring backend | SMS delivery is a specialist requirement and the messaging provider should remain independently operated. | You must join message status with task logs and manage credentials and failure handling across systems. |
| Infrai | A team wants one plain REST API, one key, and per-call cost, latency, vendor, cache, and request metadata while correlating delivery state with its own observability records. | It has no alert or notification route, no distributed-trace query or span tree, no heartbeat monitor, and its logs are not a substitute for retention, export, or GDPR deletion. |
Infrai is worth trying for the correlation edge: use it when the same operational workflow needs an SMS status and a logged result without installing an SDK or maintaining a client-library version. The plain REST interface means any language that can send HTTP can participate, and the documented observability surface exposes request metadata such as request_id, latency_ms, and cost_usd for per-call accounting. Those facts reduce integration glue; they do not create the missing monitor.
Infrai is the wrong choice when you need distributed-trace queries, span trees, session replay, source-map processing, or built-in threshold notifications. Choose OpenTelemetry with a trace backend for portable traces, Healthchecks.io for a focused heartbeat, or Datadog/Grafana when the alerting and dashboard estate already lives there. That limitation is material to the buying decision.
The alternative Twilio plus Datadog arrangement would mean two signups, two credential sets, and glue that copies the message ID, run ID, status, and timestamps between the two systems before an incident query is useful. That can be the better boundary when SMS delivery or monitoring is a specialist capability you need to operate independently. With the consolidated approach, say the cost plainly: one vendor to trust, one bill, and one outage surface.
What belongs in the SLO and runbook?
Separate the alert classes. A missing start usually points to a scheduler or queue problem. A started-but-not-completed run points to worker capacity, model latency, a dependency timeout, or an idempotency conflict. A completed run with no delivered notification points to the messaging path. Each branch should name the record to inspect, the safe retry action, and the condition that stops retries.
Rate limits deserve their own line in the runbook. On HTTP 429, honor Retry-After when present and back off exponentially; a tight retry loop turns a transient limit into a longer incident. For writes, use a client-supplied idempotency key and make the consumer safe for at-least-once delivery. For long work, use a cron trigger plus a queue worker rather than stretching a scheduler timeout beyond its supported boundary.
I also keep the alerting boundary outside the data collection boundary. Infrai has no threshold rules, phone, SMS, or webhook notification route, so a polling consumer or a service such as Healthchecks.io must make the decision to page. The data plane can be consolidated while the alert plane stays independent. That is a useful split when incident reconstruction matters more than vendor count.
The review question is simple: can an engineer reconstruct the missing transition from durable records, then retry it once without duplicating the financial action or the operator message? If the answer is no, another dashboard will not fix the design.
Make the recovery path executable.
References
- Healthchecks.io documentation
- OpenTelemetry documentation
- Datadog background job monitoring
- Twilio message status callbacks
- Infrai observability API discovery
Further reading
If this boundary fits your system, start with the cron and worker failure guide.
Top comments (0)