Imagine a sync job that starts at 2 a.m. and has produced nothing by morning. You give its code to an agent and ask, "Where did it get stuck?" It can find the timeout handler and retry settings. To find out how long this run lasted or how many times it retried, it needs a record from the run.
Logs and dashboards probably contain some of those facts. Manually pulling them into a chat is fine once in a while. A nightly job benefits from a fixed reporting convention, though: by the time something fails, you should not have to hunt through old logs to learn how long the previous run took.
The job can report a few key events as it runs, and an external system can retain them. When something goes wrong, the agent retrieves records by job ID and time. Those details do enter its analysis context, so decide which fields are safe to report and who can read them.
Record the moments that matter
Record the start and end of a run, its outcome, duration, and retry count. The job name nightly-sync identifies a series of executions; a different trace_id identifies each execution and connects its events. A scheduler restart should get a new ID. It should not be confused with an internal retry in the same run.
Define where a duration starts. If the job is scheduled at 02:00, waits a minute for a worker, and finishes at 02:02, you can report two minutes since scheduling or one minute of execution. Both can be useful, but they cannot be compared as if they were the same metric. In this example, duration_ms runs from actual execution start to the reported outcome and includes time spent waiting between internal retries.
The path is short:
Job or app --report event--> Validate and store --retrieve on demand--> Agent --analyze--> Person
An existing event platform may be enough. For small scripts and services scattered across systems, a shared reporting API is easier to govern than giving an agent direct read access to each production database. The agent gets a scoped read credential for the project.
Keep full logs in their usual place; they may contain secrets, customer details, and irrelevant stack traces. Report the small set of facts the agent will need: job name, run ID, result, and duration. Missing outcome data cannot be reconstructed later from a clever prompt.
Similarly, define retry_count as attempts made within this execution. If the scheduler starts a new process after a failure, that is another run, not an additional internal retry of the first one. A batch ID may be worth retaining with the event, but one placed in extra_params can only be read after retrieval. Frequent lookup by batch needs a planned queryable field.
Two nights, two outcomes
Both runs below are invented, not a real incident. On failure, nightly-sync submits an event under an active, authorized behavior-event schema:
{
"schema_id": "default_behavior_events",
"schema_version": 1,
"data": {
"report_source": "be",
"client_time": "2026-11-05T02:02:00+08:00",
"scene_type": "scheduled_job",
"action_type": "run_failed",
"target_type": "job",
"target_id": "nightly-sync",
"target_value": "timeout",
"trace_id": "trace-job-demo-002",
"extra_params": { "duration_ms": 120000, "retry_count": 1 }
}
}
In this example, the November 5 job starts at 2 a.m. Singapore time and reports failure 120 seconds later, so client_time is 2:02 a.m. target_id is queryable. The duration and retry count in extra_params can be read after retrieving the record, but its nested keys cannot be used as ordinary query filters. The trace_id can connect this run to other evidence; a comparison with yesterday first requires that both runs use the same job identity and reporting convention.
The invented November 4 run also starts at 2 a.m. Singapore time. It reports run_completed at 2:00:45 a.m., with duration_ms=45000 and retry_count=0. These two reported durations differ by 75 seconds. They do not tell us whether the job has been slowing down all week, or which downstream service caused the timeout.
Check the job version, input size, and definition of completion before calling this a slowdown. Processing a hundred thousand rows one night and a hundred the night before changes the comparison. The schema has an app_version field if you need to track a job version. For input size, decide what is safe to report and whether it belongs in an extension attribute or a separately planned queryable field.
HTTP 202 acknowledges receipt by the message pipeline; the event may not yet be queryable. Save each response's event ID and server_time, then check again after asynchronous loading. Assume both receipts are within November 3 at 16:00 through November 5 at 16:00 UTC, the server-time window corresponding to November 4 and 5 in Singapore. A late report could be received after that window and would need a separate query.
If the reporting client disconnects before receiving the response, the state is less clear: the runner may have submitted the failure event without seeing the acknowledgment. Reconcile before retrying rather than declaring the platform missed it. Conversely, retrieving a run_failed event only tells you what the runner reported; it does not prove that every expected output row is absent.
Compare the runs
An engineer could ask:
Find
nightly-syncevents from November 3, 2026, 16:00 to November 5, 2026, 16:00 UTC. Compare the November 4 and 5 runs in Singapore time. List each run ID, outcome, duration, and retry count, then inspect where the timeout could occur in the code.
The agent uses its read credential to query by server time and target_id, then compares the records with the code's timeout and retry paths. In Codex, a configured MCP tool can provide that query. The Codex documentation covers the connection.
Assuming both events have loaded and the query has not reached its limit, the answer might be:
The November 5 run reported
run_failedwithtimeoutafter 120 seconds and one internal retry. November 4 reportedrun_completedafter 45 seconds with no retry. That is a 75-second difference. We still do not know which dependency stalled; check the failed run's logs and downstream calls using itstrace_id.
The timeout branch in the code suggests where to look, but it does not prove this run passed through it. If the query is incomplete, the answer needs to say which records it actually saw.
Use the failed run's trace_id to find its execution logs. Did the internal retry occur before a connection was established, halfway through reading, or while writing the output? A slow dependency, a timeout threshold set too low, and queue time mistakenly included in execution time could all surface as target_value=timeout. Timing for the corresponding downstream call is what would let you narrow the cause.
Check trace_id before comparing counts. A scheduler restart is a separate execution; mixing its duration and retries with the first run would corrupt the diagnosis. If a query fills its limit, narrow the window or raise the total within the policy and follow a returned cursor for subsequent pages.
Test the failure paths first
In this example, duration_ms and retry_count live in extra_params. You can inspect them after retrieval but cannot filter directly on those nested keys. A standing question such as "which jobs ran longer than two minutes?" needs a queryable field or another analytics path. Keep reporting and reading credentials separate, and keep secrets and unnecessary personal data out of the event. The current production defaults cap a page at 100 rows and a time window at 31 days. You can query several windows, but that alone does not turn detail records into a trustworthy aggregate trend.
The meaning of run_completed matters too. It should mean that this execution finished according to the runner's contract. If the business question is whether the synced data passed a completeness check, report that check separately. The 45-second success record alone cannot establish that every output row was correctly stored the night before.
Use a dedicated job to report one success and one failure. Save both 202 receipts, wait until the records are queryable, and ask the agent to compare them. Then delay one report and see whether it confuses "not visible yet" with "never ran." That small test says more about this workflow than a fluent incident summary.
Project website: https://d2ggep2t73r4ny.cloudfront.net/
Technical and business discussion: lorry@antlertec.com
Top comments (1)
The time of the last good run and the exact record the failed run stopped on would be the first two fields to standardise, since most fixes start from what changed in between.