A scheduled task fails at 2 am. By the time an engineer looks at it, the run history shows a failed
status and a platform message about capacity. That message says what happened. It does not say
whether to run it again.
The answer depends on things the status cannot show. Has the same task failed this way before? Are
other tasks on that platform failing too? Is a retry already queued? Is the job safe to run twice?
Answering those means opening several screens and relying on whoever remembers the last incident, and
it happens before anyone has fixed anything.
That investigation is the part worth automating. It is also the part where an LLM is both most useful
and most dangerous, because the obvious next step after "here is what went wrong" is "shall I run it
again?".
The agent described here splits that in three. Rules and detectors decide what they can decide. A
model explains what needs interpreting. Code decides what may happen. Each step has a different
owner, and the model never approves its own recommendation.
Step 1: rules and detectors before the model
Not every failure needs a language model. Many have a recognisable signature, so they are matched
against a reviewed catalogue of known patterns first. Measurable conditions, a run stuck in Running, a
schedule that should have fired and did not, a queue that is not moving, are found by detectors
reading run history and thresholds.
Both paths produce findings with no model call at all, which matters for cost and latency as much as
for correctness: a well-understood failure gets the same answer every time, and it gets it without
waiting on an inference.
Step 2: the model diagnoses, from a narrowed context
AI diagnosis is kept for the incidents that need interpretation. It works from sanitised run context
and from similar incidents the organisation has already seen, so the explanation is grounded in what
happened here rather than in general advice.
Every finding is labelled AI-generated, Rule-based or Detector. That label tells the reader how much
weight to give it, and it makes the cheap paths visible rather than hidden behind a single confident
voice.
Step 3: a policy engine written in code decides
This is the part that makes the rest safe to switch on. The model can recommend a retry. Whether that
retry is allowed is decided by a policy engine in code, which checks:
- the type of failure;
- the rules the team has set;
- the current state of the run;
- any existing retries;
- the limits in force;
- the authority the organisation has granted.
A persuasive explanation cannot talk its way past those checks. A credential failure is denied because
running the job again will not renew a credential, however well the diagnosis reads.
Authority is a dial, not a switch
The agent starts off. An administrator enables it, records consent and chooses how far it may go:
- Observe. It explains problems and writes insights, and changes nothing.
- Recommend. It proposes a retry that a person approves, and checks again before running it in case the situation has moved on.
- Act within limits. It retries on its own, but only under a standing authorisation scoped to particular tasks, jobs or platforms, with limits and an expiry that an administrator can revoke at any time.
A task marked Never retry is never retried, at any level. Retrying is the only action the agent
can execute: it does not edit pipelines, renew credentials or change schedules. At every level, every
action is recorded in the audit trail.
The useful property here is that the levels let a team earn confidence from its own data. Run it at
Observe until the explanations match what your engineers would have concluded, move to Recommend when
the proposals look like decisions you would have approved anyway, and grant narrow authority only
where the record supports it.
A retry is only done when you know the outcome
Starting another run is the easy part. Knowing whether it worked is what tends to get lost, because
the person who clicked retry has moved on.
When the agent retries a run it keeps watching until the result is in. A success joins the
organisation's record of what recovers from that kind of failure. If the run fails the same way again,
the agent stops, marks the retry as futile and raises the issue for a person instead of looping. That
is the difference between temporary capacity pressure, which may clear on a second attempt, and an
expired credential, which never will.
The same machinery covers the failures that never announce themselves: a run going far longer than
usual, a scheduled run that never started, a success rate slowly slipping. Those reach you as a late
report rather than an alert, which is exactly why they are worth detecting.
What it does not do
Worth stating plainly, because the category attracts overclaiming:
- It runs on a schedule, not in real time. Native platform alerts are still the fastest way to hear that a job failed, and a task's built-in retry is still the right tool for an operation you know is safe to repeat.
- It sees the operational records of each run, not the code inside your pipelines or your business data.
- It can be wrong. That is why every finding shows its evidence.
- Error sanitisation is best effort, and administrators can switch error text off entirely.
- A retry is a real run. It can repeat side effects and it counts towards the task run allowance, so mark a task safe to repeat only when it genuinely is.
The pattern, if you are building your own
Nothing here is specific to one product. If you are putting an agent near production systems, the
split is the transferable part: let the model produce explanations, keep the decision in code you can
review and test, make authority a scoped grant with an expiry rather than a flag, and follow every
action to its outcome instead of firing and forgetting.
The full article, including the fictional failed run followed through each level, is on our website:
https://www.polysync.com.au/articles/ai-monitoring-agent?utm_source=devto&utm_medium=referral&utm_campaign=articles&utm_content=ai-monitoring-agent
About Polysync
Polysync Data Solutions is an Australian company that builds cross-platform orchestration for data,
integration and AI workloads. The AI Monitoring Agent is part of the platform; this post describes
how it was designed rather than announcing it.
Top comments (0)