A data pipeline fails. You open the logs, find the failed run, check the input file, and try to work out what happened.
Did the source send bad data? Did processing fail? Is the evidence you need even available?
That investigation is the part I'm exploring with Hermes Data Reliability: a small AWS lab that uses Hermes Agent, built by Nous Research, to investigate failed data loads and propose a next step.
The goal is to give an engineer something useful to review: an explanation, the evidence behind it, and an honest account of what remains unclear.
Start with one broken file
The example pipeline is deliberately simple:
Producer Lambda -> S3 input -> SQS -> Processor Lambda -> S3 output
The producer creates a JSON file containing five records. The processor checks that each record has a numeric amount, doubles it, and writes the result back to S3.
For the failure scenario, the producer puts this in the third record:
{
"record_id": 3,
"amount": "not-a-number"
}
The processor rejects it during validation. Now there's a concrete failure to investigate, with a known answer we can compare the agent's explanation against.
This small example keeps the investigation easy to follow. We can inspect the input, the processing attempt, and the logs without needing to understand a whole company's data platform first.
From a log message to an investigation
The workflow has three main parts.
First, collect the signal. A collector reads configured CloudWatch logs and applies matching rules defined in YAML. When a rule matches, it saves an incident in SQLite.
Then, investigate. A separate worker picks up the incident and starts Hermes. The agent chooses which available evidence to inspect within the permissions and limits set for that investigation. For a lab run, that can include nearby logs, the input file, and saved processing-attempt details.
Finally, write up the findings. The report includes the failed stage, an explanation, uncertainty, a proposed action, and references to the evidence used.
For the malformed record, a useful explanation would look something like this:
Validation failed because the third record contains a string in
amount, where the processor requires a number. Correct that value at the source and review how to reprocess the affected file. The available evidence does not establish that a corrected run has succeeded.
That's an illustrative summary, rather than output from a measured live run. An actual report also needs references to evidence returned during its investigation.
Why keep collection separate from the agent?
The collector continues working while Hermes is busy or unavailable. Incidents and collection progress are saved in the database, so a restart can resume from saved state.
This also gives ordinary code a clear job: read the configured sources, apply the rules, and queue the matches. The agent handles the investigation after a match.
There are limits on investigation time, tool calls, and attempts. Interrupted and failed attempts still count. A difficult incident shouldn't create an endless loop of model calls.
Give the agent evidence and clear boundaries
The current investigation tools provide read-only access. Engineers decide which changes to make.
Access is enforced by code outside the model. Each tool request is checked against the approved resources, time window, and remaining budget. AWS credentials stay outside the model container.
The host also records which evidence each tool call returned. Before an automatic finding is accepted, its citations are checked against those records.
That check has a specific purpose: confirm that the cited evidence was actually returned during this investigation. It cannot prove that the agent interpreted the evidence correctly. Reviewing the explanation still matters.
Logs and file contents are treated as untrusted input, too. Text inside a log does not get to grant permissions or tell the agent to change its task.
“I couldn't verify that” is useful information
An investigation can run into missing permissions, unavailable input, or incomplete results.
Those limitations belong in the report. A failed read doesn't establish that a file is missing. A partial search doesn't establish that no relevant event happened.
The project includes three evaluation scenarios: a malformed record, a log containing misleading instructions, and an unavailable input file. They let us examine both the diagnosis and how the agent handles evidence it should distrust or cannot inspect.
These are a starting point for evaluation. Broader reliability needs more cases and measured results.
Where the project stands
Today, the lab focuses on explicit failures found through configured CloudWatch log matches. Detection of silent stoppages and overdue output, along with Snowflake integration, is still future work.
I'm building this to make the investigation after a pipeline failure easier: follow the evidence, explain what happened, and suggest a next step an engineer can review. I'd love to shape it around the problems you actually run into.
If you work with data pipelines, which recurring failure would you want an agent to investigate first?
Maybe it's a file that arrives late, a schema change that breaks a load, or a job that reports success while some data never reaches its destination. I'm looking for real examples I can recreate in the lab and run the agent against.
Leave a comment below or open a GitHub issue. A short description is enough to start:
- What does your pipeline look like, and which tools does it use?
- What goes wrong, and how do you notice?
- What logs or other clues do you have, and how do you usually find the cause?
A simplified setup or an anonymized example is perfect. I'd like to turn these into reproducible scenarios and test what the agent can explain, where it gets stuck, and what evidence would help it do better. Your recurring headache could help shape the next lab example.
If this is useful to you, give the project a star on GitHub. It helps others discover it, and I'd appreciate the support.
Top comments (5)
A useful lab case would be an S3 key overwritten after the failed processing attempt. The agent can retrieve the object successfully and cite a real receipt, yet inspect a different input from the one that failed. Could the incident pin the object version or content hash, alongside the processing attempt ID? The expected result would be either a diagnosis from that exact input or an explicit statement that the original input is unavailable. That tests a different boundary from a failed read without needing a more complex pipeline.
Yea, this is an interesting case. Let me explore it. Thanks!
Keeping collection separate from the agent, and checking citations against what the tools actually returned, are the two design choices I'd copy. To your question about which failure to investigate next, I'd pick the ones that never produce an error log: a file that arrives truncated or empty, or an upstream that stops sending. A CloudWatch rule can't match a log line that doesn't exist, so those need a different kind of detector, such as row count compared with a trailing average, or "no output object by time X". They're also where an explanation is most valuable, because nobody has a stack trace to start from. For the uncertainty field, a scenario with two plausible causes at once (say, a malformed record and a throttled downstream write in the same window) would be a good test of whether the agent names both or confidently picks one.
Thanks! It seems a lot of work will be going into the collector to actually detect the various edge cases and anomalies like this. Deterministically to generate some sort of a signal for the agent to start working.
Keeping the collector separate feels like the key move here. If the agent is busy, the failure signal still gets queued instead of lost.