I gave my AI agent a small deployment history and asked how often the checks were unhealthy. It wrote the Python, ran its own tests, and came back with the right answer.
Then I read the tests. They passed, but they didn't cover everything I'd asked for.
Some context: this was a small experiment in my personal AWS lab, with made-up deployment data. There's no production incident behind it.
A history file, not production access
I started with this CSV, a small deployment history from my personal AWS lab. The data is made up. It has twelve deployment checks, five minutes apart, and each row records whether the check was healthy, the expected and running application versions, the instance counts, and any active alarms.
timestamp,status,desired_version,running_version,healthy_instances,desired_instances,active_alarms
2026-09-20T10:00:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:05:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:10:00Z,unhealthy,2026.09.20.1,2026.09.19.4,2,3,1
2026-09-20T10:15:00Z,unhealthy,2026.09.20.1,2026.09.19.4,2,3,1
2026-09-20T10:20:00Z,unhealthy,2026.09.20.1,2026.09.19.4,2,3,1
2026-09-20T10:25:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:30:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:35:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:40:00Z,unhealthy,2026.09.20.1,2026.09.20.1,2,3,0
2026-09-20T10:45:00Z,unhealthy,2026.09.20.1,2026.09.20.1,2,3,0
2026-09-20T10:50:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
2026-09-20T10:55:00Z,healthy,2026.09.20.1,2026.09.20.1,3,3,0
This is the exact file I gave the agent, so you can run the same experiment yourself.
I wanted two numbers: how many checks were unhealthy, and how long each completed unhealthy period lasted.
The second one needed a rule, so I wrote it down. A period starts at the first unhealthy check and ends at the first healthy check after it. If the file ends while checks are still unhealthy, that period stays incomplete.
Take the first unhealthy check, at 10:10. The next healthy one is at 10:25, so the period is fifteen minutes. That's what the supplied checks show, and nothing more. It isn't the exact moment a real application failed or recovered. I wanted that settled before handing the calculation to an agent.
The model wrote the Python, a tool ran it
I used Amazon Nova 2 Lite as the model, through my managed agent setup. I gave it the calculation rules, three small test cases and the target CSV. I didn't give it an implementation.
It generated the program and sent it to Amazon Bedrock AgentCore Code Interpreter, which runs code in a separate execution environment.
The flow was short:
Supplied CSV → model generates Python → Code Interpreter runs it → result comes back
On my side, I did two things: compare the returned numbers with my expected result, then read the generated tests.
The retained stream showed a single Code Interpreter call containing the generated program. The tool output had the test-pass marker and the calculated results. So I could look at what actually ran, instead of trusting the model's summary of it.
The numbers matched, the tests had a gap
Before the agent run, I had worked out the expected results myself. The output matched:
| Check | Result |
|---|---|
| Total samples | 12 |
| Unhealthy samples | 5 |
| Unhealthy sample percentage | 41.7% |
| Completed unhealthy periods | 2 |
| Period lengths | 15 minutes and 10 minutes |
The 41.7% is five unhealthy samples out of twelve. It doesn't mean the application was down for 41.7% of the time.
The program also printed MODEL_GENERATED_TESTS=PASS.
But my prompt asked for assertions, the checks that stop the program when an expectation is wrong, for every stated test expectation. The generated tests covered period counts and durations. They didn't explicitly check every expected start and end timestamp.
The timestamps in the target output were correct, so the answer wasn't wrong. The gap was in what the tests actually proved.
That's the part worth remembering. "The tests passed" means the assertions that were written passed. It doesn't mean every requirement was tested. Comparing the result and reading the assertions answered two different questions.
What this run does and doesn't show
It showed an agent generating and executing analysis code whose output matched a result I'd calculated independently. It also showed why I needed to read the generated tests.
It doesn't show that every future run would work. Malformed inputs, access restrictions, session behaviour and repeated-run checks are still follow-up evaluation.
For this kind of analysis, I'd keep three things together: the input, the code that actually ran, and its output. Then I'd check the generated tests against the original requirements before trusting the pass marker.
Top comments (0)