Your monitoring says healthy. Your agents are not.

What Broke

In the weeks leading up to our most recent production run, our local LLM agent fleet experienced a series of failures that went unnoticed for days. Looking at the failure ledger, several patterns emerge:

• Process Errors in Inference: Multiple entries show process_error failures during the secretary/consult-classify and secretary/consult-fields stages. For example:

  • On August 29th at 08:06:11 UTC, the ollama/qwen3.8:27b model encountered a process_error with a duration of 318,454ms.
  • On August 30th at 09:54:39 UTC, the same model failed again with an even longer duration of 1,813,103ms.

• Timeouts in Agent Evaluation: On September 1st, two consecutive timeouts occurred during hr/agent-evaluation:

  • 02:55:03 UTC and 03:00:06 UTC, both with the ollama/qwen3.8:27b and qwen3.6:27b models, respectively. Each timeout lasted approximately 300 seconds.

These failures indicate that while the agents were running unattended, they encountered issues that prevented them from completing their tasks. The long durations and timeouts suggest that the models were either stuck in a loop or encountered resource constraints that were not being addressed.

Why It Happened

The root cause of these failures can be attributed to several factors:

1. Model Instability: The models being used, such as ollama/qwen3.8:27b and ollama/qwen3.6:27b, may have inherent instability when running continuously without human oversight. This could be due to memory leaks, inefficient garbage collection, or other performance issues.

2. Resource Constraints: Running multiple agents simultaneously can strain system resources, leading to timeouts and process errors. The long durations recorded in the failure ledger suggest that the agents were consuming excessive CPU or memory, causing the system to become unresponsive.

3. Lack of Monitoring: Without proper monitoring in place, these failures went unnoticed for an extended period. This highlights the importance of having real-time alerts and monitoring tools to detect and address issues promptly.

What We Put in Place

To prevent similar issues in the future, we implemented several mechanical guards:

1. Automated Alerts: We set up automated alerts to notify the engineering team immediately when a process_error or timeout occurs. This ensures that any issues are addressed promptly, minimizing downtime and preventing prolonged failures.

2. Resource Monitoring: We implemented resource monitoring to track CPU and memory usage of the agents. If an agent exceeds predefined thresholds, it is automatically restarted or reconfigured to use fewer resources.

3. Model Health Checks: Regular health checks are now performed on the models to ensure they are running optimally. This includes checking for memory leaks, optimizing garbage collection, and ensuring that the models are not consuming excessive resources.

4. Failover Mechanisms: We introduced failover mechanisms to switch to a backup model or agent if the primary one fails. This ensures that the agent fleet can continue operating even if individual components encounter issues.

The Lesson

The key lesson from this experience is the importance of proactive monitoring and maintenance in unattended agent fleets. While running agents unattended can increase efficiency and productivity, it also introduces risks that must be managed carefully. By implementing automated alerts, resource monitoring, regular health checks, and failover mechanisms, we can ensure that our agent fleet remains stable and reliable, even when running for extended periods.

In conclusion, the 58-day unattended run highlighted the silent failures that can occur in local LLM agent fleets. By addressing these issues with the mechanical guards we put in place, we have improved the resilience and reliability of our system. Moving forward, we will continue to refine our monitoring and maintenance strategies to ensure that our agent fleet operates smoothly and efficiently.

Comments

[Salman Parvez]

The failure ledger is doing more work here than the alerts are. Once you have a durable record of what each agent attempted and what came back, alerting becomes a query over that record rather than a second system to keep in sync.

The class it does not cover is the one I find harder: the run that completes, returns in a normal duration, and is wrong. No process_error, no timeout, and failover does not help either - it just gets you a confidently wrong answer from a different model.

What worked for us was making the output auditable rather than the process. Every agent writes claims to one shared record, each row carries an evidence grade, and two agents disagreeing resolves to a surfaced conflict instead of a silent overwrite. Health checks tell you the fleet is up. The record tells you whether it was right.

Does your failure ledger have a slot for a run that completed and was later found wrong? That is the row I would want most in a 58-day unattended stretch.