Fixing Alert on Host Resource Pressure in Hermes-Memory-Installer

The latest update to hermes-memory-installer addresses a critical gap in operational monitoring: the alert mechanism for host resource pressure was failing to trigger under specific conditions. This fix ensures that the installer correctly detects when a node is under memory or cpu pressure, preventing deployment of memory configurations that could destabilize the host. For developers managing memory-intensive workloads, this is a practical improvement in reliability.

Context: What Hermes-Memory-Installer Does

hermes-memory-installer is a lightweight agent that configures memory allocation policies (e.g., hugepages, NUMA binding, cgroup limits) on cluster nodes. It is typically used as an init container or daemon set in Kubernetes environments. The tool checks host metrics before applying changes, and one of its key preflight checks is detecting host resource pressure — conditions where the system is struggling to meet demand, often indicated by PSI (Pressure Stall Information) values.

The Problem: Silent Failures Under Pressure

Before this fix, the alert function for resource pressure used a simple polling mechanism that read /proc/meminfo and /proc/stat and applied static thresholds (e.g., MemAvailable < 10%). This approach missed transient but sustained pressure events — for example, when a co-located process pinned memory or a kernel reclaim storm occurred. The installer would proceed, causing OOM kills or latency spikes on the host. Worse, the alert log message was often suppressed if the pressure resolved quickly, leaving no audit trail.

The Fix: PSI-Based Alerting

The core change is a shift from usage-based heuristics to kernel-native PSI metrics. The PSI interface (/proc/pressure/) provides some and full stall percentages over 10s, 60s, and 300s windows. The fix introduces a dedicated check that evaluates some_avg10 against configurable thresholds. If pressure exceeds 80% for memory or 70% for CPU for a sustained period, an alert is raised and the installation is blocked.

The update includes three improvements:

• Accurate detection: PSI captures actual stall time, not just utilization.
• Alert propagation: Alerts are now logged and emitted as Kubernetes events or to stdout, depending on the runtime.
• Configurable thresholds: Users can set MEMORY_PRESSURE_THRESHOLD and CPU_PRESSURE_THRESHOLD environment variables.

Code Example: PSI Check in Rust

Below is a simplified version of the check function from the updated codebase:

use std::fs;

fn read_psi_value(path: &str, field: &str) -> Result<f64, Box<dyn std::error::Error>> {
    let content = fs::read_to_string(path)?;
    for line in content.lines() {
        if line.starts_with(field) {
            if let Some(percent) = line.split(' ').nth(1) {
                return Ok(percent.trim_end_matches('%').parse::<f64>()?);
            }
        }
    }
    Err("Field not found".into())
}

fn check_host_pressure() -> Result<(), String> {
    let mem_some = read_psi_value("/proc/pressure/memory", "some")?;
    let cpu_some = read_psi_value("/proc/pressure/cpu", "some")?;

    let mem_threshold = std::env::var("MEMORY_PRESSURE_THRESHOLD")
        .unwrap_or("80.0".into())
        .parse::<f64>()
        .unwrap_or(80.0);
    let cpu_threshold = std::env::var("CPU_PRESSURE_THRESHOLD")
        .unwrap_or("70.0".into())
        .parse::<f64>()
        .unwrap_or(70.0);

    if mem_some > mem_threshold || cpu_some > cpu_threshold {
        return Err(format!(
            "Host resource pressure: memory {:.1}%, cpu {:.1}%",
            mem_some, cpu_some
        ));
    }
    Ok(())
}

This code reads the some line from each PSI file, extracts the avg10 percentage, and compares against (configurable) thresholds. The function returns an error and the installer aborts before making any changes.

Implications for Operators

With this fix, hermes-memory-installer now enforces a meaningful pre-condition for memory configuration. Operators should:

• Verify PSI is enabled on nodes (kernel >= 4.20, CONFIG_PSI=y).
• Adjust thresholds based on workload sensitivity.
• Monitor alert logs to identify chronically pressured nodes.

The update also adheres to the principle of failing early: instead of silently applying settings under pressure, the tool surfaces the condition immediately. This prevents cascading failures in environments where multiple installers run concurrently.

Conclusion

This update is a small but impactful change. By replacing guesswork with kernel-provided pressure metrics, hermes-memory-installer aligns with modern Linux observability practices. For developers running memory installations on shared or oversubscribed hosts, the fix provides a reliable safety net. The next time you see ALERT: host resource pressure in your logs, it means the tool is correctly prioritizing system stability.