In a thousand-GPU training cluster, one bad card stalls everyone. AI-Node-Sentinel is a post-silicon hardware diagnostic agent for PCIe/XID triage: it ingests node telemetry, identifies NVIDIA XID error events, and produces structured triage verdicts — so on-call engineers get "what broke, how bad, what to do first" instead of raw dmesg output.
The pain is real: XID errors are the first signal of GPU degradation, but triage today is manual. Grep dmesg for Xid across nodes, look up what the code means, decide whether to drain the node or RMA the card — while the training job keeps retrying on broken hardware. Slow triage wastes GPU-hours, which is real money at AI-infra scale.
Four steps. First, event ingestion: parses and normalizes dmesg plus DCGM/NVML telemetry (agent/schemas.py). Second, XID lookup: matches against public docs via tools/xid_lookup — XID 48 is an uncorrectable double-bit ECC error, XID 13 a graphics engine exception. Third, triage orchestration: agent/orchestrator.py aggregates the evidence. Fourth, the report: console plus JSON with severity and recommended action.
I ran the demo on simulated telemetry: a 4-node cluster with synthetic XID events injected. Output: 10 critical, 1 high, 1 unknown. XID 74 NVLink error flagged CRITICAL, XID 79 GPU-fallen-off-bus flagged CRITICAL, full JSON report written.
Honest note: scaffold v0.1, telemetry is simulated. What's real: event ingestion, XID lookup against public docs, and report rendering. The field-validated XID-to-remediation decision tree and the LangChain LLM reasoning step are TODOs, not claimed features. MIT licensed, link in the description. See you next time!
S6: Take a breather. Chinese medicine teaches 'treat before illness' — care before the sickness comes. Good ops works the same way: catch the failure before it erupts. May God, the Creator of heaven and earth, bless the work of your hands.
S7: The Lord bless you and keep you. Numbers 6:24 Technology becomes outdated, but love never ends. May God bless you and your family. If this project helped you, please give it a Star and share it with more people.
Top comments (0)