DEV Community: Vivaan

Before blaming GPU heat: an evidence checklist for failed AI jobs

Vivaan — Sat, 10 Oct 2026 12:42:56 +0000

ThermaCompute AI is building diagnostics to help AI infrastructure teams understand GPU failures and performance waste. Our current focus is passive investigation: useful evidence and recommendations an operator can test.

A final timeout can hide the earlier error that deserves attention. Before restarting a failed job, I would work through this checklist.

1. Reconstruct the sequence

Record the job, worker or rank, timestamps and source log. Check whether a worker error occurred before the collective timeout. An ordering is a lead to investigate, not proof that one event caused the other.

2. Separate temperature from throttling

A temperature reading alone does not prove thermal throttling. Look for supported throttle-reason telemetry, clock changes and workload throughput in the same time window. Power limiting, memory pressure and communication failures need separate consideration. Missing telemetry should remain an explicit unknown.

3. Make the next check reproducible

A useful report should include the raw evidence line, the rule that matched, what the finding does not establish, and a concrete next check. Avoid turning a log match into an exact dollar-loss estimate without a measured baseline and stated cost assumptions.

Try the free CrashLens example

We made CrashLens to turn a saved log into a focused investigation. It is an early, open-source CLI for supported OOM, NCCL and NVIDIA Xid patterns, with evidence lines and suggested checks. It does not control GPUs or establish root cause.

The synthetic demo needs Python 3.10+, no GPU or account, and no log upload:

git clone https://github.com/rohitcn-hub/thermacompute-crashlens.git
cd thermacompute-crashlens
python crashlens.py examples/synthetic.log --out demo-report

Open demo-report/report.html in your browser. Choose a new output directory if it already exists. Preview the synthetic report before installing.

For GPU operators: did the report give you one useful next check? What evidence would be missing before you would test it? Please do not post private production logs.

Optional scoped review

ThermaCompute offers an $800 Executive Thermal Architecture Audit for teams with suitable NVML/vLLM telemetry. We agree one workload/cohort and time window, then deliver a PDF with supported findings, limitations and prioritized recommendations for testing. Data handling and scope are agreed before transfer; savings and exhaustive fault detection are not guaranteed. Email vivaan.thermacompute@gmail.com with “Executive audit” to discuss fit. CrashLens remains free regardless.

Disclosure: I am the founder of ThermaCompute and maintain CrashLens. This article was prepared with AI assistance.

CrashLens: turn GPU job logs into a focused local investigation

Vivaan — Thu, 01 Oct 2026 16:30:58 +0000

ThermaCompute AI is focused on helping GPU teams understand how heat, power and workload behavior affect reliable, useful compute. Our current public tool is CrashLens: a free, local starting point for investigating saved GPU job logs.

A failed GPU job should give you a starting point

A training or inference job fails. You have pages of logs, a worker exception and perhaps a collective timeout. Which message deserves your attention first?

So we made CrashLens to turn those logs into a focused investigation. It scans for known signatures, attaches source line numbers and suggests practical checks. GPU allocation failures, host OOM events, collective timeouts, worker exits, NVIDIA Xid messages and explicit thermal/NVLink messages are kept distinct. A timeout is not proof that the reporting rank caused the failure.

CrashLens runs offline with Python's standard library. No account, API key, GPU or production SSH access is required. The CLI makes no network requests and changes no GPU settings. Your input logs and reports stay local. Review reports before sharing: secret masking is best effort and incomplete.

Try the synthetic demo

You need Python 3.10 or newer:

git clone https://github.com/rohitcn-hub/thermacompute-crashlens.git
cd thermacompute-crashlens
python crashlens.py examples/synthetic.log --out demo-report

Open demo-report/report.html locally. The directory also contains Markdown and JSON reports. Use a new output directory for each run.

Try CrashLens on GitHub

This is experimental version 0.2.0, tested on synthetic fixtures rather than validated across production fleets. It recognizes patterns, not confirmed root causes, and does not automatically fix jobs. No matches means insufficient evidence, not a healthy system.

Did it give you a useful next check? What was missing? Tell us in the comments or first-run feedback issue. Please use invented examples, not private production logs.

Optional: a deeper $80 thermal/GPU efficiency audit

CrashLens helps organize crash-log evidence. ThermaCompute's $80 audit separately reviews suitable temperature, power and reported-throttling telemetry, delivering supported findings, prioritized checks and a PDF with assumptions and limitations.

To upgrade, email vivaan.thermacompute@gmail.com with Thermal Audit in the subject, your GPU model/count, the problem you are investigating and your CSV headers. We confirm data suitability and scope before payment. The audit does not guarantee savings or include production changes; it can help define whether a larger engagement makes sense. CrashLens remains free, and the audit does not commit you to a subscription.

I'm building ThermaCompute and would value honest technical feedback.