Your training run died at epoch 47. The logs show nothing useful — just a silent hang and a CUDA out-of-memory error three hours later. Meanwhile, your GPU has been sitting at 91°C for the last twenty minutes, quietly throttling itself into uselessness.
If you're renting GPU time on a remote box, you need visibility into two numbers: utilization percentage and temperature in Celsius. This guide walks through monitoring both from the command line, building a lightweight dashboard, and automating alerts so you find out about problems before they cost you a day of compute.
## Why Remote GPU Monitoring Is Different
On your local workstation, you can hear the fans spin up. On a remote server, you get nothing — no noise, no heat, no visual feedback. A GPU that's thermally throttling still reports "running" in your process list. It just runs at 40% speed.
Thermal throttling is when a GPU reduces its clock speed to avoid damage from heat. Most data center GPUs (A100, H100, RTX 4090) start throttling between 83°C and 90°C. Above 95°C, you risk hardware degradation or an emergency shutdown.
The fix is simple: poll `nvidia-smi` on a schedule, log the output, and alert when thresholds break.
## The Foundation: nvidia-smi
`nvidia-smi` (NVIDIA System Management Interface) is the command-line tool that ships with every NVIDIA driver. It reads sensor data directly from the GPU.
The default output is human-readable but painful to parse:
bash nvidia-smi
For scripting, use the query flag with CSV output:
bash nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \ --format=csv,noheader,nounits
Sample output:
0, NVIDIA A100-SXM4-40GB, 87, 38120, 40960, 74, 245.30
1, NVIDIA A100-SXM4-40GB, 12, 2048, 40960, 41, 68.15
Fields map in order: index, name, GPU utilization %, memory used (MB), memory total (MB), temperature (°C), power draw (W).
This one command gives you everything you need for a basic monitoring loop.
## Step 1: A One-Liner Health Check
Before building anything elaborate, run this every time you SSH in:
bash nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu \ --format=csv,noheader | \ awk -F', ' '{printf "GPU %s: util=%s%% mem=%s/%s MB temp=%s°C\n", $1, $2, $3, $4, $5}'
It prints a clean per-GPU summary. If any GPU shows 0% utilization while your training script claims to be running, something is wrong — probably a dead data loader or a hung NCCL collective.
## Step 2: Log Metrics to a File
For historical data, append to a CSV on a cron schedule. Create a script:
bash #!/bin/bash # /usr/local/bin/gpu-log.sh
LOG=/var/log/gpu-metrics.csv
if [ ! -f "$LOG" ]; then echo "timestamp,gpu_index,util_pct,mem_used_mb,mem_total_mb,temp_c,power_w" > "$LOG" fi
TS=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \ --format=csv,noheader,nounits | \ awk -v ts="$TS" -F', ' '{print ts","$1","$2","$3","$4","$5","$6}' >> "$LOG"
Schedule it every 30 seconds:
bash crontab -e
- * * * * /usr/local/bin/gpu-log.sh
- * * * * sleep 30; /usr/local/bin/gpu-log.sh
Cron's minimum granularity is one minute, so the two-line trick gives you 30-second sampling. For finer resolution, use a systemd timer or just run a `while true; do ...; sleep 5; done` loop inside `tmux`.
## Step 3: Alert on Temperature Spikes
Logging is useless if nobody reads the log. Add a threshold check:
bash #!/bin/bash # /usr/local/bin/gpu-alert.sh
THRESHOLD=85
WEBHOOK_URL="https://your-webhook-endpoint.example"
nvidia-smi --query-gpu=index,temperature.gpu,utilization.gpu \ --format=csv,noheader,nounits | while IFS=', ' read -r idx temp util; do if [ "$temp" -ge "$THRESHOLD" ]; then
MSG="GPU $idx at ${temp}°C (util ${util}%). Check cooling."
curl -s -X POST "$WEBHOOK_URL" \ -H "Content-Type: application/json" \ -d "{\"text\":\"$MSG\"}" fi done
Swap the webhook for Slack, Discord, or PagerDuty — all accept a simple JSON POST. Run this every minute via cron. If you're on a shared host, keep alert frequency low to avoid rate limits.
## Step 4: A Lightweight Dashboard with nvitop
If you want a live view without leaving the terminal, install `nvitop`:
bash pip install nvitop nvitop
It's an interactive TUI (terminal user interface) that shows per-process GPU usage, memory, temperature, and power — similar to `htop` but for GPUs. It also works over SSH without any port forwarding. For most developers, this replaces the need for a browser-based dashboard entirely.
For multi-server fleets, `nvitop --monitor` runs in a non-interactive mode suitable for piping into logging systems.
## Where to Run This
You can run any of the above on a local workstation with a GPU. But the setup matters most when you're paying for remote compute, because that's where silent failures get expensive.
I've tested this monitoring stack on two providers that give you full `nvidia-smi` access and root SSH:
- **[PowerVPS](https://powervps.net/?from=32)** — bare-metal GPU instances with predictable pricing. Good when you need consistent thermal behavior over long training runs and don't want noisy-neighbor effects from virtualization.
- **[Immers Cloud](https://en.immers.cloud/signup/r/20241007-8310688-334/)** — hourly GPU rentals with a straightforward API. Handy for short experiments where you spin up a box, run the monitoring script, and tear it down.
Both expose real hardware sensors, which is the whole point — some managed platforms hide temperature data behind their own dashboards, which defeats the purpose of `nvidia-smi`.
If you're still comparing options, [Server Rental Guide](https://serverrental.store) has a breakdown of GPU server rental providers by price, region, and hardware tier. Worth a read before you commit to a monthly plan.
## Step 5: Watch for the Silent Killers
Two failure modes don't show up in a simple temperature check:
**Memory creep.** A training loop that leaks tensors will slowly fill VRAM until it OOMs. Track `memory.used` over time and alert when it grows monotonically without dropping. A quick check:
bash nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits
If this number only goes up across epochs, you have a leak.
**Utilization gaps.** If `utilization.gpu` drops below 20% for more than a minute while your job is "running," you're likely bottlenecked on data loading or stuck in a deadlock. Add a check that flags sustained low utilization.
bash # Alert if GPU 0 sits under 20% for 3 consecutive samples
LOW_COUNT=0
while true; do UTIL=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits -i 0) if [ "$UTIL" -lt 20 ]; then
LOW_COUNT=$((LOW_COUNT + 1))
[ "$LOW_COUNT" -ge 3 ] && echo "GPU 0 idle for 3 samples — investigate" && LOW_COUNT=0 else
LOW_COUNT=0
fi sleep 60 done
Run this in `tmux` alongside your training job. It's crude but catches the deadlock case that kills multi-day runs.
## Putting It Together
A complete monitoring setup is four pieces:
1. **`nvidia-smi` query** — the raw data source, runs in milliseconds. 2. **CSV logging on a cron schedule** — builds a history you can graph later. 3. **Threshold alerts via webhook** — pushes problems to where you'll see them. 4. **`nvitop` for live inspection** — the interactive fallback when something looks off.
Total setup time: about fifteen minutes. Total cost: zero. Total savings: potentially an entire training run.
## Conclusion
Remote GPU monitoring comes down to polling `nvidia-smi` on a schedule and acting on the numbers. Log utilization, memory, and temperature to a CSV. Alert when temperature crosses 85°C or utilization flatlines. Use `nvitop` when you need to see what's happening right now.
The developers who lose days to silent GPU failures are the ones who never set this up. Don't be one of them. Pick a provider that gives you real hardware access — PowerVPS or Immers Cloud both work — and wire up the scripts above before your next long run.
Top comments (0)