How to Monitor GPU Usage and Temperature on a Remote Server

A training job looked healthy from the application logs, but the server’s GPU was sitting at 92°C and barely doing any work. I caught it by SSHing in and checking nvidia-smi—a quick fix, but not a useful way to monitor a machine overnight. This guide covers live checks, lightweight logging, and what to do when you need monitoring that survives your SSH session.

Start with nvidia-smi

For NVIDIA GPUs, nvidia-smi is the quickest way to check whether a device is visible and how busy it is. Run it on the server, or remotely over SSH:

ssh user@gpu-host 'nvidia-smi'

Look at the GPU utilization, memory use, temperature, power draw, and running processes. The exact fields depend on the GPU and driver, and some values may show as N/A. GPU utilization is not the same as memory use: a job can reserve most of the GPU’s memory while doing little computation.

For a compact readout, query only the fields you care about:

ssh user@gpu-host 'nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw --format=csv'

This is useful for a spot check, but it only tells you what is happening now. It won’t tell you whether the GPU was idle or overheating ten minutes ago.

Check that the driver is working

If nvidia-smi is missing or reports that it cannot communicate with the driver, first confirm that the server has an NVIDIA GPU and that its NVIDIA driver is installed and loaded:

lspci | grep -i nvidia
nvidia-smi

You need a working driver, but you generally don’t need the CUDA toolkit just to monitor the GPU. Installing the driver depends on the operating system and kernel; use your distribution’s recommended NVIDIA driver installation method or your host’s instructions. If a driver was just installed or updated, a reboot may be required.

On a rented GPU server, also check that the GPU is attached to your instance and that you’re running the command on the host, not inside a container that lacks GPU access. A container’s visibility depends on its runtime and GPU device configuration.

Watch live from your terminal

For a quick live view, use watch:

watch -n 2 nvidia-smi

This refreshes the full output every two seconds. To watch a smaller set of metrics instead:

watch -n 2 'nvidia-smi --query-gpu=index,utilization.gpu,memory.used,temperature.gpu,power.draw --format=csv'

Press Ctrl+C to stop. For a long-running session, use tmux or screen so the terminal keeps running if your SSH connection drops. You can also use nvidia-smi’s built-in loop mode:

nvidia-smi --loop-ms=2000 --query-gpu=index,utilization.gpu,memory.used,temperature.gpu,power.draw --format=csv

A practical tip: sample at an interval that matches the problem you’re investigating. Two-second checks are useful for interactive debugging; they can miss short spikes. For longer-term trends, sampling every five or ten seconds is often enough and generates less data.

Keep a simple CSV log

When you need history, a small shell script can log metrics without adding a monitoring stack. On the server, create a script that adds a timestamped header and appends a sample every five seconds:

mkdir -p "$HOME/gpu-monitor"

cat > "$HOME/gpu-monitor/log-gpu.sh" <<'EOF'
#!/usr/bin/env bash
set -u

log="$HOME/gpu-monitor/gpu.csv"
if [ ! -e "$log" ]; then
  echo "timestamp,index,name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw" > "$log"
fi

while true; do
  nvidia-smi \
    --query-gpu=timestamp,index,name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \
    --format=csv,noheader,nounits >> "$log" 2>> "$HOME/gpu-monitor/nvidia-smi-errors.log"
  sleep 5
done
EOF

chmod +x "$HOME/gpu-monitor/log-gpu.sh"

Start it in the background:

nohup "$HOME/gpu-monitor/log-gpu.sh" >/dev/null 2>&1 &
echo $! > "$HOME/gpu-monitor/log-gpu.pid"

Check that it is writing data:

tail -n 5 "$HOME/gpu-monitor/gpu.csv"

Copy the log to your workstation when you want to inspect it:

scp user@gpu-host:~/gpu-monitor/gpu.csv .

Stop the logger with:

kill "$(cat "$HOME/gpu-monitor/log-gpu.pid")"

This is intentionally simple: it starts again only if you restart it, and it doesn’t rotate the log. For a logger that should survive reboots, use a systemd service; for a one-off job, nohup is often enough.

Another tip from practice: monitor disk space as well as GPU metrics. A frequent sampler running for days can create a surprisingly large log, especially on multi-GPU hosts. Rotate or delete old logs if you keep this running long-term.

Know what the readings mean

High utilization usually means the GPU is doing work, but it doesn’t prove the job is making good progress. Compare utilization with memory use, application logs, and the workload’s own throughput. Low utilization during a data-heavy workload can point to CPU preprocessing or slow storage rather than a GPU problem.

Temperature needs context, too. Safe operating limits vary by GPU model, and a brief warm period is different from sustained operation near the card’s limit. If temperatures stay high, check airflow, fan behavior, workload, and the manufacturer’s specifications. A remote host’s fans and cooling may not be under your control, so contact the provider if the machine appears thermally constrained.

For AMD GPUs, NVIDIA’s nvidia-smi is not the right tool. Depending on the GPU and installed ROCm version, rocm-smi can report device status and temperatures. Check the ROCm documentation for the supported commands and metrics for your hardware.

When to move beyond a CSV

A CSV is fine for a single server or a short investigation. If you need dashboards, alerts, retention, or several hosts, use a monitoring stack that collects metrics continuously. NVIDIA’s DCGM Exporter can expose GPU metrics for Prometheus, but deployment depends on the host’s NVIDIA driver, container runtime, and GPU access configuration. Keep dashboards and exporters private, and expose them through an authenticated network path rather than an open public port.

If you don’t already have a GPU machine, compare GPU model, VRAM, driver support, and access to monitoring tools before renting. I’ve used PowerVPS and Vast.ai as hosting options; availability and instance details depend on the current offerings. The Server Rental Guide is another place to compare rental options.

Conclusion

Start with nvidia-smi for a quick health check, then log a few useful metrics if you need history. For multiple servers or alerting, add a proper metrics stack—and make sure it can reach the GPU through the host’s driver and runtime configuration.