GPU Monitoring & Metrics for MLOps

#ai #webdev #mlops #gpu

GPU Hardware Circuitry
GPU Infrastructure Monitoring for MLOps: Essential nvidia-smi Commands & Metrics
When running large-scale AI workloads, memory leaks, thermal throttling, and runaway PyTorch processes can crash inference nodes without warning. System administrators managing AI infrastructure need real-time visibility into GPU VRAM utilization, power draw, and process ownership.

Advanced nvidia-smi Query Commands Instead of running standard nvidia-smi, extract structured CSV data for custom log monitors or automated scripts:

Stream GPU VRAM usage, temperature, and power consumption every 2 seconds

nvidia-smi --query-gpu=timestamp,name,temperature.gpu,utilization.gpu,utilization.memory,memory.used,memory.free --format=csv -l 2

Identifying Specific GPU Process Owners To identify which Python or Docker container is consuming GPU VRAM on multi-tenant servers:

List all active GPU compute processes with Process IDs (PIDs)

nvidia-smi pmon -s u -v
Once you locate a runaway process PID, inspect its systemd parent unit or Docker container:

Inspect process details by PID

ps -up [PID]

Find associated Docker container ID

cat /proc/[PID]/cgroup | grep docker

Setting GPU Power Limits for Stability To prevent server power supplies from tripping during sudden LLM batch inference spikes, cap the maximum wattage allowed per GPU card: