Deploy TensorRT-LLM on NVIDIA H100 & RTX 6000 — Step-by-Step Tutorial

The demand for fast, affordable Large Language Model (LLM) inference is at an all-time high. Every additional millisecond of latency and every extra dollar per million tokens directly impacts product economics. To maximize throughput and lower costs, enterprise infrastructure teams are standardizing on the two most proven, scalable, and immediately available GPU architectures on the market: the NVIDIA H100 (Hopper) and the RTX Pro 6000 (Ada Lovelace).

However, having the right raw silicon is only half the battle. To extract maximum ROI from these GPUs, you need a highly optimized software stack. That engine is TensorRT-LLM, NVIDIA's open-source library for compiling and serving large language models at production scale.

This tutorial walks through the exact steps required to deploy a large model — using Llama 3 as the working example — on H100 and RTX Pro 6000 hardware with TensorRT-LLM, from pulling the correct container to serving live inference requests via an API.

Why H100 & RTX 6000 + TensorRT-LLM Is a Game Changer

1. FP8 Precision (Native Hardware Acceleration)

Both the H100 and the RTX Pro 6000 feature native support for FP8 (8-bit floating point) quantization via their fourth-generation Tensor Cores. Cutting weight precision from FP16 to FP8 roughly halves the memory footprint of a model's weights. This has two massive compounding effects:

• Higher Model Density: More of the model fits in GPU memory. A quantized Llama 3 70B can fit comfortably across fewer GPUs, drastically cutting hardware requirements.
• Expanded Paged KV Cache: More VRAM headroom is left for the KV cache, allowing a single server to sustain much larger batch sizes and process much longer context windows simultaneously.

2. In-Flight Batching

Real-world LLM traffic doesn't arrive in neat, uniform batches. TensorRT-LLM features in-flight batching (also called continuous batching). It evaluates the request queue at every generation step. As soon as one user's request finishes, a new one is instantly inserted into the batch — meaning your GPU compute is never left sitting idle waiting for a slower request.

3. Lower Cost Per Token

Because GPU ownership or rental cost is fixed per hour, generating more tokens per hour directly lowers your effective cost per million tokens. The combination of FP8 speed and in-flight batching delivers enterprise-grade generation speeds at a fraction of the cost.

---

Hardware & Software Prerequisites

Before starting the deployment, confirm the following:

GPU Hardware (Available on GPUYard)

You will need an NVIDIA Hopper or Ada generation GPU to take full advantage of FP8 Tensor Cores.

• NVIDIA H100 Servers: The ultimate data center powerhouse, ideal for massive MoE models (DeepSeek, Mixtral) and high-concurrency production endpoints.
• NVIDIA RTX Pro 6000 Servers: A highly cost-effective 48GB VRAM powerhouse, perfect for single-GPU inference (Llama 3 8B), AI agent workflows, and dev/test environments.

Operating System & Drivers

• Ubuntu 22.04 LTS or 24.04 LTS.
• A current NVIDIA driver (e.g., v535 or newer) that supports your target CUDA toolkit version.

Verify your GPU and driver are visible:

nvidia-smi

Docker & NVIDIA Container Toolkit

TensorRT-LLM ships as prebuilt NGC containers. You must install Docker and the NVIDIA Container Toolkit to give containers direct GPU access:

# Install NVIDIA Container Toolkit (Ubuntu)
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Confirm the container runtime can see your H100 or RTX 6000
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step-by-Step Deployment Guide

Step A: Pull the Official TensorRT-LLM Container

NVIDIA publishes versioned TensorRT-LLM release containers on NGC. Replace x.xx.x with the current release tag (e.g., 0.10.0 or newer):

docker pull nvcr.io/nvidia/tensorrt-llm/release:x.xx.x

docker run --rm -it --ipc=host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus all \
  -v $(pwd):/workspace \
  nvcr.io/nvidia/tensorrt-llm/release:x.xx.x

# Sanity check inside the container
python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)"

Step B: Download the Model Weights

We will use Llama 3 as the working example. Ensure you have accepted the license terms on Hugging Face.

pip install -U "huggingface_hub[cli]"
huggingface-cli login   # Enter your Hugging Face access token

huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct \
  --local-dir /workspace/models/llama3-8b

Step C: Build the TensorRT Engine (FP8 Precision)

Building an engine is a two-part process: convert the standard Hugging Face weights to TensorRT-LLM's optimized format, and then compile an engine targeting your exact GPU.

# 1. Convert the checkpoint and quantize to FP8
python3 examples/llama/convert_checkpoint.py \
  --model_dir /workspace/models/llama3-8b \
  --output_dir /workspace/checkpoints/llama3-8b-fp8 \
  --dtype float16 \
  --qformat fp8 \
  --calib_size 1024

# 2. Compile the highly optimized TRT engine
trtllm-build \
  --checkpoint_dir /workspace/checkpoints/llama3-8b-fp8 \
  --output_dir /workspace/engines/llama3-8b-fp8 \
  --gemm_plugin fp8 \
  --max_batch_size 64 \
  --max_input_len 4096 \
  --max_seq_len 8192

Pro-Tip: The --calib_size 1024 flag runs calibration during the FP8 conversion to prevent accuracy loss. Setting --max_batch_size appropriately based on your server's total VRAM allows TensorRT-LLM to fully utilize in-flight batching.

Step D: Serve the Model using Triton Inference Server

The compiled TensorRT-LLM engine is best served through NVIDIA's Triton Inference Server, which handles HTTP/gRPC requests securely and efficiently.

docker run --rm -it --gpus all \
  --shm-size=2g \
  -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v /workspace/engines/llama3-8b-fp8:/models/llama3-8b/1 \
  nvcr.io/nvidia/tritonserver:xx.xx-trtllm-python-py3 \
  tritonserver --model-repository=/models

Once Triton reports that the model is loaded, you can send API calls to port 8000 just like you would with an OpenAI-compatible endpoint.

H100 vs. RTX Pro 6000: Which GPU is Right for You?

Both GPUs are fully supported for this workflow, but they serve different business needs:

Feature	NVIDIA H100 (SXM / PCIe)	NVIDIA RTX Pro 6000 (Ada)
Best Used For	Massive models (70B+), thousands of concurrent users, MoE	Smaller models (8B-35B), RAG pipelines, dev/test
GPU Memory	80 GB HBM3	48 GB GDDR6 w/ ECC
FP8 Tensor TFLOPS	~3,958 TFLOPS	~1,457 TFLOPS
Cost Profile	Premium, highest throughput	Highly cost-effective per GB of VRAM

Scale Your AI Infrastructure Today

Deploying a Large Language Model doesn't require waiting months for unreleased hardware. The NVIDIA H100 and RTX Pro 6000 are proven champions of production AI today. When combined with TensorRT-LLM's FP8 precision and in-flight batching, these GPUs deliver massive cost savings.

Don't let compute bottlenecks stall your product roadmap. Get instant access to top-tier AI hardware configured exactly for this workflow at GPUYard.