DEV Community

Cover image for Optimizing GPU Resource Planning with NVIDIA Dynamo
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya Originally published at docs.vultr.com

Optimizing GPU Resource Planning with NVIDIA Dynamo

#kubernetes #nvidia #gpu #llm

GPU resource allocation for LLM inference presents complex optimization challenges. Workloads exhibit variable request patterns, unpredictable sequence lengths, and strict latency requirements, making static GPU allocation inefficient. Overprovisioning wastes expensive compute resources, while underprovisioning violates service level agreements (SLAs) and degrades user experience. NVIDIA Dynamo's SLA Planner automates GPU resource management through SLA-driven profiling and intelligent autoscaling, analyzing performance requirements (time-to-first-token, inter-token latency), profiling workload characteristics, and dynamically adjusting GPU worker replicas to meet latency targets while minimizing resource consumption. This guide covers deploying Dynamo's SLA Planner on Kubernetes, including infrastructure setup with Prometheus monitoring, DynamoGraphDeploymentRequest (DGDR) configuration for automated profiling and deployment, and Grafana dashboard integration for visualizing scaling decisions and performance metrics. By the end, you'll have a Kubernetes-based LLM deployment that automatically scales GPU workers to meet SLA targets.

Prerequisites

Before you begin, ensure you:

  • Have access to a Kubernetes cluster with NVIDIA GPUs configured with the NVIDIA GPU Operator. Use a cluster with at least 2 GPU nodes.
  • Install kubectl and configure it to access your cluster.
  • Install Helm 3 for package management.
  • Install jq for JSON processing in command-line operations.
  • Create an NVIDIA NGC account and generate an NGC API key with NGC Catalog access for pulling container images.
  • Create a Hugging Face account and generate an access token for gated models like Llama.
  • Have cluster-admin permissions to install Custom Resource Definitions (CRDs) and create namespaces.

Key Components

This deployment uses several components that work together to provide automated GPU resource planning.

  • NVIDIA Dynamo Platform provides the orchestration layer with the Dynamo Operator that manages DynamoGraphDeployment (DGD) and DynamoGraphDeploymentRequest (DGDR) resources. The operator automates the profiling, configuration generation, and deployment lifecycle.
  • SLA Planner acts as the autoscaling controller that monitors inference metrics from Prometheus and adjusts worker replica counts to meet SLA targets. It uses load predictors (constant, ARIMA, Kalman, or Prophet) to forecast request rates and sequence lengths, then calculates optimal GPU allocations based on TTFT (time-to-first-token) and ITL (inter-token latency) requirements.
  • DynamoGraphDeploymentRequest (DGDR) serves as the declarative interface for deploying models with performance constraints. Users specify the model name, backend type (vLLM, SGLang, or TensorRT-LLM), SLA targets, and container images. The operator automatically profiles the configuration and generates an optimized DynamoGraphDeployment.
  • DynamoGraphDeployment (DGD) defines the low-level implementation details for a deployed model, including frontend services, prefill workers, decode workers, and the SLA planner component.
  • Prometheus collects inference metrics from the Dynamo frontend, including request counts, TTFT, ITL, input sequence length (ISL), and output sequence length (OSL).
  • Grafana visualizes planner decisions, worker counts, GPU usage, observed metrics versus predicted metrics, and correction factors.

1. Verify GPU Availability

1. List GPU nodes in your cluster:

$ kubectl get nodes -o json | jq '.items[] | select(.status.capacity."nvidia.com/gpu") | {name: .metadata.name, gpus: .status.capacity."nvidia.com/gpu"}'
Enter fullscreen mode Exit fullscreen mode

The output displays nodes with available GPUs.

{
  "name": "gpu-node-1",
  "gpus": "2"
}
{
  "name": "gpu-node-2",
  "gpus": "2"
}
Enter fullscreen mode Exit fullscreen mode

2. Verify GPU resources are allocatable:

$ kubectl describe nodes | grep -A 5 "nvidia.com/gpu"
Enter fullscreen mode Exit fullscreen mode

Ensure at least one node shows nvidia.com/gpu: 1 or higher under the Allocatable section.

2. Label GPU Nodes

1. Check if GPU nodes are already labeled:

$ kubectl get nodes --show-labels | grep nvidia.com/gpu.present
Enter fullscreen mode Exit fullscreen mode

If the output displays nodes with the nvidia.com/gpu.present=true label, skip to the next section. If no output appears, proceed with labeling.

2. Identify your GPU nodes:

$ kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

3. Label each GPU node with the Dynamo GPU label. Replace NODE_NAME with your GPU node name:

$ kubectl label node NODE_NAME nvidia.com/gpu.present=true
Enter fullscreen mode Exit fullscreen mode

Repeat this command for each GPU node in your cluster.

4. Verify the labels were applied:

$ kubectl get nodes --show-labels | grep nvidia.com/gpu.present
Enter fullscreen mode Exit fullscreen mode

3. Run Pre-Deployment Checks

1. Clone the Dynamo repository if you haven't already:

$ git clone https://github.com/ai-dynamo/dynamo.git
$ cd dynamo
Enter fullscreen mode Exit fullscreen mode

2. Check out the latest stable release. Visit the Dynamo releases page to find the latest version:

$ git checkout release/0.9.0
Enter fullscreen mode Exit fullscreen mode

3. Run the pre-deployment check script:

$ ./deploy/pre-deployment/pre-deployment-check.sh
Enter fullscreen mode Exit fullscreen mode

The script validates kubectl connectivity, default StorageClass configuration, GPU resources, and GPU Operator installation.

========================================
  Dynamo Pre-Deployment Check Script
========================================

✅ kubectl Connectivity: PASSED
✅ Default StorageClass: PASSED
✅ Cluster GPU Resources: PASSED
✅ GPU Operator: PASSED

All pre-deployment checks passed!
Your cluster is ready for Dynamo deployment.
Enter fullscreen mode Exit fullscreen mode

If any checks fail, review the error messages and resolve the issues before proceeding.

4. Install Prometheus

1. Add the Prometheus Helm repository:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
Enter fullscreen mode Exit fullscreen mode

2. Install the kube-prometheus-stack with configuration for Dynamo metric scraping:

$ helm install prometheus -n monitoring --create-namespace prometheus-community/kube-prometheus-stack \
    --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
    --set-json 'prometheus.prometheusSpec.podMonitorNamespaceSelector={}' \
    --set prometheus.prometheusSpec.probeNamespaceSelector.matchLabels=null
Enter fullscreen mode Exit fullscreen mode

3. Verify Prometheus is running:

$ kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus
Enter fullscreen mode Exit fullscreen mode

5. Install Dynamo Platform

1. Set environment variables for the installation:

$ export NAMESPACE=dynamo-system
$ export RELEASE_VERSION=0.9.0
Enter fullscreen mode Exit fullscreen mode

2. Navigate to the home directory:

$ cd ~
Enter fullscreen mode Exit fullscreen mode

3. Fetch the Dynamo CRDs Helm chart:

$ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz
Enter fullscreen mode Exit fullscreen mode

4. Install the CRDs:

$ helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz
Enter fullscreen mode Exit fullscreen mode

5. Fetch the Dynamo Platform Helm chart:

$ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz
Enter fullscreen mode Exit fullscreen mode

6. Install the Dynamo Platform with the operator enabled and the Prometheus endpoint configured:

$ helm upgrade --install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz \
    --namespace ${NAMESPACE} \
    --create-namespace \
    --set dynamo-operator.controllerManager.manager.image.tag=0.9.0 \
    --set grove.enabled=true \
    --set kai-scheduler.enabled=true \
    --set nats.reloader.image.tag=0.22.3 \
    --set prometheusEndpoint=http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090
Enter fullscreen mode Exit fullscreen mode

The command configures the following components:

  • --namespace ${NAMESPACE} and --create-namespace: Installs the platform in the dynamo-system namespace, creating it if it doesn't exist.
  • --set dynamo-operator.controllerManager.manager.image.tag=0.9.0: Specifies the Dynamo Operator version that manages DGD and DGDR lifecycle operations.
  • --set grove.enabled=true: Enables Grove, the distributed runtime that coordinates communication between frontend and worker services.
  • --set kai-scheduler.enabled=true: Enables the KAI (Kubernetes AI) scheduler for advanced GPU scheduling and resource allocation.
  • --set nats.reloader.image.tag=0.22.3: Specifies the NATS configuration reloader version that automatically reloads NATS server configuration changes without downtime.
  • --set prometheusEndpoint=...: Configures the SLA Planner to query inference metrics from the Prometheus service in the monitoring namespace.

If the installation fails with a Grove webhook certificate conflict error, uninstall the release and reinstall:

$ helm uninstall dynamo-platform --namespace ${NAMESPACE}

Then run the installation command again. The warning "tls: failed to find any PEM data in certificate input" is harmless and can be ignored.

7. Verify the operator is running:

$ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/name=dynamo-operator
Enter fullscreen mode Exit fullscreen mode 
NAME                                                              READY   STATUS      RESTARTS   AGE
dynamo-platform-dynamo-operator-controller-manager-77464cd6pcjt   2/2     Running     0          45s
Enter fullscreen mode Exit fullscreen mode

6. Create Image Pull Secrets

1. Export your NGC API key. Replace YOUR_NGC_API_KEY with your NGC API key from the NGC portal:

$ export NGC_API_KEY=YOUR_NGC_API_KEY
Enter fullscreen mode Exit fullscreen mode

2. Export your Hugging Face token. Replace YOUR_HF_TOKEN with your Hugging Face access token:

$ export HF_TOKEN=YOUR_HF_TOKEN
Enter fullscreen mode Exit fullscreen mode

3. Create a Docker registry secret for NGC access using the exported variable:

$ kubectl create secret docker-registry nvcr-imagepullsecret \
    --docker-server=nvcr.io \
    --docker-username='$oauthtoken' \
    --docker-password=${NGC_API_KEY} \
    -n ${NAMESPACE}
Enter fullscreen mode Exit fullscreen mode

4. Create a Hugging Face token secret for model downloads using the exported variable:

$ kubectl create secret generic hf-token-secret \
    --from-literal=HF_TOKEN=${HF_TOKEN} \
    -n ${NAMESPACE}
Enter fullscreen mode Exit fullscreen mode

7. Select AI Configurator System and Backend Version

The AI Configurator performs offline profiling to generate optimal deployment configurations without running actual workloads. The aicSystem parameter specifies your GPU hardware type, and the aicBackendVersion parameter specifies the inference backend version for profiling data.

Available GPU Systems

System Description Supported Backends
a100_sxm NVIDIA A100 SXM TensorRT-LLM
b200_sxm NVIDIA B200 SXM SGLang, TensorRT-LLM
gb200_sxm NVIDIA GB200 SXM TensorRT-LLM
h100_sxm NVIDIA H100 SXM vLLM, SGLang, TensorRT-LLM
h200_sxm NVIDIA H200 SXM vLLM, SGLang, TensorRT-LLM
l40s NVIDIA L40S SGLang, TensorRT-LLM

Backend Version Compatibility

vLLM Backend

GPU System Supported Versions
H100 SXM 0.12.0
H200 SXM 0.12.0

Example configuration for vLLM:

sweep:
  useAiConfigurator: true
  aicSystem: h200_sxm
  aicBackendVersion: "0.12.0"
Enter fullscreen mode Exit fullscreen mode

SGLang Backend

GPU System Supported Versions
B200 SXM 0.5.6.post2
H100 SXM 0.5.6.post2
H200 SXM 0.5.6.post2
L40S 0.5.5.post3

Example configuration for SGLang:

sweep:
  useAiConfigurator: true
  aicSystem: h200_sxm
  aicBackendVersion: "0.5.6.post2"
Enter fullscreen mode Exit fullscreen mode

TensorRT-LLM Backend

GPU System Supported Versions
A100 SXM 1.0.0
B200 SXM 1.0.0rc6, 1.2.0rc5
GB200 SXM 1.0.0rc6, 1.2.0rc5
H100 SXM 1.0.0rc3, 1.2.0rc5
H200 SXM 1.0.0rc3, 1.2.0rc5
L40S 1.0.0

Example configuration for TensorRT-LLM:

sweep:
  useAiConfigurator: true
  aicSystem: h200_sxm
  aicBackendVersion: "1.2.0rc5"
Enter fullscreen mode Exit fullscreen mode

Selecting the Correct Configuration

Match your configuration to your cluster's GPU hardware: identify your GPU type with kubectl describe nodes, select the matching aicSystem code, and choose a compatible backend version from the tables above. An unsupported combination causes the profiling job to fail with an error indicating no profiling data is available for that configuration.

8. Deploy a Model with SLA-Driven Profiling

1. Create a DGDR manifest file with SLA specifications for the NVIDIA Nemotron Nano 4B model using the vLLM backend:

$ cat << 'EOF' > nemotron-nano-4b-dgdr.yaml
apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeploymentRequest
metadata:
  name: nemotron-nano-sla
spec:
  model: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1
  backend: vllm
  autoApply: false
  profilingConfig:
    profilerImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
    config:
      sla:
        isl: 2000
        osl: 100
        ttft: 300.0
        itl: 30.0
      sweep:
        useAiConfigurator: true
        aicSystem: h200_sxm
        aicHfId: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1
        aicBackendVersion: "0.12.0"
      planner:
        plannerMinEndpoint: 1
        plannerMaxGpuBudget: 4
        plannerAdjustmentInterval: 30
  deploymentOverrides:
    workersImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
    name: vllm-disagg
EOF
Enter fullscreen mode Exit fullscreen mode

The manifest specifies a Nemotron Nano 4B model with SLA targets (ISL 2000, OSL 100, TTFT 300ms, ITL 30ms), AI Configurator offline profiling for H200 SXM GPUs (20-30 seconds versus 2-4 hours for online profiling), a planner budget of 1-4 GPUs with 30-second adjustment intervals, and autoApply: false so you can review the generated DGD before deployment.

AI Configurator System Selection: The aicSystem parameter must match your GPU hardware type. Available options are: a100_sxm, b200_sxm, gb200_sxm, h100_sxm, h200_sxm, and l40s. The aicBackendVersion parameter must match the available version for your selected system and backend combination. For vLLM backend: H100/H200 SXM supports version 0.12.0. For SGLang: H100/H200 SXM and B200 SXM support version 0.5.6.post2, L40S supports 0.5.5.post3. For TensorRT-LLM: consult the AI Configurator documentation for supported versions.

2. Apply the DGDR manifest:

$ kubectl apply -f nemotron-nano-4b-dgdr.yaml -n ${NAMESPACE}
Enter fullscreen mode Exit fullscreen mode

3. Monitor the DGDR status:

$ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -w
Enter fullscreen mode Exit fullscreen mode

The DGDR transitions through states: Pending → Profiling → Ready.

NAME                MODEL                                    BACKEND   STATE       DGD-STATE   AGE
nemotron-nano-sla   nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1   vllm      Pending                 5s
nemotron-nano-sla   nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1   vllm      Profiling               10s
nemotron-nano-sla   nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1   vllm      Ready                   35s
Enter fullscreen mode Exit fullscreen mode

4. View detailed DGDR status and events:

$ kubectl describe dgdr nemotron-nano-sla -n ${NAMESPACE}
Enter fullscreen mode Exit fullscreen mode

The output displays profiling results and the generated DGD configuration in the status.generatedDeployment field.

9. Extract and Deploy the Generated DGD

1. Extract the generated DGD configuration to a file:

$ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -o jsonpath='{.status.generatedDeployment}' > nemotron-nano-dgd-raw.yaml
Enter fullscreen mode Exit fullscreen mode

2. Add required Pod metadata environment variables and the --trust-remote-code flag to worker services using jq:

$ cat nemotron-nano-dgd-raw.yaml | jq '.spec.services |= with_entries(
    .value.extraPodSpec.mainContainer.env = [
      {"name":"POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}},
      {"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},
      {"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},
      {"name":"POD_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}}
    ] |
    if .value.componentType == "worker" then
      if (.value.extraPodSpec.mainContainer.args // []) | any(. == "--trust-remote-code") | not then
        .value.extraPodSpec.mainContainer.args += ["--trust-remote-code"]
      else
        .
      end
    else
      .
    end
  )' > nemotron-nano-dgd.yaml
Enter fullscreen mode Exit fullscreen mode

This injects Pod metadata environment variables into all services (for Grove distributed runtime coordination) and adds the --trust-remote-code flag only to worker service arguments.

3. Apply the modified DGD configuration to deploy the model:

$ kubectl apply -f nemotron-nano-dgd.yaml -n ${NAMESPACE}
Enter fullscreen mode Exit fullscreen mode

4. Verify the DynamoGraphDeployment was created:

$ kubectl get dgd -n ${NAMESPACE}
Enter fullscreen mode Exit fullscreen mode 
NAME          AGE
vllm-disagg   30s
Enter fullscreen mode Exit fullscreen mode

5. Check that the frontend and worker pods are running:

$ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/part-of=vllm-disagg
Enter fullscreen mode Exit fullscreen mode

Wait until all pods show Running status before proceeding. Initial deployment may take several minutes as worker pods download the model and load it into GPU memory; monitor with kubectl get pods -n ${NAMESPACE} -w.

10. Test the Deployment

1. Get the frontend service name:

$ kubectl get svc -n ${NAMESPACE} -l nvidia.com/dynamo-discovery-backend=kubernetes | grep frontend
Enter fullscreen mode Exit fullscreen mode 
vllm-disagg-frontend   ClusterIP   10.43.69.246   <none>   8000/TCP   2m
Enter fullscreen mode Exit fullscreen mode

2. Forward the frontend service to your local machine:

$ kubectl port-forward -n ${NAMESPACE} svc/vllm-disagg-frontend 8000:8000
Enter fullscreen mode Exit fullscreen mode

Keep this terminal session running.

3. In a new terminal, send a test inference request:

$ curl -X POST http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
      "messages": [
        {
          "role": "user",
          "content": "Explain GPU resource planning in one sentence."
        }
      ],
      "max_tokens": 50,
      "temperature": 0.7
    }'
Enter fullscreen mode Exit fullscreen mode

4. Press Ctrl+C in the port-forward terminal to stop forwarding.

Production Deployments: Port-forwarding is suitable for local testing only. For production deployments with secure external access, configure Gateway API with TLS encryption to expose the frontend service with HTTPS ingress, load balancing, and certificate management.

11. Install Grafana Dashboard

1. Apply the Grafana dashboard ConfigMap:

$ kubectl apply -n monitoring -f https://raw.githubusercontent.com/ai-dynamo/dynamo/release/0.9.0/deploy/observability/k8s/grafana-planner-dashboard-configmap.yaml
Enter fullscreen mode Exit fullscreen mode

2. Expose the Grafana service using a NodePort for quick testing:

$ kubectl patch svc prometheus-grafana \
    -n monitoring \
    -p '{"spec": {"type": "NodePort"}}'
Enter fullscreen mode Exit fullscreen mode

NodePort is for testing only. For production deployments with secure external access, configure Gateway API with TLS encryption to expose Grafana with HTTPS ingress and certificate management.

3. Get the NodePort assigned to Grafana:

$ kubectl get svc prometheus-grafana -n monitoring -o jsonpath='{.spec.ports[0].nodePort}'; echo
Enter fullscreen mode Exit fullscreen mode

4. Retrieve the Grafana admin password:

$ kubectl get secret prometheus-grafana -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d; echo
Enter fullscreen mode Exit fullscreen mode

5. Access Grafana in your browser. Replace NODE_IP with your cluster node's IP address and NODE_PORT with the port from the previous step:

http://NODE_IP:NODE_PORT
Enter fullscreen mode Exit fullscreen mode

Log in with username admin and the password retrieved in the previous step, then navigate to Dashboards → Dynamo Planner Dashboard to view planner metrics.

12. Monitor Automatic Scaling

The SLA Planner continuously monitors request rates, input/output sequence lengths, TTFT, and ITL metrics from Prometheus, then adjusts prefill and decode worker replicas every adjustment interval (30 seconds in this configuration) to meet SLA targets.

Understanding Planner Behavior

As your application receives inference requests, the planner:

  1. Collects Metrics: Gathers request rates, sequence lengths, and latency metrics from the Dynamo frontend via Prometheus.
  2. Predicts Load: Uses the configured load predictor (constant, ARIMA, Kalman, or Prophet) to forecast future traffic patterns.
  3. Calculates Requirements: Determines optimal worker counts needed to meet TTFT and ITL SLA targets based on predicted load.
  4. Scales Workers: Increases or decreases prefill and decode worker replicas by updating the DynamoGraphDeployment.
  5. Applies Corrections: Adjusts predictions based on observed performance versus expected performance using correction factors.

The dashboard visualizes worker counts, cumulative GPU-hours consumed, observed request/TTFT/ITL metrics, load predictions, correction factors, and a timeline of scaling decisions. Use the Namespace dropdown to filter metrics for your specific deployment namespace (dynamo-system in this guide).

Monitor planner logs:

$ kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=vllm-disagg-0-planner,app.kubernetes.io/part-of=vllm-disagg --tail=50 -f
Enter fullscreen mode Exit fullscreen mode 
2026-02-28T00:01:53.952354Z  INFO planner_core.run: New adjustment interval started!
2026-02-28T00:01:53.962025Z  INFO planner_core.observe_metrics: Observed num_req: 0.00 isl: 0.00 osl: 0.00
2026-02-28T00:01:53.962037Z  INFO planner_core.observe_metrics: Observed ttft: 0.00ms itl: 0.00ms
2026-02-28T00:01:53.962463Z  INFO planner_core.predict_load: Predicted load: num_req=0.00, isl=0.00, osl=0.00
2026-02-28T00:01:53.962592Z  INFO planner_core._compute_replica_requirements: Predicted number of engine replicas: prefill=1, decode=1
2026-02-28T00:01:53.966200Z  INFO kubernetes_connector.set_component_replicas: prefill component VllmPrefillWorker already at desired replica count 1, skipping
Enter fullscreen mode Exit fullscreen mode

Press Ctrl+C to stop following the logs.

13. Understanding Load Predictors

The SLA Planner uses load predictors to forecast future request patterns and proactively scale resources. Choose from four prediction models based on your workload's traffic patterns.

Constant — assumes the next interval's load equals the current load; use for stable workloads with long adjustment intervals.

planner:
  loadPredictor: constant
Enter fullscreen mode Exit fullscreen mode

ARIMA — uses auto-ARIMA to fit optimal model parameters for time-series data with trends and seasonality.

planner:
  loadPredictor: arima
  loadPredictorLog1p: true
Enter fullscreen mode Exit fullscreen mode

Kalman — provides fast online forecasting with smooth adaptation using a local linear trend Kalman filter.

planner:
  loadPredictor: kalman
  kalmanQLevel: 0.1
  kalmanQTrend: 0.01
  kalmanR: 1.0
Enter fullscreen mode Exit fullscreen mode

Prophet — handles complex seasonal patterns and trend changes using Facebook's Prophet model.

planner:
  loadPredictor: prophet
  prophetWindowSize: 1000
  loadPredictorLog1p: true
Enter fullscreen mode Exit fullscreen mode

Next Steps

  • Tune plannerMaxGpuBudget and plannerAdjustmentInterval for your real traffic patterns instead of the fast demo values used here.
  • Swap in ARIMA, Kalman, or Prophet load predictors once you have enough traffic history to evaluate forecasting accuracy.
  • Put Gateway API with TLS in front of the frontend and Grafana services before exposing either outside your cluster.
  • Try a different backend (SGLang or TensorRT-LLM) and GPU system combination from the AI Configurator tables to compare profiling results.

For the full guide with additional tips, visit the original article on Vultr Docs.

Top comments (0)