DEV Community: GPUStack

Day 0 Deployment of GLM-5.2-FP8-DSpark on GPUStack

GPUStack — Wed, 22 Jul 2026 10:28:37 +0000

This article is curated from a real-world deployment experience shared by a GPUStack community member. GLM-5.2-FP8-DSpark is an enhanced version of GLM-5.2-FP8 that applies Speculative Decoding by loading an external draft model (Speculator) provided by Red Hat AI. It keeps the original model weights unchanged while adding an additional draft model to improve decoding throughput. On the day the model was released (Day 0), a community member deployed it on an 8× H20-141G environment through GPUStack and conducted a performance comparison against the original GLM-5.2-FP8 under the same hardware conditions.

Let's start with the conclusion. The results from this test are mixed and should be interpreted objectively:

Single-concurrency scenario: In the GPUStack Playground, DSpark achieved approximately 171 TPS, while the original GLM-5.2-FP8 achieved approximately 72 TPS, delivering around a 2.2× improvement.
High-concurrency scenario (64K input / 3K output): DSpark achieved 59.8 TPS output throughput, which was actually lower than the 99.2 TPS achieved by the original MTP approach. Due to the lower acceptance rate of the external draft model on the Random dataset, DSpark did not improve performance under high concurrency and even introduced regression.
Key takeaway: With the specific external draft model + GLM-5.2-FP8 combination tested in this article, DSpark is currently better suited for single-concurrency / low-concurrency interactive scenarios where first-token latency and response throughput are critical, or for research and evaluation purposes. For high-concurrency production workloads, waiting for draft models with higher acceptance rates is recommended. (The conclusions above only apply to the draft model and dataset used in this test, and do not represent the upper limit of the DSpark speculative decoding approach.)

The following sections provide the complete deployment process and benchmark results.

1. Deploying GLM-5.2-FP8-DSpark on GPUStack

DSpark requires a newer vLLM image (v0.25.0) to support speculative decoding with an external draft model. With GPUStack's pluggable vLLM backend architecture, we only need to add a new vLLM backend version. The entire process can be completed through the GPUStack Web UI with just a few clicks.

① Add a New vLLM Backend Version

From the left navigation menu, go to Inference Backend, find the vLLM card, and click Edit.

Under Version Configuration, click Add Version and create a new version named 25. Enter the following image:

vllm/vllm-openai:v0.25.0、

Select CUDA as the framework and save the configuration. A new backend version named 25-custom will then become available.

② Create a Deployment and Select Model & Backend

Go back to the Deployments page and click Deploy Model in the upper-right corner. Configure the basic information as follows:

Source: Select ModelScope
Repository ID: ZhipuAI/GLM-5.2-FP8
Backend: Select vLLM
Backend Version: Select the newly added 25-custom

③ Configure Backend Parameters

In the Advanced section, configure the backend parameters as follows (using an 8× H20-141G environment as an example):

bash --max-model-len 131072 --tensor-parallel-size 8 --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 --enable-expert-parallel --trust-remote-code --gpu-memory-utilization=0.9 --max-num-batched-tokens 32768 --max-num-seqs 16 --speculative-config '{"model":"/var/lib/gpustack/cache/model_scope/RedHatAI/GLM-5.2-speculator.dspark","num_speculative_tokens":7,"method":"dspark"}'

The key parameter is --speculative-config:

Set method to dspark
Set model to the downloaded external draft model path

If the engine takes longer to load, add the environment variable:

bash VLLM_ENGINE_READY_TIMEOUT_S=3600

to extend the ready timeout.
https://

Draft model download:
https://modelscope.cn/models/RedHatAI/GLM-5.2-speculator.dspark

Submit the deployment. Once the instance status changes to Running, the deployment is complete.

2. Performance Benchmark: DSpark vs Original GLM-5.2-FP8

Under the same hardware environment (8× H20-141G), we compared the two models from two perspectives: single-concurrency interactive inference and high-concurrency batch inference.

Scenario 1: Single-Concurrency Interactive Inference

Run a conversation directly in the GPUStack Playground and monitor the real-time output throughput.

GLM-5.2-FP8-DSpark:
Output throughput reached 171.6 Tokens/s, remaining stable in the 170+ range.

Original GLM-5.2-FP8:
Output throughput reached 71.81 Tokens/s.

Under single-concurrency workloads, DSpark achieved approximately 171 TPS, around 2.2× faster than the original version (~72 TPS). This demonstrates the typical benefit of speculative decoding in low-concurrency interactive scenarios.

Scenario 2: High-Concurrency Batch Inference (64K Input / 3K Output, 10 Requests)

Using GPUStack's built-in benchmark tool, we tested the model with a Random dataset containing 64K input tokens / 3K output tokens and 10 requests (with an actual average concurrency of approximately 6).

The baseline for comparison is the original MTP approach with speculative decoding enabled.

Original MTP (method: mtp, num_speculative_tokens: 5):

Total throughput: 2216.39 Tokens/s
Output throughput: 99.22 Tokens/s
TPOT: 63.92 ms

DSpark (method: dspark, num_speculative_tokens: 7):

Total throughput: 1336.12 Tokens/s
Output throughput: 59.82 Tokens/s
TPOT: 103.08 ms

Under high-concurrency workloads, DSpark's output throughput (59.8 TPS) did not improve and instead regressed, performing significantly below the original MTP approach (99.2 TPS). TPOT was also higher.

The main reason is that the external draft model provided by Red Hat AI achieved a relatively low acceptance rate on the Random dataset. When speculative decoding has insufficient acceptance, the additional draft computation overhead can negatively impact overall throughput.

Performance Summary

Scenario	Metric	Original GLM-5.2-FP8	GLM-5.2-FP8-DSpark	Comparison
Single-concurrency (Playground)	Output TPS	≈ 71.8	≈ 171.6	≈ 2.2×
High-concurrency 64K/3K	Total Throughput (TPS)	2216.39 (MTP)	1336.12	≈ 0.60×
High-concurrency 64K/3K	Output Throughput (TPS)	99.22 (MTP)	59.82	≈ 0.60×
High-concurrency 64K/3K	TPOT (ms)	63.92 (MTP)	103.08	Higher

Summary

The performance characteristics of GLM-5.2-FP8-DSpark show a clear trade-off:

Single-concurrency / low-concurrency scenarios:

Speculative decoding provides significant benefits. In this test, DSpark achieved 1.2–2.2× the throughput of the original GLM-5.2-FP8, making it suitable for interactive workloads that are sensitive to single-request throughput and response latency.
High-concurrency scenarios:

With the external draft model used in this test, the acceptance rate on the Random dataset was relatively low. As a result, throughput did not improve significantly and was even lower than the original MTP approach. It is currently not recommended for similar high-concurrency production workloads.
Recommendation:

Based on the model and workload tested in this article, DSpark is currently more suitable for research validation and low-concurrency scenarios. For high-concurrency production deployment, it is recommended to wait for draft model versions with higher acceptance rates on these workloads.

(The conclusions above only apply to the draft model and dataset used in this test, and do not represent the upper limit of the DSpark speculative decoding approach.)

Thanks to GPUStack's pluggable vLLM backend architecture, adding a new image version, integrating an external draft model, or switching between different inference configurations can all be completed through the Web UI with just a few clicks. This enables teams to evaluate new models from Day 0 with minimal adaptation effort.

Test Environment: 8× H20-141G ｜ GPUStack v2 ｜ vLLM v0.25.0 (25-custom image)

Thanks to the GPUStack community member for sharing this real-world deployment experience.

Where Did Your GPU Resources Go? GPUStack Usage Tracking Gives You the Answer at a Glance

GPUStack — Wed, 08 Jul 2026 02:17:30 +0000

You bought a stack of GPUs and deployed a bunch of models. Then, at the end of the month, your boss asks: “Are these GPUs actually worth the cost?” Can you answer?

If not, this article is for you.

GPUStack’s new Usage feature is now available. It provides clear visibility into three key types of resource consumption: token usage, GPU/CPU instance runtime, and storage usage.

Who is using the resources, how much they are using, and which models the usage is associated with — you no longer have to guess.

1. Why Do You Need It?

In a shared GPU cluster with multiple users and multiple models, the biggest pain point is not always “not enough compute.” More often, it is: “Where did the compute go?”

You want to calculate costs, but cannot get usage details broken down by user.
Platform admins want to know which model consumes the most resources, but have to dig through logs.
Business teams ask for more GPU capacity, but no one can clearly explain the actual utilization of existing resources.

GPUStack Usage is built to answer exactly these questions. It connects the full workflow from collection → aggregation → visualization, making every unit of compute consumption traceable.

2. Five Tabs Covering All Usage Scenarios

Click the bar chart icon in the left navigation bar to enter the Usage page. You will see five tabs:

Tab	What You Can See
Summary	A complete overview of token, compute, and storage usage
Tokens	LLM token usage, broken down by model / user / API key
GPU Instances	GPU/CPU instance runtime, broken down by instance type / instance / user
Storage	Storage usage, broken down by storage volume / user
Resource Events	Resource lifecycle audit logs that explain where each number comes from

What exactly is being measured?

Resource	When It Is Measured	Granularity
Token	When an inference request is served through the gateway	Daily
GPU/CPU Instance	When an instance is in the “running (billable)” state; usage stops accumulating after the instance is stopped or deleted	Hourly
Storage	Measured from creation to deletion, regardless of whether it is mounted or used	Hourly

3. Scenario 1: View Usage by User, Clearly See Who Used What

This is one of the most important capabilities in a shared cluster. GPUStack has two built-in roles: Admin and User.

Admin: Can view usage across all users and drill down into any member through the “filter by user” control.
User: Can only view their own usage details and cannot see other users’ data.

In other words, platform admins can easily answer questions such as: “How many tokens did this user consume this month?” or “Who used the most GPU hours?” For cost allocation and quota planning, the data is already there.

💡 The detailed tables in each tab support grouping by user. This is only visible to admins, making each user’s usage bill clear and easy to review.

For regular users, the Usage page shows a dedicated usage bill of their own: how many tokens they used this month, how many GPU hours they ran, and how much storage they occupied. Everything is clear at a glance. This helps users stay aware of their usage, compare it against their quota, and avoid overuse.

They cannot see anyone else’s data, so the permission boundary remains clear and the experience is more secure.

4. Scenario 2: Break Down Usage by API Key to See Which Application Is Driving Costs

Viewing usage by user gives you a people-oriented perspective. Viewing usage by API Key gives you an application-oriented perspective — and in many cases, applications are the real source of consumption.

In practice, each connected application or business pipeline usually uses a dedicated API Key. The Tokens tab supports grouping by API Key, so you can directly see:

Which application / integration partner is continuously consuming tokens
Whether the cost of a business line has suddenly increased
Which keys are barely used and can be reclaimed

By combining the dimensions of users and applications, cost attribution becomes much more complete.

5. Scenario 3: Clearly Track Usage Across Different Types of Models

Running large language models, embedding models, and open-source models of different sizes in the same cluster? No problem.

At the top of the Tokens tab, GPUStack shows the key metrics directly: input / output / total tokens, API requests, and number of models used. The detailed table below can be grouped by model, showing:

Input tokens, including cached input tokens, which are marked separately
Output tokens
Total tokens
API requests
Last active time

This lets you quickly identify which models are receiving the most traffic, which models were requested but are barely used, and whether a model should be taken offline or scaled up. The data helps you make the decision.

📌 Easily verify the effect of cache optimization: The Input Tokens Cached field separately marks the portion of input tokens that hit the prompt cache. After optimizing prompt caching, you can check the proportion of this number to directly quantify how many input tokens were saved. Whether the optimization worked is clear at a glance.

Also: Even if a model has been deleted, its historical usage is still retained in the table and marked as Deleted. It is still included in the total usage, so historical records are not lost.

6. Scenario 4: GPU Compute Usage Is No Longer a Black Box

The GPU Instances tab breaks down instance runtime clearly. The key is that it distinguishes between two metrics that are often confused:

Metric	Meaning
Instance Hours	The wall-clock runtime of an instance, regardless of how many GPUs it uses
GPU Hours	Accelerator runtime = runtime × number of GPUs

For example, a 2-GPU instance running for 1 hour equals 2 GPU hours, but only 1 instance hour.

This distinction lets you calculate actual GPU usage instead of being misled by the number of instances. The trend chart can also be grouped by instance type / instance / user, helping you understand the current situation before scaling capacity.

⚙️ CPU instances do not have accelerator cards, so they are counted only in Instance Hours, not GPU Hours.

7. Scenario 5: Storage Usage, Including Idle Volumes

The Storage tab focuses on storage capacity usage over time. The key metrics are GB-Days and GB-Hours: capacity × duration. Usage can be broken down by storage volume / user.

The important detail is this: storage is measured from creation to deletion, regardless of whether it is mounted. This means there are no blind spots in the usage records. Even if a volume is not mounted or accessed, as long as it still exists, its usage is counted in GB-Days. It will not disappear from the bill simply because it is “not being used.”

This makes forgotten idle volumes easy to find. Sort by GB-Days, identify the volumes occupying the most capacity, check whether they are still used by any instance, and clean up what should be removed.

8. Scenario 6: Find Zombie Resources and Stop Paying for Waste

After a cluster has been running for a while, it often accumulates resources that no one remembers: models that are no longer called, or instances that were never stopped. They no longer create value, but they still occupy resources.

For resources that are measured only when used, such as models and GPU/CPU instances, each detailed table includes a Last Active column. It records the last time the resource generated usage. Once the resource is no longer used, usage stops accumulating and Last Active stays at that moment. Long-inactive models and forgotten stopped instances immediately become visible, making cleanup much more targeted.

9. Scenario 7: View Trends and Plan Capacity Without Guesswork

Both the Summary page and the individual tabs include trend charts. With adjustable granularity by hour / day / week / month, you can see how usage changes over time instead of only looking at a static total.

Is usage stable, or is it growing faster?
When did the growth inflection point occur, and does it match a business launch?
Based on the current trend, how long can existing capacity last?

Once the trend is clear, capacity planning moves from guesswork to data-driven decisions. Trend charts can also be split by group, making structural changes easy to spot.

10. Scenario 8: Every Number Has a Source, and Resource Events Show the Full Lifecycle

Each tab above shows the result: how many GPU hours were used, how many GB-Days were consumed, and so on. The Resource Events tab shows where those numbers come from. The lifecycle of every instance and storage volume is recorded as events.

Sorted in reverse chronological order, it becomes a resource timeline. The lifecycle of an instance is easy to follow, with colored labels showing each state clearly:

🟢 Created → 🔵 Started and measured → 🟠 Stopped and no longer measured → 🔴 Deleted

This means:

Trace usage for reconciliation: Why does an instance have this number of Instance Hours? Check the period from Started to Stopped, and you can see exactly how long it ran.
Filter and investigate: Filter by date / resource type / event type / name to quickly answer questions such as “which volumes were deleted last week?” or “which instance was repeatedly started and stopped?”
Explain inactivity: If Last Active in Scenario 6 shows that an instance has stopped generating usage, come here to check when it was Stopped or Deleted.

11. Put the Data to Work: Export Usage for Reconciliation

Seeing the data is not enough. You also need to take it out and use it.

The Tokens, GPU Instances, and Storage tabs all support export through the download icon. You can preview the currently filtered details and then download them. Monthly billing, financial reconciliation, internal reports — once you have the data, you can work with it directly, without asking engineers to write custom scripts.

12. Details That Make Usage Tracking Reliable

Unified Time Zone

All timestamps across tabs, including trend buckets, Last Active, and event time, are displayed using the same rollup time zone. This ensures that calendar boundaries are aligned across all tabs. It can be configured through the GPUSTACK_USAGE_ROLLUP_TIMEZONE environment variable and follows the server’s local time zone by default.

Flexible Common Controls

Date range, defaulting to the last 30 days, filtering by user / model / API Key, refresh, metric switching, grouping, and granularity adjustment are all available out of the box.

Data Retention and Archiving

Aggregated token, compute, and storage usage data is retained for about 13 months by default, and then archived by background tasks. The retention window and archive schedule can be configured through the GPUSTACK_*_RETENTION_MONTHS and GPUSTACK_*_ARCHIVE_CRON environment variables.

Final Thoughts

Compute is expensive, but compute you cannot see clearly is even more expensive.

GPUStack Usage makes every token, every GPU hour, and every GB of storage measurable, traceable, and attributable. Whether you are in finance handling cost allocation, an administrator planning capacity, or a business owner trying to prove ROI, it gives you answers backed by data.

Upgrade to the latest version of GPUStack, open the Usage page, and see where your compute resources are really going.

Day 0 Benchmark: Deploying DeepSeek-V4-Flash-DSpark on GPUStack Doubles Throughput

GPUStack — Wed, 01 Jul 2026 14:05:03 +0000

This article is based on a community benchmark contributed by a GPUStack user. DeepSeek-V4-Flash-DSpark enhances DeepSeek-V4-Flash by adding a Speculative Decoding module. Using the same model weights with an additional speculative decoder, it significantly improves both inference throughput and Time to First Token (TTFT).

On Day 0 of the model release, a GPUStack community member deployed and benchmarked DeepSeek-V4-Flash-DSpark on an 8× H20-141G setup, comparing it against the original DeepSeek-V4-Flash (DSV4F) under identical deployment settings. Here are the key results:

Single-stream throughput: In the 1K input / 1K output workload, DSpark achieved 195 tokens/s, approximately 2× the throughput of the original model (96 tokens/s)；
Overall throughput: In the 64K input / 3K output workload with 10 concurrent requests, DSpark reached 338 tokens/s, approximately 1.7× higher than the original (198 tokens/s)；
Time to First Token (TTFT): Reduced to roughly half that of the original model.

The following sections walk through the complete deployment process and benchmark results.

1. Deploying DSpark on GPUStack

GPUStack comes with the SGLang inference backend built in. To deploy DSpark, simply add a container image that supports the model. The entire process can be completed in just a few clicks through the web UI.

① Go to Inference Backends and edit SGLang.

From the left navigation menu, go to Inference Backends, locate the SGLang card, then click the menu in the upper-right corner and select Edit.

② Add a DSpark-compatible image

Under Version Configuration, click Add Version to create a new version named dspark, then specify the following container image:

swr.cn-north-4.myhuaweicloud.com/desaysv/gpustack/sglang-dspark:v1.0

Select CUDA as the framework. Set the image entrypoint override to sglang serve, and set the command to: --model-path {{model_path}} --host {{worker_ip}} --port {{port}}。

③ Create a Deployment

Return to the Deployments page and click Deploy Model in the upper-right corner. For Source, select ModelScope.

④ Select the Model and Inference Backend

Search for and select deepseek-ai/DeepSeek-V4-Flash-DSpark.
Set the Inference Backend to SGLang.
Select dspark-custom as the backend version.

⑤ Configure Backend Parameters

Under Advanced, configure the backend parameters as follows (using an 8× H20-141G setup as an example):

--context-length 1000000
--trust-remote-code
--tp-size 8
--ep-size 8
--moe-runner-backend flashinfer_mxfp4
--speculative-moe-runner-backend flashinfer_mxfp4
--speculative-algorithm DSPARK
--speculative-eagle-topk 1
--speculative-num-steps 1
--mem-fraction-static 0.85
--cuda-graph-max-bs 32
--max-running-requests 32
--disable-overlap-schedule

⑥ Configure Environment Variables

Add the following environment variable to ensure the required dependencies are installed correctly:

Key	Value
`PYPI_PACKAGES_INSTALL`	`-U distro -i https://mirrors.aliyun.com/pypi/simple/`

⑦ Start the Deployment and Monitor the Logs

Once submitted, GPUStack will start the deployment. In the logs, you should see CUDA Graph capture, Application startup complete, and Uvicorn listening on the inference port, indicating that the model has started successfully:

⑧ Verify the Deployment Status

Once the instance reaches the Running state, the deployment is complete.

⑨ Verify the Deployment

Open the Playground and send a few prompts to the model. The real-time throughput indicator in the lower-right corner should show an output rate of 185.94 tokens/s, with single-stream throughput remaining stable at around 200 tokens/s.

⑩ View the Inference Service Port

If you want to run benchmarks directly against the service, open the instance details to find the inference service IP address and port (in this example, 10.91.3.213:40048).

2. Benchmark Results: DSpark vs. DSV4F

Under identical hardware and deployment settings, we benchmarked the original DeepSeek-V4-Flash (with MTP enabled) against DSpark across two workloads. All benchmarks were conducted using SGLang's built-in bench_serving tool.

Scenario 1: 1K Input / 1K Output (Single Request)

HF_ENDPOINT=https://hf-mirror.com python3 -m sglang.bench_serving \
    --backend sglang --port 40048 \
    --dataset-name random --random-input-len 1024 --random-output-len 1024 \
    --random-range-ratio 1.0 --num-prompts 1 \
    --max-concurrency 1 --request-rate inf --host <Inference Server IP Address>

Original DSV4F: Output throughput: 96.20 tokens/s, TTFT: 300.45 ms, Accept length: 2.71

DSpark（DSV4FD）： Output throughput 195.18 tok/s，TTFT 129.34 ms，Accept length 4.42

Single-stream throughput reached 195 tokens/s, approximately 2× that of the original DSV4F (96 tokens/s), while Time to First Token (TTFT) was reduced to about half.

Scenario 2: 64K Input / 3K Output (10 Concurrent Requests)

HF_ENDPOINT=https://hf-mirror.com python3 -m sglang.bench_serving \
    --backend sglang --port 40048 \
    --dataset-name random --random-input-len 64000 --random-output-len 3000 \
    --random-range-ratio 1.0 --num-prompts 10 \
    --max-concurrency 1 --request-rate inf --host <Inference Server IP Address>

Original DSV4F (MTP enabled): Output throughput: 198.60 tokens/s, Speculative acceptance rate: 20.91%, Acceptance length: 1.21

DSpark（DSV4FD）： Output throughput 338.17 tok/s，Accept length 4.90

In the long-context workload, DSpark achieved 338 tokens/s, approximately 1.7× the throughput of the original DSV4F (198 tokens/s), nearly doubling overall throughput.

Summary

Workload	Metric	Original DSV4F	DSpark (DSV4FD)	Improvement
1K Input / 1K Output	Output Throughput (tokens/s)	96.20	195.18	≈ 2.0×
1K Input / 1K Output	TTFT (ms)	300.45	129.34	≈ 0.43×
1K Input / 1K Output	Acceptance Length	2.71	4.42	—
64K Input / 3K Output	Output Throughput (tokens/s)	198.60	338.17	≈ 1.7×
64K Input / 3K Output	Acceptance Length	1.21	4.90	—

Conclusion

Across both single-request and long-context workloads, DeepSeek-V4-Flash-DSpark consistently delivered around 2× the throughput of the original DSV4F, while reducing Time to First Token (TTFT) to roughly half. Achieving these gains requires nothing more than switching to the DSpark model weights and container image with the integrated speculative decoding module.

With GPUStack's built-in SGLang inference backend, the entire deployment can be completed in just a few clicks through the web UI, making the model production-ready on Day 0.

Benchmark environment: 8× H20-141G | GPUStack v2 | SGLang 0.5.14 (using the dspark-custom image based on a patched SGLang 0.5.14)

Acknowledgments: Thanks to the GPUStack community member for sharing the benchmark results.

GPUStack v2.2: From Model Serving to Token Operations, from Compute Pooling to GPU-as-a-Service

GPUStack — Tue, 30 Jun 2026 02:34:35 +0000

Deploying a model and bringing it online is only the starting point of AI service delivery.

As large language model applications move into scaled production, AI infrastructure is entering an inevitable phase of maturity — from simply being able to run to becoming operations-ready.

This shift is not just about adding more features. It reflects an evolution in platform positioning: from reliably serving inference workloads to becoming the infrastructure foundation that can truly support enterprise AI service delivery.

At this stage, the core challenge lies in advancing two areas in parallel: model serving must deliver operations-grade reliability and visibility, while compute management must expand from “serving inference workloads” to “unified allocation of the diverse resources required by AI.”

GPUStack v2.2 continues to move deeper in both directions: model serving is evolving from available to operations-ready, while compute management is extending from unified scheduling to on-demand services.

Deepening Support for Model Inference Scenarios and Lifecycle Management

The stability of model serving is not only a deployment-stage concern. After an instance starts running, issues such as OOM errors, hanging inference requests, and silent process crashes are often the more common problems in production environments.

Previously, GPUStack’s health checks mainly covered the startup phase. Once an instance started successfully, the platform had no way to detect issues that occurred later. Faulty instances could remain in the service pool and continue receiving traffic, resulting in silent failures that would only be addressed after someone noticed and handled them manually.

In v2.2, health probing is extended across the entire runtime lifecycle. The platform continuously checks the actual inference capability of each instance. When an abnormal instance is detected, it is immediately removed from the service pool and automatically restarted. Once recovered, it is automatically added back. Service availability is now proactively maintained by the platform, rather than relying on manual inspections or user reports.

Troubleshooting capabilities have also been systematically enhanced. In production environments, what teams often need most is a complete record of what happened at the time of failure. Previously, this meant logging into the terminal and manually checking logs. v2.2 introduces three types of log access:

Historical logs before restart, allowing you to view the complete output before an instance crashed, instead of losing access to past failure logs after a restart;

Distributed sub-instance logs, allowing you to inspect the output of each node separately in multi-node deployments and quickly identify where the issue occurred;

Ray container logs, allowing you to view Ray container logs directly in the UI without troubleshooting through terminal commands.

Most production troubleshooting workflows can now be completed end to end within the GPUStack UI.

For distributed inference, v2.2 introduces vLLM MP auto-distributed mode.

Previously, GPUStack only supported Ray-based auto-distributed vLLM deployments. MP-based distributed deployments had to be configured manually, and GPUStack could not automatically spin up all distributed instances.

With the rapid evolution of vLLM, the new MP-based distributed mode offers clear advantages over Ray-based auto-distributed deployments in terms of operational overhead and inference performance.

Users can now choose the vLLM auto-distributed deployment strategy that best fits their needs.

Another notable update is support for Multi-LoRA.

In enterprise environments, fine-tuning models for different business scenarios is a common requirement. Previously, each LoRA Adapter had to run as a separate model instance, causing GPU memory overhead to grow linearly with the number of tasks and leading to significant resource waste.

With v2.2, multiple LoRA Adapters can be mounted to the same base model instance and switched dynamically. This allows the same hardware to support more fine-tuned tasks while significantly improving GPU memory utilization.

Token Usage Governance for Model Services

Models may be running, but where exactly are tokens being consumed? This question may not be obvious at an early stage with limited usage. But once multiple teams and applications start sharing the same platform, it quickly becomes an operational pain point.

GPUStack previously supported usage statistics by model and by user, helping teams understand overall consumption trends.

However, these two dimensions were not sufficient for precise attribution.

When different applications and business lines share multiple keys under the same user account, usage cannot be clearly separated, making cost accounting difficult to perform.

v2.2 introduces usage statistics at the API Key level. Token consumption for each key is metered independently, allowing administrators to clearly see which caller is consuming what, and how much. This provides a direct basis for cross-team cost attribution and quota management.

Another important change is giving visibility back to users. Previously, users who wanted to understand their own consumption had to ask administrators to pull the data for them.

v2.2 introduces self-service personal usage queries on the user side. Consumption history by model and by time range can now be viewed directly in the UI, without going through a request process.

Once metering capabilities are in place, token consumption is no longer a black box. It becomes operational data that can support quota allocation, internal chargeback, and cost analysis.

Enhanced Production Deployment Capabilities

Platform capabilities can only be fully realized when supported by a solid deployment experience. In v2.2, GPUStack addresses key gaps in enterprise production deployment across three areas.

Kubernetes has become a mainstream choice for enterprise infrastructure. However, deploying GPUStack in K8s environments previously lacked a standardized cloud-native path.

v2.2 provides an official Helm Chart, enabling installation and configuration through Helm in a single streamlined process. This allows GPUStack to fit directly into existing GitOps workflows and CI/CD systems, significantly reducing the operational cost of deployment and upgrades.

v2.2 also expands database compatibility with support for OceanBase and openGauss, giving teams more flexibility in enterprise deployment environments.

On the network topology side, v2.2 supports a Worker-to-Server one-way access mode.

In cross-region or cross-network-boundary deployment scenarios, many environments make it difficult to establish bidirectional connectivity between Server and Worker.

With one-way networking, Worker nodes only need to access the Server, while the Server does not need to initiate reverse connections. This removes a key networking barrier for unified management of multi-region clusters.

From Compute Pooling to GPU Services

Unified scheduling of heterogeneous compute resources has always been one of GPUStack’s core capabilities — bringing GPUs from different vendors and with different specifications into a single compute pool for unified scheduling and monitoring. Previously, however, this compute pool primarily served one scenario: inference.

Data scientists often need an interactive development environment, while algorithm engineers may need dedicated GPUs for experimentation and debugging. In many teams, these needs were handled outside GPUStack through separate systems, resulting in fragmented resource allocation and metering.

v2.2 introduces the GPU Instance Service, bringing the process of “allocating an isolated GPU environment” under unified platform management.

Users can request isolated GPU instances on demand, specify the GPU vendor, model, and quantity, and select runtime templates that include storage mounts and port configurations. Once the instance is ready, they can access it through SSH or the web.

Usage is metered centrally by the platform and shares the same scheduling and usage system as inference services.

This marks an expansion of GPUStack’s compute service model: the same compute resource pool can serve as the foundation for inference services, while also being exposed as on-demand GPU instances.

With unified scheduling and unified metering, management no longer becomes fragmented simply because resources are used in different scenarios. Future capabilities such as resource scheduling for training and fine-tuning and more fine-grained virtualized compute partitioning can naturally build on this foundation.

The Next Stage of AI Infrastructure Platforms

The evolution of AI infrastructure follows a clear path: from being able to run models, to operating model services reliably, and then to unified management and allocation of the compute resources required by AI.

v2.2 takes a step forward in both dimensions. On the model serving side, deeper inference scenario support, improved instance lifecycle management, a token usage governance system, and enhanced production deployment capabilities provide the foundation for scaled service delivery.

On the compute side, the introduction of the GPU Instance Service expands resource allocation and metering beyond inference scenarios.

Both directions point toward the same goal: making GPUStack an infrastructure foundation that enterprises can truly rely on for AI service delivery.

Get started:

GitHub: https://github.com/gpustack/gpustack

Documentation: https://docs.gpustack.ai

GPUStack v2.2 Enterprise Edition: Coming Soon

The v2.2 open-source release lays the foundation for platform-level operational capabilities.

In more complex enterprise environments, there is another layer of capabilities that the open-source edition is not designed to fully cover. When a platform needs to serve multiple isolated tenants, manage compute consumption down to each API Key and model route, ensure end-to-end high availability, and support Token/GPU billing management, these are the challenges that GPUStack Enterprise Edition is built to address.

GPUStack v2.2 Enterprise Edition is designed for enterprise scenarios that require organization-level governance and commercial operations. It will support multi-tenant isolation, fine-grained quota, rate limiting, and access control, production-grade high availability, resource topology visualization, and billing management. Stay tuned.

If you are interested in the Enterprise Edition, feel free to contact us to learn more and explore early access opportunities.

Contact us: https://gpustack.ai/contact/

GPUStack MaxKB: Build a Powerful and Easy-to-Use Open-Source Enterprise AI Agent Platform

GPUStack — Tue, 10 Mar 2026 02:54:50 +0000

GPUStack × MaxKB: Build a Powerful and Easy-to-Use Open-Source Enterprise AI Agent Platform

Summary: By leveraging GPUStack for efficient model deployment and management, and connecting those models to MaxKB, you can easily build an AI assistant with knowledge base retrieval + intelligent Q&A capabilities.

As AI applications become increasingly common within organizations, more teams are beginning to focus on two core challenges:

How to efficiently manage and deploy local large models
How to quickly build enterprise knowledge bases and AI Agents

If you are looking for solutions to both problems, the combination of GPUStack + MaxKB is well worth exploring.

GPUStack: Focuses on GPU resource management and model deployment, supporting multi-node clusters and multi-model services.
MaxKB: An open-source enterprise knowledge base and AI application platform that enables rapid development of knowledge-based Q&A systems and AI Agents.

By connecting GPUStack-provided model services to MaxKB, you can easily build a practical enterprise AI knowledge assistant.

This article will walk through the entire process from scratch.

📌 What You'll Learn

Deploy the latest GPUStack v2.1.0
Deploy required models in GPUStack
Obtain GPUStack model connection information
Deploy MaxKB
Connect GPUStack models in MaxKB
Practical example: Build a GPUStack documentation knowledge base

Install GPUStack v2.1.0

1. Install GPUStack Server

sudo docker run -d --name gpustack-server \
  --restart unless-stopped \
  -p 80:80 \
  -v gpustack-data:/var/lib/gpustack \
  -v /data/gpustack_cache:/var/lib/gpustack/cache \
  gpustack/gpustack:v2.1.0 \
  --bootstrap-password "123" \
  --debug

After running the command above, open your browser and visit:

http://your_host_ip

You will enter the GPUStack UI.

Default login credentials:

admin / 123

2. Create a Cluster

GPUStack manages worker nodes in units called Clusters.

When deploying GPUStack Server for the first time, you will be prompted to create your first cluster. Click:

Create Your First Cluster

Follow the UI instructions to complete the setup.

You can also go to the Clusters page from the sidebar and click Add Cluster to create one manually.

3. Add a Worker

After creating a cluster, the system will prompt you to Add Worker.

Follow the instructions in the UI.

You can also add one manually via the Workers page in the sidebar.

Run the diagnostic command provided in the guide interface.

If the drivers and container runtime are correctly installed, you will see two OK messages.

If not configured appears, follow the provided links to check dependency documentation and install the missing components according to your environment.

Model Cache Volume Mount: Mount this directory to the model cache directory /var/lib/gpustack/cache.

GPUStack Data Volume: Mount this directory to the data directory /var/lib/gpustack.

Then run the Worker startup command:

sudo docker run -d --name gpustack-worker \
   -e "GPUSTACK_RUNTIME_DEPLOY_MIRRORED_NAME=gpustack-worker" \
   -e "GPUSTACK_TOKEN=gpustack_7b42996d3f5571d5_8181f986537c100369eaa2dfcf6d6359" \
   --restart=unless-stopped \
   --privileged \
   --network=host \
   --volume /var/run/docker.sock:/var/run/docker.sock \
   --volume gpustack-worker-data:/var/lib/gpustack \
   --volume /data/gpustack_cache:/var/lib/gpustack/cache \
   --runtime nvidia \
   gpustack/gpustack:v2.1.0 \
   --server-url http://192.168.50.14 \
   --worker-ip 192.168.50.14

Deploy Models in GPUStack

Click Deployments in the sidebar to open the model deployment page.

If no models are currently deployed, you will see a Deploy Now button in the center of the page.

Click it to enter the Model Catalog, select the desired model, and follow the prompts to deploy it.

Additional deployment methods are available under the Deploy Model menu in the top-right corner.

For this tutorial, we deploy the following three models:

Qwen3-Reranker-4B
Qwen3-Embedding-4B
Qwen3.5-35B-A3B

GPU memory allocation can be adjusted according to your environment.

Deploy Qwen3-Reranker-4B

After deployment, you can test it in the Playground.

Deploy Qwen3-Embedding-4B

After deployment, test it in the Playground.

Deploy Qwen3.5-35B-A3B

Here we additionally set the PYPI_PACKAGES_INSTALL environment variable to upgrade the transformers library.

After deployment, test it in the Playground.

Obtain GPUStack Model Access Information

Open the Routes page from the sidebar.

Click the three-dot menu next to the Route, then select:

API Access Info

Record the following information:

Base URL
Model Name
API Key

Example:

Base URL: http://192.168.50.14/v1

Model Name:
qwen3.5-35b-a3b
qwen3-reranker-4b
qwen3-embedding-4b

API Key:
gpustack_xxxxxxxxxxxxxxxxx

You can create an API Key following the instructions in the UI.

Deploy MaxKB

MaxKB supports one-command Docker deployment:

docker run -d --name=maxkb --restart=always -p 8080:8080 -v ~/.maxkb:/opt/maxkb 1panel/maxkb

Default credentials:

admin / MaxKB@123..

Upon first login, you will be prompted to change the password.

Follow the instructions to update it.

Connect GPUStack Models in MaxKB

In the top navigation bar of MaxKB, select Model.

Click Add Model in the upper-right corner.

Note
API URL and API Key will only appear after entering the Base Model and pressing Enter.

Add the following models in the same way:

qwen3-reranker-4b
qwen3-embedding-4b

For qwen3-reranker-4b, you must enable Generic Proxy:

This is because MaxKB uses the endpoint:

/v2/rerank

After configuration, it should look like this:

Practical Example: Build a GPUStack Documentation Knowledge Base

Open the Knowledge page at the top and click Create to create a knowledge base.

Select Web Knowledge.

Enter the GPUStack documentation URL.

MaxKB will automatically crawl and parse the page content.

After crawling is complete:

Create an AI Agent

Go to the Agent page.

Click Create to create a new Agent.

After completing the configuration, click Publish.

Once published successfully, you can start chatting with the agent.

Chat Demo

Open the chat interface:

Example result:

🙌 Join the GPUStack Community

If you have already started using GPUStack,
or are exploring local large models / GPU resource management / AI infrastructure,
you are welcome to join our community group to exchange practical experience, pitfalls, and best practices.

https://discord.gg/QAzGncGs

No More Token Anxiety: Build an “Unlimited-Use” Local AI Assistant with GPUStack + OpenClaw

GPUStack — Fri, 06 Mar 2026 02:32:29 +0000

Over the past two years, more and more teams have integrated AI into their daily workflows.
But soon, a practical issue emerged:

The more the model is used, the faster Tokens are consumed, and both costs and psychological pressure rise accordingly.

Many people rely on AI to improve efficiency, while at the same time having to “use it sparingly” and “let it think less.”
In the end, AI instead becomes a carefully budgeted consumable.

If AI can run on your own GPU,
without being billed by Token, available for conversation at any time, and running long-term inside collaboration tools,
then it truly feels like a real “work assistant.”

Based on the local model capabilities provided by GPUStack, combined with OpenClaw (supporting multiple collaboration platforms such as WhatsApp, Telegram, Discord, Slack, Lark, etc.) and Telegram,
this article will walk through step by step how to build a truly usable, sustainably running, and almost Token-worry-free local AI assistant.

📌 What This Article Covers

Deploying a model with GPUStack
Creating a Telegram bot application and configuring permissions
Installing, configuring, and key considerations for OpenClaw
First-time authorization and connectivity testing on the Telegram side
Practical example: Let the assistant star the GPUStack project
Built-in assistant commands
Useful OpenClaw commands and resource links

I. Deploy a Model with GPUStack and Prepare Access Information

Before connecting OpenClaw, we need to complete model deployment in GPUStack and obtain the model service access information.

This section will use Qwen3.5-35B-A3B as an example to demonstrate the complete process from
Custom inference backend → Deploy model → Obtain access information.

1. Environment Preparation and Version Information

GPUStack version: v2.0.3
Custom inference backend image: vllm/vllm-openai:qwen3_5
Model weights: Qwen/Qwen3.5-35B-A3B

⚠️ OpenClaw has requirements for the model context window:
Minimum 16K, recommended 128K or above.

2. Configure Custom Inference Backend (vLLM)

In the GPUStack console, go to:

“Inference Backends” → “Edit vLLM” → “Add Version”

3. Deploy the Qwen3.5-35B-A3B Model

Example parameters:

--tensor-parallel-size=2
--mm-encoder-tp-mode data
--mm-processor-cache-type shm
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

If you encounter:

text Error 803: system has unsupported display driver / cuda driver combination

You can try adding the environment variable:

bash LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/nvidia/lib:/usr/lib/x86_64-linux-gnu

4. Obtain GPUStack Model Access Information

Record the following three items:

API Base URL
Model ID
API Key (create it in GPUStack)

II. Create a Telegram Bot

Open Telegram and search for BotFather
Open the BotFather APP

Create a new Bot and fill in the basic information

Copy the Bot Token

For details, please refer to: https://docs.openclaw.ai/channels/telegram

III. Install and Configure OpenClaw

Demo environment: Ubuntu 24.04

1. One-Click Installation

bash curl -fsSL https://openclaw.ai/install.sh | bash

The script will automatically install dependencies such as Node and Git.

2. Interactive Configuration Wizard

Model/Auth Provider Select Custom Provider (Any OpenAI or Anthropic compatible endpoint)

Enter the GPUStack API Base URL / API Key

Select Telegram for Channel

Paste the Bot Token

IV. First-Time Authorization and Testing

Send a message to the bot in Telegram
On first use, it will prompt for Pairing authorization

On the server, run:

bash openclaw pairing approve telegram <Pairing-Code>

V. Practical Example: Let the Bot Star the GPUStack Project

1. Prepare a GitHub PAT

Use Tokens (classic)
Check the repo permission

2. Write to Environment Variables

bash vim ~/.openclaw/.env

Restart:

bash openclaw gateway restart

3. Send a Command to the Bot

Result:

VI. Common Commands

/new: Start a new session
/status: Check bot status
/reset: Reset context
/model: View / switch model

VII. Useful OpenClaw Commands and Resources

Common CLI Commands

bash openclaw logs --follow openclaw doctor openclaw gateway --help openclaw dashboard openclaw tui

Documentation and Ecosystem

📘 https://docs.openclaw.ai
🌐 https://clawhub.ai

Conclusion: When AI Becomes Infrastructure, Not a Consumable

Looking back, the essence of Token anxiety is not that models are expensive, but that AI is treated as an “external consumable resource.”

When models run in the cloud and capabilities are controlled by others,
we become accustomed to careful budgeting, limiting usage, and controlling call frequency.

But when the model truly runs on your own GPU,
when inference capability, context, and tool calls all become part of your infrastructure,
the role of AI changes accordingly—

It is no longer a paid API call each time,
but a readily available, long-term online, continuously evolving work assistant.

This is exactly the significance of combining GPUStack and OpenClaw:
Let AI return from a “cost item” to “productivity.”

If you already have GPU resources,
you might as well try it yourself and truly integrate AI into your daily workflow.

When you no longer worry about Tokens,
you will truly begin to make good use of AI.

🙌 Join the GPUStack Community

If you have already started using GPUStack,
or are exploring local large models / GPU resource management / AI Infra,
you are welcome to join our community group to exchange practical experience, pitfalls, and best practices together.

https://discord.gg/QAzGncGs

Building Your Private ChatGPT and Knowledge Base with AnythingLLM and GPUStack

GPUStack — Tue, 12 Nov 2024 05:00:48 +0000

AnythingLLM [https://github.com/Mintplex-Labs/anything-llm] is an all-in-one AI application that runs on Mac, Windows, and Linux. Its goal is to enable the local creation of a personal ChatGPT using either commercial or open-source LLMs along with vector database solutions. AnythingLLM goes beyond being a simple chatbot by including Retrieval-Augmented Generation (RAG) and Agent capabilities. These features allow it to perform a variety of tasks, such as fetching website information, generating charts, summarizing documents, and more.

AnythingLLM can integrate various types of documents into different workspaces, enabling users to reference document content during chats. This provides a easy way to organize workspaces for different tasks and documents.

In this article, we will introduce how to build a personal ChatGPT with knowledge base using AnythingLLM + GPUStack.

Run models with GPUStack

GPUStack is an open-source GPU cluster manager for running large language models (LLMs). It enables you to create a unified cluster from GPUs across various platforms, including Apple MacBooks, Windows PCs, and Linux servers. Administrators can deploy LLMs from popular repositories like Hugging Face, allowing developers to access these models as easily as they would access public LLM services from providers such as OpenAI or Microsoft Azure.

Unlike Ollama, GPUStack is a cluster solution designed to aggregate GPU resources from multiple devices to run models.

To deploy the Chat Model and Embedding Model on GPUStack:

• Chat Model: llama3.1

• Embedding Model: bge-m3

And you need to create an API key. This key will be used by AnythingLLM to authenticate when accessing the models API deployed on GPUStack.

Install and configure AnythingLLM

AnythingLLM offers packages for Mac, Windows, and Linux, you can download from https://anythingllm.com/download. After installation, open AnythingLLM to begin the setup process.

Configure LLM Provider

First, configure the chat model. Search for OpenAI, select Generic OpenAI:

And fill in the details for the model deployed on GPUStack:

Save and configure embedding model.

Configure Embedding Provider

AnythingLLM includes a lightweight embedding model, all-MiniLM-L6-v2, which offers limited performance and context length. For more powerful embedding capabilities, you can either opt for public embedding services or run open-source embedding models. Here, we’ll configure the embedding model bge-m3, which is running on GPUStack. Set the embedding provider to Generic OpenAI and fill in the relevant configuration.

Then create a workspace, and we can use AnythingLLM after it's completed.

Use AnythingLLM

Chat with LLM

Select a workspace, create a new thread, and send your question to the LLM:

Fetch website content

Click the upload button next to the workspace, enter the website URL in the Fetch website box, and fetch the website content.

The fetched website content will be sent to the embedding model for vectorization and then stored in the vector database.

Check the content fetched from the website:

Documents embedding

Click the upload button next to the workspace, then click the upload box and upload a document. The document will be sent to the embedding model for vectorization and then stored in the vector database.

Check the content of embedded documents:

For more information, please read the AnythingLLM documentation: https://docs.anythingllm.com/

Conclusion

In this tutorial, we have introduced how to use AnythingLLM + GPUStack to aggregate GPUs across multiple devices and build an all-in-one AI application for RAG and AI Agents.

GPUStack provides a standard OpenAI-compatible API, which can be quickly and smoothly integrated with various LLM ecosystem components. Wanna give it a go? Try to integrate your tools/frameworks/software with GPUStack now and share with us!

If you encounter any issues while integrating GPUStack with third parties, feel free to join GPUStack Discord Community and get support from our engineers.

Building Free GitHub Copilot Alternative with Continue + GPUStack

GPUStack — Fri, 23 Aug 2024 17:00:00 +0000

Click here to read original post

Continue is an open-source alternative to GitHub Copilot, this is an open-source AI coding assistant that allows to connect various large language models(LLMs) within VS Code and JetBrains to build custom code autocompletion and chat capabilities. It supports:

Code parsing
Code autocompletion
Code optimization suggestions
Code refactoring
Code implementations Inquiring
Documentation online searching
Terminal errors parsing

and more. It assists developers in coding and enhancing their development efficiency.

In this tutorial, we are going to use Continue + GPUStack to build a free GitHub Copilot locally, providing developers with an AI-paired programming experience.

Running Models with GPUStack

First, we will deploy the models on GPUStack. There are three model types recommended by Continue:

Chat model: select llama3.1, this is the latest open-source model trained by Meta.
Autocompletion model: select starcoder2:3b, a highly advanced autocompletion model trained by Hugging Face.
Embedding model: select nomic-embed-text, which supports a context length of 8192 tokens, it outperforms OpenAI ada-002 and text-embedding-3-small models for both short and long context tasks.

After deploying the models, you are also required to create an API key in the API Keys section for authentication when Continue accesses the models deployed on GPUStack.

Installing and Configuring Continue

Continue provides extensions for both VS Code and JetBrains. In this article, we will use VS Code as an example. Install Continue from the VS Code extension store:

Once installed, drag the Continue extension to the right panel to avoid conflict with the file explorer:

Then, select the settings button in the bottom-right corner to edit Continue's configuration and connect to the models deployed on GPUStack. Replace the sections for "models", "tabAutocompleteModel", and "embeddingsProvider" with your own GPUStack-generated API Key:

{
  "models": [
    {
      "title": "Llama 3.1",
      "provider": "openai",
      "model": "llama3.1",
      "apiBase": "http://192.168.50.4/v1-openai",
      "apiKey": "gpustack_f58451c1c04d8f14_c7e8fb2213af93062b4e87fa3c319005"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Starcoder 2 3b",
    "provider": "openai",
    "model": "starcoder2",
    "apiBase": "http://192.168.50.4/v1-openai",
    "apiKey": "gpustack_f58451c1c04d8f14_c7e8fb2213af93062b4e87fa3c319005"
  },
  "embeddingsProvider": {
    "provider": "openai",
    "model": "nomic-embed-text",
    "apiBase": "http://192.168.50.4/v1-openai",
    "apiKey": "gpustack_f58451c1c04d8f14_c7e8fb2213af93062b4e87fa3c319005"
  }
}

Get to Use Continue

After configuring Continue to connect to the GPUStack-deployed models, go to the top-right corner of the Continue plugin interface and select Llama 3.1 model. Now you are able to use the features we mentioned at the beginning of this tutorial:

Code Parsing: Select the code, press Cmd/Ctrl + L, and enter a prompt to let the local LLM parse the code:
Code Autocompletion: While coding, press Tab to let the local LLM attempt to autocomplete the code:
Code Refactoring: Select the code, press Cmd/Ctrl + I, and enter a prompt to let the local LLM attempt to optimize the code:

The LLM will provide suggestions, and you can decide whether to accept or reject them:

Inquire About Code Implementation: You can try @Codebase to ask questions about the codebase, such as how a certain feature is implemented:
Documentation Search: Use @Docs and select the document site you wish to search for and ask your questions, enabling you to find the results you need:

For more information, please read the official Continue documentation: https://docs.continue.dev/how-to-use-continue

Conclusion

In this tutorial, we have introduced how to use Continue + GPUStack to build a free local GitHub Copilot, offering AI-paired programming capabilities at no cost to developers.

If you encounter any issues while integrating GPUStack with third parties, feel free to join GPUStack Discord Community and get support from our engineers.

Introducing GPUStack: An open-source GPU cluster manager for running LLMs

GPUStack — Thu, 25 Jul 2024 17:00:54 +0000

What is GPUStack?

We are thrilled to launch GPUStack, an open-source GPU cluster manager for running Large Language Models (LLMs). Even though LLMs are widely available as public cloud services, organizations cannot easily host their own LLM deployments for private use. They need to install and manage complex clustering software such as Kubernetes and then figure out how to install and manage the AI tool stack on top. Popular ways to run LLMs locally, such as LMStudio and LocalAI, works on a single machine.

GPUStack allows you to create a unified cluster from any brand of GPUs in Apple MacBooks, Windows PCs, and Linux servers. Administrators can deploy LLMs from popular repositories such as Hugging Face. Developers can then access LLMs just as easily as accessing public LLM services from vendors like OpenAI or Microsoft Azure.

For more details about GPUStack, visit:

GitHub repo: https://github.com/gpustack/gpustack

User guide: https://docs.gpustack.ai

Why GPUStack?

Today, organizations who want to host LLMs on a cluster of GPU servers have to do a lot of work to integrate a complex software stack. By using GPUStack, organizations no longer need to worry about cluster management, GPU optimization, LLM interference engines, usage and metering, user management, API access, and dashboard UI. GPUStack is a complete software platform for building your own LLM-as-a-Service (LLMaaS).

As the following figure illustrates, the admin deploys models into GPUStack from a repository like HuggingFace, and then developers can connect to GPUStack to use these models in their applications.

Key features of GPUStack

GPU cluster setup and resource aggregation

GPUStack aggregates all GPU resources within a cluster. It is designed to support all GPU vendors, including Nvidia, Apple, AMD, Intel, Qualcomm, and others. GPUStack is compatible with a laptops, desktops, workstations, and servers running MacOS, Windows, and Linux.

The initial release of GPUStack supports Windows PCs and Linux servers with Nvidia graphics cards, and Apple Macs.

Deployment and Inference for Models

GPUStack supports distributed deployment and inference of LLMs across a cluster of GPU machines.

GPUStack selects the best inference engine for running the given LLM on the given GPU. The first LLM inference engine supported by GPUStack is LLaMA.cpp, which allows GPUStack to support GGUF models from Hugging Face and all models listed in the ollama library (ollama.com/library).

You can run any model on GPUStack by first converting it to GGUF format and uploading it to Hugging Face or Ollama library.

Support of other inference engines, such as vLLM, is on our roadmap and will be provided in the future.

Note: GPUStack will automatically schedule the model you select to run on machines with appropriate resources, relieving you of manual intervention. If you want to assess the resource consumption of your chosen model, you can use our GGUF Parser project: https://github.com/gpustack/gguf-parser-go. We intend to provide more detailed tutorials in the future.

Although GPU acceleration is recommended for inference, we also support CPU inference, though the performance isn't as good as GPU. Alternatively, using a mix of GPU and CPU for inference can maximize resource utilization, which is particularly useful in edge or resource-constrained environments.

Easy integration with your applications

GPUStack offers OpenAI-compatible APIs and provides an LLM playground along with API keys. The playground enables AI developers to experiment with and customize your LLMs, and seamlessly integrate them into AI-enabled applications.

Additionally, you can use the metrics GPUStack provides to understand how your AI applications utilize various LLMs. This helps administrators manage GPU resource consumption effectively.

Observability metrics for GPUs and LLMs

GPUStack provides comprehensive metrics performance, utilization, and status monitoring.

For GPUs, administrators can use GPUStack to monitor real-time resource utilization and system status. Based on these metrics:

Administrators perform scaling, optimization, and other maintenance operations.
GPUStack adjusts its model scheduling algorithm.

For LLMs, developers can use GPUStack to access metrics like token throughput, token usage, and API request throughput. These metrics help developers evaluate model performance and optimize their applications. GPUStack plans to support auto-scaling based on these inference performance metrics in future releases.

Authentication and access control

GPUStack also provides authentication and role-based access control (RBAC) for enterprises. Users on the platform can have either admin or regular user roles. This guarantees that only authorized administrators can deploy and manage LLMs and that only authorized developers can utilize them.

GPUStack Use Cases

GPUStack unlocks a world of possibilities for running LLMs on any GPU vendors. Here are just a few examples of what you can achieve with GPUStack:

Aggregate existing MacBooks, Windows PCs, and other GPU resources to offer a low-cost LLMaaS for a development team.
In limited resource environments, aggregate multiple edge nodes to provide LLMaaS on CPU resources.
Create your own enterprise-wide LLMaaS in your own data center for highly sensitive workloads that cannot be hosted in a cloud.

Getting Started with GPUStack

Installation

Linux or MacOS

GPUStack provides a script to install it as a service on systemd or launchd based systems. To install GPUStack using this method, execute:

curl -sfL https://get.gpustack.ai | sh -

Now you have deployed and started the GPUStack server, which serves as the first worker node. You can access the GPUStack page via http://myserver (Replace with the IP address or domain of the host you installed).

Log in to GPUStack with username admin and the default password. You can run the following command to get the password for the default setup:

cat /var/lib/gpustack/initial_admin_password

To add additional worker nodes and form a GPUStack cluster, please run the following command on each worker node:

curl -sfL https://get.gpustack.ai | sh - --server-url http://myserver --token mytoken

Replace http://myserver with your GPUStack server URL and mytoken with your secret token for adding workers. To retrieve the token in the default setup from the GPUStack server, use the following command:

cat /var/lib/gpustack/token

Or follow the instructions on GPUStack to add workers:

Windows

Run PowerShell as administrator, then run the following command to install GPUStack:

Invoke-Expression (Invoke-WebRequest -Uri "https://get.gpustack.ai" -UseBasicParsing).Content

You can access the GPUStack page via http://myserver (Replace with the IP address or domain of the host you installed).

Log in to GPUStack with username admin and the default password. You can run the following command to get the password for the default setup:

Get-Content -Path (Join-Path -Path $env:APPDATA -ChildPath "gpustack\initial_admin_password") -Raw

Optionally, you can add extra workers to form a GPUStack cluster by running the following command on other nodes:

Invoke-Expression "& { $((Invoke-WebRequest -Uri "https://get.gpustack.ai" -UseBasicParsing).Content) } -ServerURL http://myserver -Token mytoken"

In the default setup, you can run the following to get the token used for adding workers:

Get-Content -Path (Join-Path -Path $env:APPDATA -ChildPath "gpustack\token") -Raw

For other installation scenarios, please refer to our installation documentation at: https://docs.gpustack.ai/docs/quickstart

Serving LLMs

As an LLM administrator, you can log in to GPUStack as the default system admin, navigate to Resources to monitor your GPU status and capacities, and then go to Models to deploy any open-source LLM into the GPUStack cluster. This enables you to provide these LLMs to regular users for integration into their applications. This approach helps you to efficiently utilize your existing resources and deliver stable LLM services for various needs and scenarios.

Access GPUStack to deploy the LLMs you need. Choose models from Hugging Face (only GGUF format is currently supported) or Ollama Library, download them to your local environment, and run the LLMs:

GPUStack will automatically schedule the model to run on the appropriate Worker:

You can manage and maintain LLMs by checking API requests, token consumption, token throughput, resource utilization status, and more. This helps you decide whether to scale up or upgrade LLMs to ensure service stability.

Integrating with your applications

As an AI application developer, you can log in to GPUStack as a regular user and navigate to Playground from the menu. Here, you can interact with the LLM using the UI playground.

Next, visit API Keys to generate and save your API key. Return to Playground to customize your LLM by adjusting the system prompt, adding few-shot learning examples, or resizing prompt parameters. When you're done, click View Code and select your preferred code format (curl, Python, Node.js) along with the API key. Use this code in your applications to enable communication with your private LLMs.

you can access the OpenAI-compatible API now, for example, use curl as the following:

export GPUSTACK_API_KEY=myapikey
curl http://myserver/v1-openai/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GPUSTACK_API_KEY" \
  -d '{
    "model": "llama3",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!"
      }
    ],
    "stream": true
  }'

Join Our Community

Please find more information about GPUStack at: https://gpustack.ai.

If you encounter any issues or have suggestions for GPUStack, feel free to join our Community for support from the GPUStack team and to connect with fellow users globally.

We are actively enhancing the GPUStack project and plan to introduce new features in the near future, including support for multimodal models, additional accelerators like AMD ROCm or Intel oneAPI, and more inference engines. Before getting started, we encourage you to follow and star our project on GitHub at gpustack/gpustack to receive instant notifications about all future releases. We welcome your contributions to the project.

About Us

GPUStack is brought to you by Seal, Inc., a team dedicated to enabling AI access for all. Our mission is to enable enterprises to use AI to conduct their business, and GPUStack is a significant step towards achieving that goal.

Quickly build your own LLMaaS platform with GPUStack! Start experiencing the ease of creating GPU clusters locally, running and using LLMs, and integrating them into your applications.