DEV Community

Cover image for Deploying DeepSeek V3 (LLM) Using SGLang
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya • Originally published at docs.vultr.com

Deploying DeepSeek V3 (LLM) Using SGLang

#ai #llm #gpu #docker

DeepSeek V3 is a 671B-parameter Mixture-of-Experts language model: Multi-head Latent Attention and DeepseekMoE architecture, pre-trained on 14.8 trillion tokens, tuned with RL for strong reasoning at relatively efficient inference cost. This guide deploys it via SGLang in a ROCm-supported container on an AMD Instinct MI300X GPU server, then verifies inference over HTTP.

Prerequisites: access to an AMD Instinct MI300X GPU instance (large VRAM is required for this model's size).

Deploy DeepSeek V3

1. Install the Hugging Face CLI and start the model download in the background — it's large, so kick it off early and continue with the next steps while it completes:

$ pip install huggingface_hub[cli]
$ huggingface-cli download deepseek-ai/DeepSeek-V3
Enter fullscreen mode Exit fullscreen mode

Downloads to $HOME/.cache/huggingface.

2. Clone SGLang and build the ROCm container (can take up to 30 minutes):

$ git clone https://github.com/sgl-project/sglang.git
$ cd sglang/docker
$ docker build --build-arg SGL_BRANCH=v0.4.2 -t sglang:v0.4.2-rocm620 -f Dockerfile.rocm .
Enter fullscreen mode Exit fullscreen mode

If you hit error: RPC failed; curl 56 GnuTLS recv error during the build, add these lines to Dockerfile.rocm before the repo-cloning steps:

RUN git config --global http.postBuffer 1048576000
RUN git config --global https.postBuffer 1048576000
Enter fullscreen mode Exit fullscreen mode

Connection timeouts during build are usually transient — just re-run; Docker caches completed layers.

3. Run the inference server:

$ docker run -d --device=/dev/kfd --device=/dev/dri --ipc=host \
    --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    -v $HOME/dockerx:/dockerx -v $HOME/.cache/huggingface:/root/.cache/huggingface \
    --shm-size 16G -p 30000:30000 sglang:v0.4.2-rocm620 \
    python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code --host 0.0.0.0 --port 30000
Enter fullscreen mode Exit fullscreen mode

Runs detached with GPU device access, mounted caches, 16GB shared memory, and tensor parallelism across 8 GPUs (--tp 8), serving on port 30000.

4. Test inference:

$ curl http://localhost:30000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d "{\"model\": \"deepseek-ai/DeepSeek-V3\", \"messages\": [{\"role\": \"user\", \"content\": \"What are the key architectural ideas behind Mixture-of-Experts models?\"}], \"temperature\": 0.7}"
Enter fullscreen mode Exit fullscreen mode

5. Optional — expose the port externally:

$ sudo ufw allow 30000
Enter fullscreen mode Exit fullscreen mode

Next Steps

DeepSeek V3 is serving inference through SGLang's OpenAI-compatible API on port 30000. From here:

  • Front the server with a reverse proxy and TLS if exposing it beyond localhost
  • Tune --tp to match your available GPU count
  • Explore SGLang's batching and caching options for higher throughput under concurrent load

For the full guide, visit the original article on Vultr Docs.

Top comments (1)

Collapse
 
daymondhyper profile image
DaymondHyper •

Useful read. My main lesson with agents was that the prompt matters less than the guardrails around it. I moved most of my rules into code checks and the flip rate dropped a lot. Do you do the same or do you keep it all in the prompt?