The Ray LLM DLC combines Ray Serve, vLLM, NIXL and EFA to operate one model across multiple Amazon EKS GPU nodes through a unified API.
AWS has launched a multi-node Ray LLM Deep Learning Container for serving one large language model across multiple GPU nodes on Amazon EKS. Its reference deployment runs Qwen3.5-9B with prefill and decode on separate nodes while exposing a single OpenAI-compatible HTTP endpoint.
The container includes vLLM as the inference engine, Ray Serve as the request-serving layer, NVIDIA NIXL for transferring the KV cache, and the Elastic Fabric Adapter stack for communication between nodes. AWS says it validates the components together to prevent version drift across the CUDA runtime, framework and serving layer. The built-in build_pd_openai_app helper connects YAML configurations for prefill and decode without a custom handler or image.
AWS’s architecture distinguishes the CPU system node from the Ray head itself. The m7i.xlarge CPU system node hosts system workloads, the KubeRay operator and a CPU-only Ray head pod; the Ray head is not the host node. Two g6.8xlarge GPU worker nodes run the inference stages: prefill processes the prompt and creates the KV cache, while decode reads and extends that cache as it generates tokens.
NIXL moves the KV cache between the GPU workers through its LIBFABRIC backend over EFA. AWS warns that without an explicit LIBFABRIC setting, NIXL defaults to UCX and operates at TCP speed even on EFA without reporting an error. The GPU node group must stay within one Availability Zone because EFA cannot cross zones, and each worker requests one GPU and one EFA interface.
The example uses a 4,096-token context, bfloat16 precision and a 90% GPU-memory utilization limit. The enforce_eager setting skips CUDA Graph capture to reduce peak GPU memory, but AWS says the effect depends on the model and input. Worker containers run in privileged mode because IPC_LOCK alone is insufficient when NIXL registers CUDA VRAM with the EFA network adapter.
The release standardizes a distributed serving stack rather than introducing a new model or inference method. It can support models too large for one GPU, deployments requiring more throughput, and systems that scale prefill and decode independently. Operational requirements remain substantial: users need Kubernetes and KubeRay, at least 64 vCPUs of quota for two g6.8xlarge instances, correctly configured EFA, and verification that LIBFABRIC was actually selected.
AWS identifies retrieval-augmented generation, document summarization and agentic applications as potential workloads, particularly for long or uneven prompts. The service supports /v1/models and /v1/chat/completions, while larger deployments can add decode replicas or use tensor and pipeline parallelism. AWS reports that model-weight loading in its stated Qwen3.5-9B configuration fell from around 145 seconds to about 2 seconds on restart, but presents no broader benchmark; deployment also uses billable AWS resources that should be removed to avoid continuing charges.
| Component | Configuration | Purpose |
|---|---|---|
| CPU system node | m7i.xlarge, CPU | Hosts system workloads, the KubeRay operator and the CPU-only Ray head pod |
| GPU workers | 2 × g6.8xlarge | Separate prefill and decode stages |
| Model | Qwen3.5-9B | Multi-node inference example |
| vLLM settings | 4,096 tokens, bfloat16, 90% VRAM | Context, precision and memory limit |
| KV-cache transfer | NIXL, LIBFABRIC, EFA | High-speed communication between GPU nodes |
Sources
Originally published by AI Feed: AWS Launches Ray LLM Container for Multi-Node Inference.
Top comments (0)