TechLatest

Posted on Sep 1 Originally published at Medium on Aug 21

Qwen3.8–27B Setup Guide: vLLM, Ollama, and API Integration Step-by-Step

#qwen3 #vllm #opensource #ollama

Following the widespread adoption of the Qwen3.5 and Qwen3.6 series, Alibaba’s Qwen Team has released Qwen3.8, the most capable generation in their open-model family to date.

At the center of this release is Qwen3.8–27B, a compact, deployment-friendly dense model that punches far above its weight class. Unlike traditional text-only LLMs, Qwen3.8–27B is a native vision-language model capable of understanding images, STEM diagrams, documents, and even hour-scale videos. Built on a hybrid architecture combining Gated DeltaNet (linear attention) and Gated Attention, it delivers substantial gains across coding, professional office work, scientific research, and long-horizon autonomous agent tasks.

Managed Inference Alternative: For teams seeking scalable inference without infrastructure maintenance, Qwen3.8–27B will be available as a hosted version on Qwen Cloud with 1M context length by default and official built-in tools.

Key Specifications & Highlights

Model Type: Causal Language Model with Native Vision Encoder (Dense)
Parameters: 27 Billion
Architecture: Hybrid Gated DeltaNet + Gated Attention (16 × [3 × (DeltaNet → FFN) → 1 × (Attention → FFN)])
Context Length: 262,144 tokens natively (extensible up to 1,000,000 tokens via YaRN)
Vision Capabilities: Native image and video understanding (up to hour-scale videos)
Thinking Control: Flexible reasoning modes (xhigh, medium, low) with preserved thinking across multi-turn conversations
License: Apache 2.0 (Fully open for commercial use)

Benchmark Performance: A New Bar for Coding and Coworking

Qwen3.8–27B sets new state-of-the-art records for open models of its size, competing directly with much larger proprietary systems.

Text, Coding, and Agent Benchmarks

Vision-Language (VL) and Multimodal Benchmarks

Hardware & System Prerequisites

Because Qwen3.8–27B is a dense 27B model (unlike MoE models where only a fraction of parameters are active), it requires more VRAM to load the full weights. However, its hybrid linear-attention architecture makes inference highly memory-efficient during long-context generation.

Method 1: Deploy Qwen3.8–27B Locally with vLLM

For production workloads, high-throughput scenarios, and agentic pipelines, vLLM is the recommended serving engine. It provides excellent support for Qwen3.8’s hybrid architecture, vision encoders, and 1M context extension via YaRN.

Step 1: Create an Isolated Python Environment

Create a dedicated conda environment and install PyTorch with CUDA 12.6 support.

conda create -n qwen38 python=3.11 -y
conda activate qwen38

pip install torch torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/cu126

Step 2: Install vLLM and Dependencies

Install the latest version of vLLM, which includes optimized kernels for Qwen3.8’s Gated DeltaNet layers and multimodal processing.

pip install vllm>=0.8.5 transformers accelerate qwen-vl-utils

python -c "import vllm; print(f'vLLM: {vllm. __version__ }')"

Step 3: Launch the vLLM Server (Standard 262K Context)

Launch the OpenAI-compatible API server on your local GPU. This configuration serves the model with its native 262K context window.

export MODEL_ID=Qwen/Qwen3.8-27B

vllm serve $MODEL_ID \
    --tensor-parallel-size 2 \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.90 \
    --dtype bfloat16 \
    --enable-chunked-prefill \
    --reasoning-parser qwen3

Step 4: Launch with 1M Context Extension (YaRN)

To extend the context window to 1,000,000 tokens for long-horizon tasks, enable YaRN RoPE scaling using the --hf-overrides flag.

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve $MODEL_ID \
    --tensor-parallel-size 2 \
    --max-model-len 1000000 \
    --gpu-memory-utilization 0.95 \
    --dtype bfloat16 \
    --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'

Video Processing Tip: To enable higher frame-rate sampling for hour-scale videos, launch vLLM with --media-io-kwargs '{"video": {"num_frames": -1}}' and override the longest_edge parameter in your video preprocessor config to 469762048.

Method 2: Run Qwen3.8–27B Using Ollama

For developers who want a quick, local setup without managing Python dependencies, Ollama provides a streamlined experience. Since Qwen3.8–27B is a dense 27B model, running it at full precision requires enterprise GPUs, but Ollama makes it easy to run quantized versions on consumer hardware like the RTX 4090 or RTX 5090.

Step 1: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Step 2: Pull and Run the Model

Pull the official Qwen3.8–27B model from the Ollama registry. Ollama automatically selects the best quantization for your available VRAM.

ollama pull qwen3.8:27b

ollama run qwen3.8:27b

Step 3: Run Multimodal Inference (Images)

You can pass local images directly to the model via the Ollama CLI to leverage its native vision capabilities.

ollama run qwen3.8:27b "Describe the mathematical diagram in this image: ./math-diagram.png"

API Integration & Usage Guide

Whether you are using vLLM, SGLang, or the upcoming Qwen Cloud API, Qwen3.8 uses an OpenAI-compatible Chat Completions endpoint.

Recommended Sampling Parameters

To get the best performance, use the official sampling parameters based on your desired mode:

Thinking Mode (Default): temperature=1.0, top_p=0.95, top_k=20, presence_penalty=0.0
Instruct / Non-Thinking Mode: temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5

Step 1: Text-Only Inference with Streaming and Reasoning

By default, Qwen3.8 operates in thinking mode. The following Python script demonstrates how to stream both the internal reasoning trace and the final answer, while preserving thinking blocks for multi-turn consistency.

from openai import OpenAI
import os

client = OpenAI(
    base_url=os.environ.get("OPENAI_BASE_URL", "http://localhost:8000/v1"),
    api_key=os.environ.get("OPENAI_API_KEY", "EMPTY"),
)

messages = [{"role": "user", "content": "Write a Python function to merge two sorted linked lists."}]

completion = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "enable_thinking": True, # Enabled by default
            "preserve_thinking": True, # Retains reasoning across turns
        },
        "top_k": 20,
    },
    reasoning_effort="xhigh", # Options: "xhigh" (default), "medium", "low"
    temperature=1.0,
    top_p=0.95,
    stream=True,
    stream_options={"include_usage": True},
)

reasoning_content = ""
answer_content = ""
is_answering = False

print("\n" + "=" * 20 + "Reasoning" + "=" * 20 + "\n")

for chunk in completion:
    if not chunk.choices:
        print("\nUsage:", chunk.usage)
        continue

    delta = chunk.choices[0].delta

    # Capture reasoning tokens
    if hasattr(delta, "reasoning_content") and delta.reasoning_content is not None:
        if not is_answering:
            print(delta.reasoning_content, end="", flush=True)
        reasoning_content += delta.reasoning_content

    # Capture final answer tokens
    if hasattr(delta, "content") and delta.content:
        if not is_answering:
            print("\n" + "=" * 20 + "Answer" + "=" * 20 + "\n")
            is_answering = True
        print(delta.content, end="", flush=True)
        answer_content += delta.content

# Append to message history for multi-turn preserved thinking
messages.append({
    "role": "assistant",
    "content": answer_content,
    "reasoning_content": reasoning_content,
})

Step 2: Vision-Language Inference (Image Input)

Qwen3.8–27B natively understands images. Pass an image URL or base64-encoded string alongside your text prompt to perform visual math, document extraction, or chart analysis.

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg"
                }
            },
            {
                "type": "text",
                "text": "The centres of the four illustrated circles are in the corners of the square. The two big circles touch each other and also the two little circles. With which factor do you have to multiply the radii of the little circles to obtain the radius of the big circles?"
            }
        ]
    }
]

chat_response = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=messages,
    temperature=0.7,
    top_p=0.8,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False}, # Direct answer mode
    }, 
)

print("Chat response:", chat_response.choices[0].message.content)

Step 3: Video Understanding

You can pass direct video URLs to the model. When using vLLM, you can control the frame sampling rate (fps) via mm_processor_kwargs.

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video_url",
                "video_url": {
                    "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4"
                }
            },
            {
                "type": "text",
                "text": "How many porcelain jars were discovered in the niches located in the primary chamber of the tomb?"
            }
        ]
    }
]

chat_response = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=messages,
    extra_body={
        "mm_processor_kwargs": {"fps": 2, "do_sample_frames": True},
    }, 
)

print("Chat response:", chat_response.choices[0].message.content)

Best Practices for Agentic and Long-Horizon Tasks

To achieve optimal performance when building autonomous agents with Qwen3.8–27B, follow these official guidelines:

Tune Reasoning Effort Wisely: While setting reasoning_effort="low" produces faster per-turn responses, it can lead to insufficient analysis in complex agentic tasks, resulting in failures and repeated retries that increase overall latency. Use "xhigh" for complex planning and "medium" for standard tool-calling.
Allocate Adequate Output Length: For agentic workflows within the 1M context window, configure your framework to allow up to 262,144 tokens for reasoning content and 131,072 tokens for the final response.
Leverage Preserved Thinking: Keep preserve_thinking=True (the default) for multi-turn agent loops. This prevents the model from redundantly re-reasoning about facts in previous turns, saving both time and KV-cache memory.
Adjust YaRN Factor for Typical Workloads: If your application typically processes around 524K tokens rather than the full 1M, change the YaRN factor from 4.0 to 2.0 in your config overrides to maintain higher accuracy on shorter texts.

Conclusion

Qwen3.8–27B represents a massive leap forward for open-weight AI. By combining a highly efficient hybrid DeltaNet-Attention architecture with native vision and video encoders, it delivers performance that rivals closed-source frontier models like Claude Opus 4.6 Max on key coding and computer-use benchmarks — all while remaining small enough to deploy on localized infrastructure.

With its flexible reasoning controls, 1M token context window, and permissive Apache 2.0 license, Qwen3.8–27B is ready to serve as the backbone for your next generation of multimodal software engineers, autonomous desktop agents, and visual research assistants.