DEV Community

Cover image for How to Use Qwen 3.5 with Ollama
Preecha
Preecha

Posted on

How to Use Qwen 3.5 with Ollama

TL;DR

Ollama provides the easiest way to run Qwen 3.5 small models (0.8B, 2B, 4B, and 9B) locally on your Mac, Linux, or Windows machine. With a simple ollama run command, you can access capable AI features without cloud API costs. Download Ollama, pull a model, and start chatting in under 5 minutes.

Image

Introduction

Running large language models locally has become very popular, and Ollama makes it straightforward. If you want to use Alibaba's Qwen 3.5 models without sending data to the cloud or paying per-token fees, Ollama is the answer.

Try Apidog today

This guide walks you through everything you need to know about running Qwen 3.5 small models with Ollama. Whether you need the compact 0.8B model for quick tasks or the larger 9B model for complex reasoning, we'll cover installation, usage, and integration.

Why Use Ollama for Qwen 3.5

Ollama has become the go-to solution for local LLM deployment:

  • Simple Setup: No complex Docker or Python setups. Download one app and you're ready.
  • Privacy First: Your data stays on your machine. This matters for business data or anything sensitive.
  • No API Costs: After downloading models, running them is free. No per-token fees or subscriptions.
  • Offline Capability: Use AI anywhere, even without internet.
  • Hardware Acceleration: Ollama automatically uses GPU acceleration when available, making local inference fast.

Installing Ollama

Mac Installation

If you have a Mac, installation takes seconds:

# Download from ollama.com or use Homebrew
brew install ollama
Enter fullscreen mode Exit fullscreen mode

That's it. Ollama will automatically detect Apple Silicon (M1/M2/M3) and use Metal for GPU acceleration.

Linux Installation

For Linux servers or WSL:

# Quick install
curl -fsSL https://ollama.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Windows Installation

Windows users can download the installer. The Windows version supports GPU acceleration via DirectML.

Image

Verification

After installation, verify everything works:

ollama --version
Enter fullscreen mode Exit fullscreen mode

You should see the version number. Now let's pull some Qwen models.

Running Qwen 3.5 Models

Pulling Your First Model

Ollama makes downloading models simple:

# 9B model
ollama run qwen3.5:9b

# 4B model
ollama run qwen3.5:4b 

# 2B model 
ollama run qwen3.5:2b 

# 0.8B model
ollama run qwen3.5:0.8b
Enter fullscreen mode Exit fullscreen mode

Each model download takes a few minutes depending on your internet speed. The 2B model is around 1.5GB, while the 9B model is about 5GB.

Starting a Chat Session

Once pulled, start chatting immediately:

ollama run qwen3.5:9b
Enter fullscreen mode Exit fullscreen mode

You'll see a prompt where you can type directly:

>>> What is quantum computing in simple terms?
Quantum computing is a type of computation where...
Enter fullscreen mode Exit fullscreen mode

Type your questions and press Enter. Press Ctrl+D to exit.

Listing Available Models

See what you have installed:

ollama list
Enter fullscreen mode Exit fullscreen mode

Output shows each model, its size, and when you last used it.

Removing Models

Free up disk space by removing models you don't need:

ollama remove qwen3.5:9b
Enter fullscreen mode Exit fullscreen mode

Model Comparison and Selection

Choosing the right model depends on your hardware and use case:

Model Parameters Approx. Model Size RAM Needed Best For
Qwen3.5-0.8B 0.8B ~1.6 GB ~9 GB Ultra-light edge & mobile use
Qwen3.5-2B 2B ~4 GB ~9 GB Lightweight assistants, small agents
Qwen3.5-4B 4B ~8 GB ~14 GB Smart autocomplete, better reasoning
Qwen3.5-9B 9B ~18 GB ~19 GB Strong general assistant

Recommendation for most users: Start with qwen3.5:2b. It offers the best balance of capability and speed. Upgrade to 4B or 9B only if you need more reasoning power.

Ollama API for Developers

Ollama runs a local API server that your applications can call at http://localhost:11434.

Basic Chat Completion 

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:0.8b",
    "messages": [
      {"role": "user", "content": "What is Python?"}
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

Image

Streaming Responses

For real-time output:

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:9b",
    "messages": [{"role": "user", "content": "Count to 5"}],
    "stream": true
  }'
Enter fullscreen mode Exit fullscreen mode

Generation Endpoint

For non-chat prompts:

curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:0.8b",
    "prompt": "Write a haiku about coding"
  }'
Enter fullscreen mode Exit fullscreen mode

Integrating with Your Applications

Python Integration 

import requests

url = "http://localhost:11434/api/chat"
payload = {
    "model": "qwen3.5:0.8b",
    "messages": [{"role": "user", "content": "Explain recursion"}]
}

response = requests.post(url, json=payload)
print(response.json()["message"]["content"])
Enter fullscreen mode Exit fullscreen mode

JavaScript/Node.js Integration 

const response = await fetch('http://localhost:11434/api/chat', {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: JSON.stringify({
    model: "qwen3.5:0.8b",
    messages: [{role: 'user', content: 'What is an API?'}]
  })
});

const data = await response.json();
console.log(data.message.content);
Enter fullscreen mode Exit fullscreen mode

Testing Your API with Apidog

  1. Create a new POST request to http://localhost:11434/api/chat
  2. Set headers:
    • Content-Type: application/json
  3. Add request body: 
{
  "model": "qwen3.5:0.8b",
  "messages": [{"role": "user", "content": "Hello"}]
}
Enter fullscreen mode Exit fullscreen mode

Image

Performance and Hardware Requirements

Expected Performance

Model Apple Silicon Tokens/sec
0.8B M1/M2 40-50
2B M1/M2 20-30
4B M1/M2 10-15
9B M3 Max 15-20

Memory Requirements

  • 0.8B: 2GB available RAM
  • 2B: 4GB available RAM
  • 4B: 8GB available RAM
  • 9B: 16GB available RAM

Troubleshooting Common Issues

Issue Solution
"Ollama not found" Restart terminal or add to PATH
Slow Performance Check GPU usage with ollama list
Model Download Fails Retry with stable internet
API Connection Refused Start Ollama service: ollama serve
Out of Memory Use smaller model or free up RAM

Conclusion

Ollama makes running Qwen 3.5 models locally straightforward. The combination of simple commands and hardware acceleration makes it ideal for developers.

FAQ

Can I use Ollama with other Qwen models?

Yes, check ollama.com/library for supported models.

How do I update models?

Run ollama pull qwen3.5:2b to get updates.

Can I run multiple models?

Yes, but each consumes memory.

Is my data secure?

Yes, all processing happens locally.

Can I fine-tune models?

No, Ollama is for inference only.

Top comments (0)