How to Deploy DeepSeek-R1 & Qwen on Azure with Scale-to-Zero ($0 Idle Bill)
If you are building AI applications, you quickly hit a fork in the road:
- Proprietary APIs (OpenAI/Anthropic): Great to start, but per-token costs ramp up quickly with volume, and enterprise clients often restrict sending sensitive data to external third-party endpoints.
- Self-Hosting Open Models (DeepSeek-R1, Qwen 2.5, Llama 3.2): Total data privacy, but traditional cloud deployments on Azure require dedicated GPU virtual machines running 24/7. That can easily cost $600 to $1,800 every month—even when nobody is using the app overnight.
In this guide, I'll walk through how we solved this by building a lightweight FastAPI gateway deployed to Azure Container Apps (ACA) configured with automated scale-to-zero.
The Architecture at a Glance
The gateway sits between client applications and the underlying Hugging Face Transformers model:
-
100% Drop-in OpenAI Compatibility: Implements
/v1/chat/completionsand/v1/models. Existing LangChain, LlamaIndex, or frontend chat UIs only need theirbase_urlpointed to the Azure endpoint. - Scale-to-Zero Rule: Azure Container Apps HTTP scaler monitors traffic. If 5 minutes pass with zero incoming requests, the container replicas terminate completely. Compute cost while idle is $0.00.
- On-Demand Spin-up: The moment an HTTP request hits the gateway, Azure provisions the container replica. Cold start takes ~25–35 seconds on CPU/GPU, but once warm, responses return in milliseconds.
-
Real-Time Token Streaming: Implemented Server-Sent Events (SSE) with Hugging Face's
TextIteratorStreamerfor full typewriter streaming in client UIs.
1. The FastAPI Gateway Core
Here is the simplified implementation of the OpenAI-compatible streaming endpoint:
import json
import asyncio
from fastapi import FastAPI, Depends, HTTPException, Security
from fastapi.responses import StreamingResponse
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from pydantic import BaseModel
from typing import List, Optional
app = FastAPI(title="Private Azure AI Gateway")
security = HTTPBearer()
class ChatMessage(BaseModel):
role: str
content: str
class ChatCompletionRequest(BaseModel):
model: str
messages: List[ChatMessage]
stream: Optional[bool] = False
temperature: Optional[float] = 0.7
max_tokens: Optional[int] = 512
@app.post("/v1/chat/completions")
async def chat_completions(
req: ChatCompletionRequest,
auth: HTTPAuthorizationCredentials = Security(security)
):
if req.stream:
async def event_generator():
# Stream tokens using Server-Sent Events (SSE)
tokens = ["Hello", " from", " self-hosted", " DeepSeek!"]
for token in tokens:
chunk = {
"id": "chatcmpl-123",
"object": "chat.completion.chunk",
"choices": [{"delta": {"content": token}, "finish_reason": None}]
}
yield f"data: {json.dumps(chunk)}\n\n"
await asyncio.sleep(0.05)
yield "data: [DONE]\n\n"
return StreamingResponse(event_generator(), media_type="text/event-stream")
return {
"id": "chatcmpl-123",
"object": "chat.completion",
"choices": [{
"message": {"role": "assistant", "content": "Sample response"},
"finish_reason": "stop"
}]
}
2. Configuring Azure Container Apps Scale-to-Zero
In the deployment configuration, the secret sauce is setting min_replicas = 0:
{
"min_replicas": 0,
"max_replicas": 3,
"scale_to_zero": true,
"idle_timeout_seconds": 300
}
When deployed via Azure CLI:
az containerapp create \
--name deepseek-gateway \
--resource-group my-ai-rg \
--environment my-aca-env \
--image myacr.azurecr.io/deepseek-gateway:latest \
--target-port 8080 \
--ingress external \
--min-replicas 0 \
--max-replicas 3 \
--scale-rule-name http-rule \
--scale-rule-type http \
--scale-rule-http-concurrency 10
3. Drop-in OpenAI Client Integration
In your existing Python application, you do not need to install custom Hugging Face libraries. Just point the official OpenAI SDK to your Azure endpoint:
from openai import OpenAI
client = OpenAI(
base_url="https://deepseek-gateway.azurecontainerapps.io/v1",
api_key="your-secure-bearer-key"
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
messages=[{"role": "user", "content": "Write a Python script to sort a list of dicts."}],
stream=True
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="")
Open Source Repo & Production Boilerplate
The complete open-source repository is available on GitHub:
👉 GitHub: faizalbagwan786/azure-model-deployer
If you want the production-ready boilerplate with pre-configured NVIDIA CUDA 12.1 GPU Dockerfiles, vision support (SmolVLM), test suite, and 1-command Azure CLI automated deployment scripts, check out the Gumroad Release ($10).
If you are working on deploying custom private models on Azure or optimizing cloud AI costs, feel free to connect or ask questions in the comments below!
Top comments (0)