Our Ollama /LLM Production Setup

Response to Ollama Production Queries to the previous post

Excellent questions — you've clearly been in the trenches with Ollama at scale! Let me address your specific concerns about our production setup, now updated with our actual deployment architecture.

---

On GPU Residency & VRAM Thrashing

You're absolutely right — this is the single biggest operational challenge with Ollama in production. We've implemented a hybrid strategy:

Our Approach:

1. Hot Models Pinned in Memory:
   • qwen3:8b (primary inference)
   • bge-m3 (embedding service)
   • qwen3:4b-instruct (NER-specific)

These stay resident with OLLAMA_KEEP_ALIVE=-1 in the container environment.

1. Cold Models on Separate Instance:

   • gemma3:4b, llama3.2, custom RatnaDveLinga
   • These run on a secondary Ollama instance with a lower concurrency limit
   • The load balancer routes to this instance only for specific workloads

2. VRAM Monitoring:
   

   # We run this in a sidecar container
   watch -n 1 nvidia-smi --query-gpu=memory.used,memory.free --format=csv

The Keep-Alive Tuning We Use:

# docker-compose environment section
environment:
  - OLLAMA_KEEP_ALIVE=5m  # Not -1 to avoid holding all models
  - OLLAMA_NUM_PARALLEL=4
  - OLLAMA_MAX_LOADED_MODELS=3  # Critical! Prevents thrashing

Key insight: Setting OLLAMA_MAX_LOADED_MODELS to 3 forces eviction behavior to be predictable rather than reactive. We learned this after a production incident where p99 latency jumped from 200ms to 8 seconds during a batch processing job!

---

On Concurrent Load & Service Architecture

Great observation about OLLAMA_NUM_PARALLEL — default is 1 in many versions! Here's our production design at scale:

Our Actual Deployment Architecture:

Service	Instance Count	RAM (GB)	CPU (Cores)	Purpose
Ollama	21	-	-	LLM inference engine
Ollama-WebUI	21	-	-	Web interface for model management
forms-service-ner-v4-api	2	15	7.5	Production NER API endpoints
forms-service-ner-v4-celery	3	15	7.5	Async NER task workers (prod)
forms-service-ner-qa-api	2	15	3.9	QA environment NER APIs
forms-service-ner-qa-celery-server	6	15	3.9	QA async task workers
forms-service-ner-api(sales)	4	15	3.9	Production -specific NER APIs
forms-service-ner-celery(sales)	4	15	3.9	Production async task workers
signal-summarizer-llm	2	-	-	Main summarizer service
signal-summarizer-llm workflow-aggregator	9	-	-	Workflow orchestration
signal-summarizer-llm worker-model1	9	-	-	Model execution workers
signal-summarizer-llm celery-flower	1	-	-	Celery monitoring dashboard

Load Distribution Strategy:

                    ┌──────────────────────────────────────┐
                    │       KONG API Gateway               │
                    │  (Least Connection Algorithm)        │
                    └─────────────┬────────────────────────┘
                                  │
            ┌─────────────────────┼─────────────────────┐
            │                     │                     │
    ┌───────▼──────┐    ┌────────▼────────┐    ┌───────▼──────┐
    │  NER Services │    │  Summarizer     │    │  Ad-hoc      │
    │  (17 pods)    │    │  Services       │    │  Queries     │
    └───────┬──────┘    │  (20 pods)      │    └───────┬──────┘
            │           └────────┬────────┘            │
            └─────────────────────┼─────────────────────┘
                                  │
                    ┌─────────────▼─────────────┐
                    │   21 Ollama Instances      │
                    │   (Distributed Pool)       │
                    └────────────────────────────┘

How KONG's Least Connection Load Balancing Helps:

1. NER Service Distribution (17 instances total):

• Production NER: 2 API + 3 Celery = 5 instances (high memory/CPU)
• QA NER: 2 API + 6 Celery = 8 instances (lower CPU, more workers)
• Sales NER: 4 API + 4 Celery = 8 instances (balanced)

KONG routes traffic to the least loaded instance, which is crucial for:

• Preventing any single NER instance from being overwhelmed by large document batches
• Allowing Celery workers to scale independently based on queue depth
• Isolating QA traffic from production traffic

2. Summarizer Service Distribution (20 instances total):

• Main service: 2 instances
• Workflow aggregators: 9 instances (coordinating multi-step summaries)
• Model workers: 9 instances (doing the heavy lifting)
• Celery flower: 1 instance (monitoring)

Request Flow with Backpressure:

# Simplified version of our queue management with distributed instances
class OllamaLoadBalancer:
    def __init__(self):
        # Track load across all 21 Ollama instances
        self.instance_pool = self.discover_ollama_instances()

    def route_request(self, model, prompt, service_type):
        # Find least loaded instance via KONG
        if service_type == 'ner':
            # Route to NER-specific Celery queue
            # Celery distributes across 13 NER workers (2+3+4+4)
            instance = self.get_least_loaded(['forms-ner-v4', 'forms-ner-qa', 'forms-ner-sales'])
        elif service_type == 'summarizer':
            # Route to summarizer Celery queue
            # Celery distributes across 9 model workers
            instance = self.get_least_loaded(['summarizer-workflow', 'summarizer-model1'])

        # Set appropriate parallelism per service type
        if service_type == 'ner':
            ollama_timeout = 30  # seconds
            ollama_concurrency = 4  # OLLAMA_NUM_PARALLEL
        else:  # summarizer
            ollama_timeout = 120  # seconds (larger context)
            ollama_concurrency = 2  # careful with VRAM

    def get_least_loaded(self, services):
        # Query KONG's least-connection algorithm
        return kong_api.get_least_connected_instance(services)

---

What We've Learned at This Scale

The Good:

• 21 Ollama instances across servers provide excellent fault tolerance
• Least connection algorithm naturally handles hot spots
• Separate QA vs Production instances (8 NER QA pods) allow safe testing
• Sales-specific NER (8 pods) isolates business-critical workloads

The Challenges:

1. Model Consistency: Keeping all 21 Ollama instances synchronized with same models and versions

   • Solution: We use a shared NFS volume for models, or have a model-sync cron job

2. Celery Worker Configuration: With 13 NER Celery workers and 9 summarizer workers, we had to tune:
   

   # Celery config
   CELERYD_PREFETCH_MULTIPLIER = 1  # Prevent worker hogging
   CELERY_ACKS_LATE = True  # Re-queue if worker crashes
   CELERY_TASK_TIME_LIMIT = 300  # 5 min max
   CELERY_TASK_SOFT_TIME_LIMIT = 240

1. KONG Routing Complexity: With least-connection across heterogeneous instance types (some 7.5 CPU cores, some 3.9), we had to implement:

   -- Kong plugin to weigh instances by capacity
   -- Instance with 7.5 cores should get ~2x traffic of 3.9 core instance

---

Specific Q & A

Q: "Given your NER and summarizer run on the large-document path, how are you handling concurrent load — one instance per service, or a shared pool with request queuing in front?"

A: Distributed pool with service-specific queues:

• NER: 17 total instances (5 prod + 8 QA + 4 sales) each with OLLAMA_NUM_PARALLEL=4 → 68 concurrent NER requests possible
• Summarizer: 20 total instances (2 main + 9 workflow + 9 workers) with OLLAMA_NUM_PARALLEL=2 → 40 concurrent summarizer requests possible
• All fronted by KONG with least-connection algorithm → no single point of overload

The key insight: With 21 Ollama instances, even if one gets busy, KONG routes to the next least-loaded. Celery handles the queue depth and retries.

---

Critical Monitoring in Production

We track these metrics per service:

Service	Critical Metrics	Alert Threshold
NER (All)	Queue length > 50 per celery worker	Scale up workers
NER (Prod)	99th percentile latency > 3s	Investigate model/instance
Summarizer	99th percentile latency > 8s	Check GPU usage
Ollama Instances	Memory usage > 80%	Evict cold models
KONG	504/502 errors rate > 1%	Check instance health

---

One More Thing: Metrics Are Your Friend

With 21 Ollama instances and 37 service instances (NER + Summarizer), observability is critical:

# Distributed tracing across services
# We use OpenTelemetry with Jaeger
# This helps identify if latency is in:
# 1. KONG routing
# 2. Celery queuing
# 3. Ollama inference
# 4. Network between instances

---

Bottom line: Your assessment is spot-on. The production challenges with Ollama aren't getting it running — they're keeping it running predictably under mixed workloads across 21 instances. We've found that explicit model management with OLLAMA_MAX_LOADED_MODELS, proper Celery tuning, and KONG's least-connection balancing are non-negotiable at this scale.

The 21-instance deployment gives us:

• High availability (can lose 2-3 instances without impact)
• Scalability (can route QA vs Prod traffic separately)
• Performance (least-connection ensures no single instance is overwhelmed)

Happy to dive deeper into any of these aspects further 🚀