Original URL: https://www.uglypear.com/en/blog/compression-task-queue-design.html
How to process compression tasks asynchronously? This article details Celery + Redis task queue design — task sharding, priority queues, retry mechanisms, and dead letter queues, with a 10,000-file async compression architecture plan and complete Celery configuration examples. Async task queues are essential for handling large-scale batch compression workloads.
1. Why Compression Tasks Need Async Queues
File compression is a typical CPU-intensive + IO-intensive task. Compressing a 100MB PDF may take 5–15 seconds — if handled with a synchronous interface, the HTTP connection hangs for a long time, and single-machine concurrency is very poor. Async queues decouple "submission" and "execution": the client submits a task and immediately gets a task_id, the Worker compresses in the background, and returns the result via callback or polling when done.
SmartSlim Network Edition uses a FastAPI + Celery + Redis + MinIO architecture, with 12 concurrent tasks running stably on a single machine. In Kubernetes deployments, HPA can auto-scale between 3–10 replicas. This architecture has supported multiple enterprise customers with daily volumes of tens of thousands of file compression requests.
| Processing Method | Concurrency | Response Latency | Failure Handling | Applicable Scale |
|---|---|---|---|---|
| Synchronous processing | Poor (blocks HTTP connections) | 5–60s | No retry, direct error | <10 files |
| Thread pool | Medium (limited by thread count) | 1–5s | Manual implementation needed | 10–100 files |
| Celery async queue | High (Worker horizontal scaling) | <200ms | Auto retry + dead letter queue | 100–100000 files |
| Kubernetes + queue | Very high (HPA elastic scaling) | <100ms | Complete fault tolerance system | >100000 files |
2. Celery Task Queue Architecture Explained
The Celery task queue consists of four core roles: Producer, Broker, Worker, and Backend. Understanding the responsibilities of these four layers is essential for correctly configuring a compression task queue.
Celery's configuration directly determines the queue's throughput and stability. The table below shows recommended configurations for compression scenarios, verified in SmartSlim's production environment.
Compression tasks have different priorities: user real-time compression requests need fast response, while scheduled archiving tasks can run slowly. Use Redis priority queues for differentiated scheduling — high-priority tasks are consumed by Workers first.
| Component | Technology Choice | Responsibility | Key Configuration |
|---|---|---|---|
| Producer | FastAPI | Receives HTTP requests, constructs tasks, delivers to Broker | task.apply_async(queue=...) |
| Broker | Redis 7.x | Stores pending task messages, supports priority queues | broker_url, visibility_timeout |
| Worker | Celery 5.x | Consumes tasks, calls Rust compression engine to execute compression | concurrency, prefork pool |
| Backend | Redis | Stores task status and return results | result_backend, result_expires |
| Storage Layer | MinIO | Stores original files and compressed files | S3-compatible protocol, multipart upload |
3. Practical Case: 10000 Files Async Compression
This is an enterprise data archiving scenario: 10000 historical documents (mixed PDF/Word/images, average 8MB per file, total about 80GB) need unified compression and archiving. Requirement: complete within 1 hour, compression ratio no less than 60%.
Solution design: Split into 100 subtasks by 100 files per shard, batch submit using Celery group, with 12 Workers consuming concurrently. Each subtask serially calls the Rust compression engine to compress 100 files.
Result: Completed 10000 file compression in 44 minutes, compression ratio 67.3%, 17 corrupted files automatically entered the dead letter queue for manual handling. The overall architecture was stable, with peak CPU utilization of 89%, peak memory usage of 4.2GB, and no OOM or task loss.
Compression task failures fall into two categories: temporary errors (IO timeout, insufficient memory, excessive concurrency) and deterministic errors (file corruption, unsupported format). Temporary errors have a high probability of success on retry, while deterministic errors are meaningless to retry. The table below provides retry and dead letter decision strategies.
Retry configuration uses Celery's autoretry_for and retry_backoff, with initial backoff of 60 seconds, maximum 600 seconds, and random jitter to avoid avalanches. Tasks in the dead letter queue are periodically scanned by an independent monitoring task, triggering WeChat Work/DingTalk alerts to notify operations for handling.
For the complete compression API calling method, refer to Compression API Guide: REST Interface Design.
| Parameter | Recommended Value | Description |
|---|---|---|
| broker_url | redis://:password@redis:6379/0 | Redis as message broker, independent DB to avoid conflicts |
| result_backend | redis://:password@redis:6379/1 | Result storage uses independent DB, isolated from Broker |
| task_serializer | json | JSON serialization, cross-language compatible |
| result_serializer | json | Results also use JSON |
| accept_content | ['json'] | Only accept JSON, security hardening |
| timezone | Asia/Shanghai | Unified timezone |
| task_acks_late | True | ACK only after task completion, no task loss on crash |
| worker_prefetch_multiplier | 1 | Each Worker prefetches only 1 task, avoiding long task starvation |
| task_time_limit | 600 | Hard timeout 600s per task |
| task_soft_time_limit | 540 | Soft timeout 540s, triggers SoftTimeLimitExceeded |
| task_reject_on_worker_lost | True | Reject task on Worker abnormal exit, re-queue |
| result_expires | 86400 | Results auto-cleaned after 24 hours |
4. Queue Configuration Recommendations for Different Scenarios
Different business scenarios have different requirements for throughput, latency, and reliability, requiring differentiated queue configurations. The table below provides recommended configurations for common scenarios.
A general principle: real-time scenarios use high-priority queues + small shards + fast retry, batch scenarios use normal queues + large shards + exponential backoff, and classified scenarios use strict auditing + small shards + multi-level retry. For the complete enterprise batch compression solution, refer to Enterprise Batch Compression Solution: 10000 File Processing in Practice.
| Queue Name | Priority | Routing Rule | Typical Tasks |
|---|---|---|---|
| compression_high | 9 (highest) | User real-time requests | Single file instant compression |
| compression_normal | 5 (default) | Batch tasks | Batch compression 100–500 files |
| compression_low | 1 (lowest) | Scheduled archiving | Nightly full archive compression |
| dlq_queue | — (dead letter) | Tasks that failed retry | Manual investigation or compensation handling |
5. Frequently Asked Questions (FAQ)
Q1: How to implement async processing for Celery compression tasks?
Use Celery + Redis to build an async task queue: FastAPI receives requests and delivers tasks to the Redis Broker, Celery Workers consume tasks from the Broker and call the Rust compression engine to execute compression, writing results to Backend and MinIO storage. A single task.apply_async executes asynchronously, and status is polled via task.id. With 12 Workers concurrent on a single machine, 10000 files sharded into 100 batches can complete in 40 minutes.
Q2: How to auto-retry failed compression tasks?
Use Celery's autoretry_for parameter to configure auto-retry, setting max_retries=3, retry_backoff=True (exponential backoff, initial 60 seconds), retry_backoff_max=600 seconds, retry_jitter=True (random jitter to avoid avalanches). Tasks that still fail after 3 retries are automatically routed to the dead letter queue dlq_queue for manual or compensation task handling. It's recommended to retry temporary errors (IO timeout/insufficient memory) and send deterministic errors (file corruption/unsupported format) directly to dead letter.
Q3: How to shard 10000 files for batch compression?
Shard by 100 files per shard, totaling 100 subtasks. Batch submit using Celery group or chord, with 12 Workers consuming in parallel, each subtask serially compressing 100 files. Average compression time per file is 3 seconds, about 5 minutes per shard, and 100 shards in parallel complete in about 40 minutes overall. Too small a shard size (e.g., 1 per shard) has high scheduling overhead, too large (e.g., 1000 per shard) has high retry cost on failure — 100 is the empirical optimal value.
Q4: Which is more suitable for compression task queues — Celery or RQ?
Celery is recommended for compression tasks. Celery supports task sharding (group/chord), priority queues, scheduled tasks, task chains, and dead letter queues — complete functionality; RQ is lighter but lacks sharding and priority. Compression scenarios commonly require batch sharding, priority scheduling, and failure retry, all natively supported by Celery. Performance-wise, both are based on Redis with comparable throughput. SmartSlim Network Edition uses a FastAPI + Celery + Redis + MinIO architecture, running stably with 12 concurrent tasks on a single machine.
| Metric | Parameter | Measured Value | Description |
|---|---|---|---|
| Total files | 10000 | — | Mixed formats, average 8MB each |
| Shard granularity | 100 files/shard | 100 shard subtasks | Balance scheduling overhead and retry cost |
| Worker concurrency | 12 | prefork mode | Single machine 12-core CPU |
| Single file compression time | — | Average 3.2s | Rust engine medium level |
| Single shard time | — | About 5.3 min | 100 files serial |
| Overall time | — | About 44 min | 100 shards/12 concurrent |
| Compression ratio | — | 67.3% | 80GB→26.2GB |
| Failures | — | 17 | File corruption, entered dead letter after retry |
6. Summary
The standard solution for compression task async processing is the Celery + Redis task queue, with the core being the four-layer decoupling of Producer/Broker/Worker/Backend. For 10000-file batch compression, sharding by 100 files with 12 Workers concurrent, completion in about 40 minutes, compression ratio 60%–70%. Retry strategies should distinguish temporary errors (exponential backoff retry) from deterministic errors (direct to dead letter), combined with 6 monitoring metrics to ensure queue stability.
Remember three points: first, task_acks_late=True ensures no task loss on crash; second, worker_prefetch_multiplier=1 avoids long task starvation; third, the dead letter queue must be configured with monitoring alerts. Choose the right queue architecture and sharding strategy, and both throughput and stability of the compression service can reach a new level.
| Error Type | Typical Exception | Handling Strategy | Retry Count | Final Destination |
|---|---|---|---|---|
| Temporary-IO | ConnectionError, TimeoutError | Exponential backoff retry | 3 times | Success or dead letter |
| Temporary-Resource | MemoryError, OOMKilled | Extended backoff + downgrade | 2 times | Success or dead letter |
| Deterministic-File | FileCorrupted, ParseError | No retry, direct dead letter | 0 times | dlq_queue |
| Deterministic-Format | UnsupportedFormat | No retry, direct dead letter | 0 times | dlq_queue |
| Deterministic-Permission | PermissionDenied | No retry, alert | 0 times | dlq_queue + alert |
| Monitoring Metric | Collection Method | Alert Threshold | Action |
|---|---|---|---|
| Queue backlog | Redis LLEN | >500 | Trigger Worker scaling |
| Task failure rate | Celery events | >5% | Investigate logs + pause submission |
| Dead letter queue length | Redis LLEN dlq | >10 | WeChat Work alert |
| Worker alive count | Celery inspect | <10 | Auto-restart Worker |
| Average task duration | Flower monitoring | >30s | Check large files + downgrade |
| CPU utilization | node_exporter | >95% | Throttle + scale |
| Scenario | Worker Count | Shard Granularity | Priority Queue | Retry Strategy |
|---|---|---|---|---|
| Personal instant compression | 2 | No sharding | high | Fast retry 3 times |
| Enterprise batch archiving | 12 | 100 files/shard | normal/low | Exponential backoff 3 times |
| Government classified processing | 4 | 50 files/shard | high | Strict retry + audit |
| E-commerce platform images | 16 | 200 files/shard | normal | Fast retry 2 times |
| Nightly scheduled archiving | 8 | 500 files/shard | low | Slow backoff 5 times |
| Real-time video transcoding | 24 | Single file | high | No retry, alert on failure |
FAQ
Q: How to design a Celery-based compression task queue?
A: Architecture: 1) Celery workers — consume compression tasks from Redis/RabbitMQ. 2) Task definitions — each compression job is a Celery task with retry logic. 3) Result backend — store task results in Redis or database. 4) Monitoring — use Flower for real-time worker monitoring. SmartSlim provides a reference Celery configuration with task routing, rate limiting, and priority queues.
Q: How to implement task sharding for large batches?
A: Task sharding splits a large batch into smaller chunks for parallel processing: 1) Group tasks into chunks of 50-100 files. 2) Submit each chunk as a separate Celery task. 3) Use Celery groups to track completion of all chunks. 4) Implement a chord callback for post-processing. SmartSlim's sharding implementation scales linearly with worker count, processing 10,000 files in ~30 minutes with 8 workers.
Q: How to handle failed compression tasks?
A: Failure handling strategy: 1) Automatic retry — Celery's task.retry() with exponential backoff (retry 3 times, delays: 60s, 300s, 900s). 2) Max retries exceeded — move to dead letter queue. 3) Dead letter processing — log failure details, notify admin, store for manual review. 4) Partial success — complete remaining tasks, report failures in final summary. SmartSlim's Celery configuration includes comprehensive failure handling.
Q: What monitoring and alerting should be set up?
A: Monitoring stack: 1) Celery monitoring — Flower dashboard for worker status and task queues. 2) Application metrics — Prometheus metrics for task throughput, success rate, processing time. 3) Log aggregation — ELK stack (Elasticsearch, Logstash, Kibana) for compression logs. 4) Alerting — Grafana alerts for queue depth > 1000, failure rate > 5%, worker downtime. SmartSlim provides a complete monitoring configuration with pre-built Grafana dashboards.
Summary
The key to compression task queue design: celery... lies in identifying the sources of bloat and handling them accordingly. Choose the right compression strategy based on your scenario, prioritizing the largest contributors. SmartSlim can handle all compression steps in one click.
Related:
Top comments (0)