DEV Community: Uglypear Data

Email Attachment Too Large? Mail Limits and Compression Strategies

Uglypear Data — Thu, 13 Aug 2026 04:12:49 +0000

Original URL: https://www.uglypear.com/en/blog/email-attachment-compression.html

Can't send email attachments that are too large? This article summarizes attachment size limits for QQ Mail, WeCom, Outlook, and Gmail, detailing 3 compression strategies — file compression, split archives, and cloud storage relay, with a comparison table and real case studies. The key is understanding your email provider's limits and choosing the right strategy for each situation.

1. Comparison of Attachment Size Limits by Email Provider

Different email providers have strict limits on attachment size, and these limits are hard — exceeding by even 1KB means the email won't send. Understanding the limits of the recipient's email provider is the first step in choosing a compression strategy. The table below summarizes the attachment limits of mainstream domestic email providers and office tools.

As the table above shows, WeCom, Outlook, and DingTalk have the strictest limits (20MB), Gmail is 25MB, and QQ and 163 Mail are relatively lenient (50MB). This means attachments sent to enterprise users should ideally be kept under 20MB to be safe. An 80MB PDF report cannot be sent through any mainstream email provider without compression — compression is a necessary step.

Email/Tool	Standard Attachment Limit	Large Attachment Feature	Over-Limit Handling
QQ Mail	50MB	Supported, up to 3GB (7-day validity)	Auto-convert to large attachment link
WeCom	20MB	WeDrive share link	Prompt to use WeDrive
Outlook (web)	20MB	OneDrive link	Auto-upload to OneDrive
Gmail	25MB	Google Drive link	Auto-upload to Drive
163 Mail	50MB	Supported, up to 2GB	Prompt to use Netdisk
DingTalk	20MB	DingDrive share link	Prompt to use DingDrive

2. Detailed Explanation of 3 Email Attachment Compression Strategies

Based on file type and size, there are 3 mainstream strategies for dealing with attachment over-limit issues. Each strategy differs in principle, applicable scenario, and operational difficulty. The table below provides an overall comparison first, followed by detailed explanations.

File compression is the preferred strategy — compressing the file itself before sending both solves the attachment over-limit problem and reduces transfer time and storage consumption. Different file types require different compression strategies: PDF documents mainly rely on embedded image downsampling and font subsetting, images rely on format conversion (PNG to WebP/JPEG) and resolution reduction, and Office documents rely on compressing embedded resources and cleaning up redundancy.

Taking the most common PDF compression as an example, SmartSlim, based on its Rust compression engine, automatically identifies the image types within a PDF and assigns strategies: scanned documents are downsampled to 150DPI and converted to grayscale JPEG (90%+ compression), photos are reduced to 200DPI and converted to color JPEG quality 80 (80% compression), and vector charts are kept as-is with only structural optimization. This content-based assignment works better than uniform parameters — an 80MB scanned contract can be compressed to 6MB, while text and charts remain virtually lossless.

For the complete methodology of file compression, see the Complete Guide to File Compression.

Some files still exceed email limits even after compression — for example, a collection of high-definition images that's still 60MB after compression, or video files that are inherently incompressible. In these cases, use split archive compression: split a large package into multiple smaller packages under the limit, send them in multiple emails, and the recipient downloads all volumes and merges them for extraction.

The volume size should be set to 80% of the email limit to leave a margin. WeCom limits 20MB, so set volumes to 15MB; QQ Mail limits 50MB, so set volumes to 40MB. A 60MB compressed package split into 15MB volumes will produce 4 packages (15+15+15+15), sent in 4 emails. Note that all volumes must use the same format (ZIP or 7Z), and the recipient needs the corresponding extraction tool.

When the file itself is an incompressible format (such as video, or an already-compressed ZIP package), or when the size exceeds 100MB, sending it directly as an attachment is impractical. In this case, use cloud storage relay: upload the file to cloud storage, generate a share link, and only send the link and extraction code in the email. The recipient clicks the link to download the file themselves, unconstrained by email limits.

Common cloud storage services and their limits: Baidu Netdisk single file limit 4GB (free), Alibaba Cloud Drive single file limit 100GB (free), Tencent WeDrive single file limit 1GB (enterprise 100GB), OneDrive single file limit 250GB. For internal enterprise use, enterprise cloud drives (WeDrive/DingDrive/OneDrive for Business) are recommended, as they allow access permission control and provide audit logs.

Strategy	Principle	Applicable Scenario	Expected Compression Rate	Operational Difficulty
File compression	Downsampling + format conversion + redundancy removal	PDF/images/Office documents	70%–92%	★☆☆☆☆
Split archive	Split large package into smaller packages	Files still over-limit after compression	No size reduction, split for sending	★★☆☆☆
Cloud storage relay	Upload to cloud and send share link	Video/ultra-large data packages	No size reduction, link replaces file	★☆☆☆☆

3. Real-World Case: 120MB Project Report Compressed to 18MB for Sending

A consulting firm needed to send a quarterly project report to a client via WeCom. The report was in PDF format, originally 120MB, containing 80 pages of text, 45 on-site project photos, 12 data charts, and 3 pages of scanned signature pages. WeCom's attachment limit is 20MB, and the client's IT policy did not allow the use of external cloud drives, so it had to be sent as a direct attachment.

Document characteristics: 80 pages A4, 45 photos at 300DPI JPEG format (totaling 85MB), 12 data charts in vector PDF format (totaling 15MB), 3 scanned signature pages at 600DPI (totaling 18MB), embedded fonts 2MB.

Result: 18MB, a compression rate of 85%, below WeCom's 20MB limit. The on-site photos showed virtually no difference to the naked eye on screen, the scanned signature pages had clearly legible text and official seals, and the data charts had sharp lines with no loss. The entire process was completed locally, and the report did not pass through any online server, meeting the confidentiality requirements for business documents.

File Type	Original Size	Compressed Size	Compression Rate	Quality Impact
Scanned contract PDF	80MB	6MB	92.5%	Text clearly legible
Project report PDF (with images)	120MB	18MB	85.0%	Virtually no difference for screen reading
Product manual PDF	45MB	8MB	82.2%	Slight loss of image detail
Word bid document	50MB	5MB	90.0%	Text and charts unaffected
Image collection (PNG)	200MB	30MB	85.0%	Needs split archive for sending

4. Attachment Sending Recommendations for Different Scenarios

Different types of files have very different size characteristics and compression potential. The table below provides attachment sending strategy recommendations by common office scenario.

One general principle: Compress first whenever possible (reduces both size and transfer time), use split archives if still over-limit after compression, and use cloud storage for ultra-large incompressible files. Confirm the recipient's email limit first, then decide on a strategy — attachments sent to WeCom users should be kept under 20MB to be safest.

If you need to compare different compression tools for selection, see the File Compression Tool Selection Guide: 7-Dimension Evaluation.

Target Email	Attachment Limit	Recommended Volume Size	Number of Splits for 60MB File
WeCom	20MB	15MB	4 packages
Outlook	20MB	15MB	4 packages
Gmail	25MB	20MB	3 packages
QQ Mail	50MB	40MB	2 packages

5. Frequently Asked Questions

Q1: What to do when an email attachment exceeds the email limit?

Three options: First, use a compression tool to compress the file under the limit (e.g., compress a PDF from 80MB to 6MB). Second, if it can't be compressed enough, use split archive compression to break it into multiple packages under the limit and send them in multiple emails. Third, for ultra-large files, use cloud storage relay — upload to cloud storage and generate a share link, sending only the link in the email. A 120MB project report can first be compressed to 18MB, then split into 2 volumes of 9MB each to fit WeCom's 20MB limit. The key is to first understand the recipient's specific email limit, then choose the corresponding strategy.

Q2: Will PDF quality degrade after compression for email?

With proper compression, the difference is virtually imperceptible to the naked eye. 150DPI is sufficient for screen reading, and scanned documents reduced to 150DPI and converted to grayscale JPEG can be compressed by over 90% while keeping text clearly legible. Photo content reduced to 200DPI with JPEG quality 80 shows virtually no difference on screen or A4 print. The key is to assign strategies by content type — keep text pages as vector, downsample photo pages, and grayscale scan pages — to avoid blurriness from a one-size-fits-all approach. SmartSlim automatically identifies PDF content types and assigns the optimal strategy.

Q3: What are the attachment limits for WeCom and QQ Mail?

WeCom has a single-file attachment limit of 20MB, suitable for sending compressed documents. QQ Mail has a standard attachment limit of 50MB, and its large attachment feature supports files up to 3GB but requires the recipient to download from the server. Outlook limits 20MB, and Gmail limits 25MB. Before sending, confirm the recipient's email limit, then decide whether to use direct attachment or cloud storage relay. For cross-enterprise sending, it's recommended to prepare attachments according to the strictest limit (20MB).

Q4: How does the recipient extract a split archive?

Place all volume packages in the same folder, right-click the first package (usually with a .part1 or .zip.001 suffix) and select extract, and the tool will automatically recognize and merge the remaining volumes. Make sure all volumes are fully downloaded — missing any one will prevent extraction. We recommend using 7-Zip or SmartSlim, which are compatible with both ZIP and 7Z split formats. Do not rename the volume packages, as this may cause recognition failure.

Step	Operation	Key Parameters	Size Change
1	On-site photo compression	300DPI→150DPI, JPEG q80	120→35MB
2	Scanned signature page compression	600DPI→150DPI, grayscale JPEG q72	35→19MB
3	Data chart optimization	Vector preserved, remove redundant objects	19→18.5MB
4	Font subsetting + repackaging	Keep only used characters	18.5→18MB

6. Summary

Email attachments can't be sent because the file size exceeds the email provider's hard limit. The 3 strategies have clear logic: File compression is the first choice (downsampling + format conversion + redundancy removal, 70%–92% compression rate), split archive compression is the backup (split packages for sending when still over-limit after compression), and cloud storage relay is the fallback (send links for ultra-large incompressible files). A 120MB project report can be reliably compressed to 18MB using the "compress first, then send" strategy, fitting all mainstream email providers.

Remember three points: First, confirm the recipient's email limit before choosing a strategy — preparing for 20MB is safest for enterprise users. Second, prioritize compressing the file itself (reduces both size and transfer time), and use split archives or cloud storage only when compression isn't enough. Third, use local compression tools for sensitive business documents — don't upload them to online compression websites. Choose the right strategy, and even the largest email attachments can be sent.

Scenario	Typical File	Original Size	Recommended Strategy
Business contract signing	Scanned PDF	20–80MB	File compression: scan pages downsampled + grayscale JPEG, 90%+ compression
Project report delivery	Image/text PDF/Word	50–200MB	File compression: image downsampling + font subsetting, 80%–90% compression
Design draft review	PSD/AI/PNG	100–500MB	File compression + split archive: export JPEG then compress, split for sending
Video material delivery	MP4/MOV	200MB–2GB	Cloud storage relay: compress video then upload to cloud and send link
Data report sending	Excel/PDF	5–30MB	File compression: Excel redundancy cleanup + PDF optimization
Bid document submission	Word/PDF	30–80MB	File compression: image compression + font subsetting + redundancy cleanup

FAQ

Q: What are the attachment size limits for major email providers?
A: QQ Mail: 50MB for free, 2GB for VIP. WeCom (Enterprise WeChat): 100MB. Outlook/Hotmail: 34MB. Gmail: 25MB. 163 Mail: 50MB for free, 100MB for VIP. If your file exceeds these limits, use compression first, then split archives, or switch to cloud storage relay. SmartSlim can compress files to fit within provider limits.

Q: How to send files over 100MB via email?
A: Files over 100MB cannot be sent directly via email (most providers limit to 25-50MB). Three solutions: 1) Compress heavily — use 7Z with ultra compression to minimize size. 2) Split archive — use volume splitting in WinRAR/7-Zip (e.g., 20MB volumes). 3) Cloud relay — upload to cloud storage (Baidu Netdisk, WeTransfer, Google Drive) and share the link. SmartSlim integrates with cloud storage services for seamless large-file sharing.

Q: What is the best compression format for email attachments?
A: ZIP is the most compatible format for email attachments — all recipients can open it without additional software. If the recipient has 7-Zip or WinRAR, 7Z offers 30-50% better compression. For maximum compression, use 7Z with LZMA2 algorithm. For encrypted files, use ZIP with AES-256 encryption (supported by Windows, macOS, and major archivers).

Q: How to split a large file into smaller volumes for email?
A: Use WinRAR or 7-Zip's volume splitting feature: in WinRAR, select 'Split to volumes' and set size (e.g., 20MB). In 7-Zip, use '-v20m' parameter. The tool creates multiple numbered files (.part1.rar, .part2.rar, etc.). The recipient needs all parts in the same folder to extract. SmartSlim supports automatic volume splitting with customizable size limits.

Summary

The key to email attachment too large? mail... lies in identifying the sources of bloat and handling them accordingly. Choose the right compression strategy based on your scenario, prioritizing the largest contributors. SmartSlim can handle all compression steps in one click.

What's the Difference Between DPI and PPI? Essential for Scan Compression

Uglypear Data — Wed, 12 Aug 2026 02:39:46 +0000

Original URL: https://www.uglypear.com/en/blog/dpi-vs-ppi-guide.html

What's the difference between DPI and PPI? This article explains from three angles: definition, usage scenarios, and conversion relationships, with a scan DPI selection guide — comparing size and sharpness for screen reading at 150 DPI, printing at 300 DPI, and archiving at 600 DPI. Understanding this distinction is essential for effective scanned document compression.

1. DPI vs PPI Definition Comparison

DPI and PPI are the two most easily confused concepts in document scanning and image processing. Although both units are "dots/pixels per inch," they describe resolution in different domains. Getting the distinction clear is the prerequisite for making the right choices in scanning and compression.

Simple mnemonic: DPI governs the "physical world" (ink dots on paper), PPI governs the "digital world" (pixels on screen). A scanner's advertised "600DPI scan resolution" means it samples 600 dots per inch; once stored as a digital image, those dots become pixels. An A4 page scanned at 600DPI has a digital image pixel dimension of 4961x7016.

Comparison Item	DPI	PPI
Full name	Dots Per Inch	Pixels Per Inch
Meaning	Physical dots per inch	Pixels per inch
Application domain	Printing, scanning	Screen display, digital images
Devices	Printers, scanners	Monitors, phone screens
Affects	Print clarity, scan precision	Screen display fineness
Typical values	150/300/600/1200	72/96/150/326
Modifiable?	Set at scan time, not after	Digital images can be resampled

2. Scanned File Size by DPI

The relationship between scanned file size and DPI follows the square law: pixel count is proportional to DPI squared, and so is file size. This is the key formula for understanding scanned file size changes.

DPI-size relationship formula: File size ≈ (DPI / baseline DPI)^2 × baseline size. Taking an A4 page (8.27x11.69 inches) as an example, grayscale JPEG quality 80:

The table above clearly shows: doubling DPI from 150 to 300 quadruples size; doubling from 300 to 600 quadruples again. A 1200DPI A4 color page reaches 136MB — 64x that of 150DPI. This is why blindly chasing high DPI causes file size to explode.

Now real-world data for different page types, comparing grayscale and color modes:

Typical PPI values by device, to help understand PPI in real devices:

DPI	Pixel Dimensions	Total Pixels	Grayscale JPEG	Color JPEG	Relative to 150DPI
72	595x842	500K	0.2MB	0.5MB	0.23x
150	1240x1754	2.17M	0.8MB	2.1MB	1x
200	1654x2339	3.87M	1.4MB	3.8MB	1.78x
300	2480x3508	8.70M	3.2MB	8.5MB	4x
600	4961x7016	34.81M	12.8MB	34.0MB	16x
1200	9922x14032	139M	51.2MB	136.0MB	64x

3. Case Study: DPI Selection for an 80-Page Contract Archive

An enterprise needed to scan and archive 80 pages of paper contracts, while also emailing an electronic version. We compared three DPI approaches.

Document characteristics: 80-page A4 contracts, with handwritten signatures and stamped pages, mostly plain text, no color images. Goal: clear, readable archive + emailable version (25MB limit).

The final approach was a dual-version strategy: scan at 300DPI grayscale as the master (25.6MB), then use SmartSlim to downsample to 150DPI for the email version (6.4MB). The master was archived to the server, and the email version was sent via enterprise WeChat. The entire process was done locally — contract content was never uploaded to any third-party server, meeting commercial confidentiality requirements.

Document Type	Pages	150DPI Grayscale	300DPI Grayscale	300DPI Color	600DPI Grayscale
Plain-text contract	10	1.8MB	7.2MB	19.0MB	28.8MB
Mixed text/image report	30	6.5MB	26.0MB	68.5MB	104.0MB
Engineering drawing A1	1	3.2MB	12.8MB	34.0MB	51.2MB
ID card scan	1	0.15MB	0.6MB	1.6MB	2.4MB

4. Scenario Recommendations: How to Choose DPI

Different use scenarios have different clarity requirements. The core principle for choosing DPI is "enough is enough" — too high wastes storage and bandwidth, too low affects usability. The table below gives DPI recommendations for common scenarios.

A practical tip: when SmartSlim processes scanned files, its Rust compression engine automatically analyzes document type — downsampling plain-text pages to 150DPI, keeping image-containing pages at 300DPI, and preserving original DPI for signature/stamp pages. This per-page dispatch strategy works better than uniform DPI reduction, maximizing compression ratio while ensuring clarity.

For more image compression methods, see Image Compression Guide: JPG/PNG/WebP Format Comparison and PDF Compression Principles and Methods in Depth.

Device Type	Screen Size	Resolution	Typical PPI	Clarity Assessment
Phone (flagship)	6.1 in	2532x1170	460PPI	Retina-level, extremely fine
Phone (mid-range)	6.5 in	2400x1080	405PPI	Clear, sufficient for daily use
Tablet	11 in	2360x1640	264PPI	Clear for document reading
Laptop	14 in	1920x1080	157PPI	Office-standard clarity
Monitor	27 in	2560x1440	109PPI	Mainstream desktop office
Large monitor	32 in	3840x2160	138PPI	4K clarity

5. Frequently Asked Questions

Q1: What exactly is the difference between DPI and PPI?

DPI (Dots Per Inch) is a printing/scanning concept — dots per inch printed or scanned, used for physical output devices. PPI (Pixels Per Inch) is a digital image concept — pixels per inch, used for screen display and digital images. The units are the same but the domains differ: scanners use DPI to describe scan precision, monitors use PPI to describe screen clarity. Casual mixing doesn't cause serious misunderstanding, but in professional contexts they must be strictly distinguished.

Q2: What DPI is best for scanned files?

Depends on the use. For screen reading and email, 150DPI is sufficient — small size and clearly readable text; for print output, 300DPI ensures print quality; for long-term archiving, 600DPI preserves maximum detail but is larger; for OCR, 300DPI or above gives the highest recognition rate. General advice: 150-200DPI for ordinary documents, 300DPI for important contracts, 600DPI for historical archives.

Q3: Does higher DPI mean larger file size?

Yes — DPI and file size have a square relationship. Size is proportional to DPI squared — doubling DPI quadruples size. For example, an A4 page at 150DPI is about 0.8MB, at 300DPI about 3.2MB, at 600DPI about 12.8MB. So when choosing DPI, balance clarity and size — higher is not always better. SmartSlim can automatically recommend the best DPI by use and compress.

Q4: Does higher PPI mean a clearer screen?

Within a certain range, yes. Above 300PPI, the human eye can barely distinguish higher pixel density (Apple's Retina standard is 326PPI). But PPI is also affected by screen size, viewing distance, and content type. A phone screen (5-7 inches) at 300PPI is sufficient; a monitor (24-32 inches) at 100-150PPI is fine, because the viewing distance is greater. Blindly chasing high PPI increases GPU load and power consumption, with diminishing returns.

Approach	Scan DPI	Color Mode	Total Size	Clarity Assessment	Applicable Scenario
Approach A	150DPI	Grayscale	6.4MB	Clear for screen, slightly blurry for print	Email
Approach B	300DPI	Grayscale	25.6MB	Clear for both screen and print	Daily archive + print
Approach C	600DPI	Grayscale	102.4MB	Perfect detail, zoomable	Long-term archive

6. Summary

Although DPI and PPI share the same unit, they belong to two domains — physical printing and digital display. The core of scanned file compression is choosing the right DPI: 150DPI for screen reading, 300DPI for printing, 600DPI for archiving. Remember the square relationship "doubling DPI quadruples size," and you can find the best balance between clarity and file size.

Practical advice: determine the use first, then choose DPI — don't blindly chase high resolution. If you already have a high-DPI scan that needs shrinking, use SmartSlim for downsampling compression — built on its self-developed Rust compression engine with per-page intelligent dispatch, supporting 10 categories and 40+ formats including PDF/images/video/Office/OFD, with local compression and data staying on-prem.

Use Case	Recommended DPI	Color Mode	Per-Page Size (A4)	Notes
Screen reading/email	150DPI	Grayscale	0.8MB	Clearly readable text, small size
Daily printing	300DPI	Grayscale	3.2MB	Print quality meets office needs
High-quality printing	300DPI	Color	8.5MB	Recommended for color documents
Long-term archiving	600DPI	Grayscale	12.8MB	Maximum detail, for important files
OCR	300DPI	Grayscale	3.2MB	300DPI gives highest recognition (99%+)
Engineering drawings	400DPI	Grayscale	5.7MB	Line detail needs higher DPI
Photos/albums	600DPI	Color	34.0MB	Color photos need high DPI for color
Web publishing	96DPI	Color	0.9MB	Matches screen, fast loading

FAQ

Q: What is the difference between DPI and PPI?
A: DPI (Dots Per Inch) refers to the number of ink dots a printer produces per inch, affecting print quality. PPI (Pixels Per Inch) refers to the number of pixels per inch in a digital image, affecting screen display resolution. In practice, the terms are often used interchangeably, but they describe different domains: DPI for output (print), PPI for input (digital). For scanning, the scanner's DPI setting determines the pixel dimensions of the resulting digital image.

Q: What DPI should I use for scanning documents?
A: 150 DPI: Suitable for screen reading and email (small file, clear text). 300 DPI: Standard for printing and OCR (good quality, moderate file size). 600 DPI: Recommended for archival and important documents (highest quality, large file). A 100-page document at 150 DPI is about 30MB, at 300 DPI about 120MB, at 600 DPI about 480MB. SmartSlim can compress high-DPI scans to much smaller sizes while maintaining readability.

Q: How does DPI affect scanned file size?
A: File size increases with the square of DPI. Doubling DPI quadruples the pixel count and file size. For example, an A4 page at 150 DPI (1240x1754 pixels) is about 300KB as JPEG. At 300 DPI (2480x3508 pixels) it's about 1.2MB (4x larger). At 600 DPI (4960x7016 pixels) it's about 4.8MB (16x larger than 150 DPI). SmartSlim allows you to reduce DPI for non-critical documents while maintaining readability.

Q: Can I change DPI after scanning?
A: Yes, you can downsample (reduce) DPI after scanning, but you cannot increase it without quality loss. Downsampling from 600 DPI to 300 DPI preserves quality. Upsampling from 150 DPI to 300 DPI creates pixelated results. Best practice: scan at a higher DPI for archival, then create lower-DPI copies for distribution. SmartSlim supports batch DPI adjustment for scanned documents.

Summary

The key to what's the difference between dpi... lies in identifying the sources of bloat and handling them accordingly. Choose the right compression strategy based on your scenario, prioritizing the largest contributors. SmartSlim can handle all compression steps in one click.

Compression Task Queue Design: Celery + Redis Async Processing

Uglypear Data — Tue, 11 Aug 2026 04:44:16 +0000

Original URL: https://www.uglypear.com/en/blog/compression-task-queue-design.html

How to process compression tasks asynchronously? This article details Celery + Redis task queue design — task sharding, priority queues, retry mechanisms, and dead letter queues, with a 10,000-file async compression architecture plan and complete Celery configuration examples. Async task queues are essential for handling large-scale batch compression workloads.

1. Why Compression Tasks Need Async Queues

File compression is a typical CPU-intensive + IO-intensive task. Compressing a 100MB PDF may take 5–15 seconds — if handled with a synchronous interface, the HTTP connection hangs for a long time, and single-machine concurrency is very poor. Async queues decouple "submission" and "execution": the client submits a task and immediately gets a task_id, the Worker compresses in the background, and returns the result via callback or polling when done.

SmartSlim Network Edition uses a FastAPI + Celery + Redis + MinIO architecture, with 12 concurrent tasks running stably on a single machine. In Kubernetes deployments, HPA can auto-scale between 3–10 replicas. This architecture has supported multiple enterprise customers with daily volumes of tens of thousands of file compression requests.

Processing Method	Concurrency	Response Latency	Failure Handling	Applicable Scale
Synchronous processing	Poor (blocks HTTP connections)	5–60s	No retry, direct error	<10 files
Thread pool	Medium (limited by thread count)	1–5s	Manual implementation needed	10–100 files
Celery async queue	High (Worker horizontal scaling)	<200ms	Auto retry + dead letter queue	100–100000 files
Kubernetes + queue	Very high (HPA elastic scaling)	<100ms	Complete fault tolerance system	>100000 files

2. Celery Task Queue Architecture Explained

The Celery task queue consists of four core roles: Producer, Broker, Worker, and Backend. Understanding the responsibilities of these four layers is essential for correctly configuring a compression task queue.

Celery's configuration directly determines the queue's throughput and stability. The table below shows recommended configurations for compression scenarios, verified in SmartSlim's production environment.

Compression tasks have different priorities: user real-time compression requests need fast response, while scheduled archiving tasks can run slowly. Use Redis priority queues for differentiated scheduling — high-priority tasks are consumed by Workers first.

Component	Technology Choice	Responsibility	Key Configuration
Producer	FastAPI	Receives HTTP requests, constructs tasks, delivers to Broker	task.apply_async(queue=...)
Broker	Redis 7.x	Stores pending task messages, supports priority queues	broker_url, visibility_timeout
Worker	Celery 5.x	Consumes tasks, calls Rust compression engine to execute compression	concurrency, prefork pool
Backend	Redis	Stores task status and return results	result_backend, result_expires
Storage Layer	MinIO	Stores original files and compressed files	S3-compatible protocol, multipart upload

3. Practical Case: 10000 Files Async Compression

This is an enterprise data archiving scenario: 10000 historical documents (mixed PDF/Word/images, average 8MB per file, total about 80GB) need unified compression and archiving. Requirement: complete within 1 hour, compression ratio no less than 60%.

Solution design: Split into 100 subtasks by 100 files per shard, batch submit using Celery group, with 12 Workers consuming concurrently. Each subtask serially calls the Rust compression engine to compress 100 files.

Result: Completed 10000 file compression in 44 minutes, compression ratio 67.3%, 17 corrupted files automatically entered the dead letter queue for manual handling. The overall architecture was stable, with peak CPU utilization of 89%, peak memory usage of 4.2GB, and no OOM or task loss.

Compression task failures fall into two categories: temporary errors (IO timeout, insufficient memory, excessive concurrency) and deterministic errors (file corruption, unsupported format). Temporary errors have a high probability of success on retry, while deterministic errors are meaningless to retry. The table below provides retry and dead letter decision strategies.

Retry configuration uses Celery's autoretry_for and retry_backoff, with initial backoff of 60 seconds, maximum 600 seconds, and random jitter to avoid avalanches. Tasks in the dead letter queue are periodically scanned by an independent monitoring task, triggering WeChat Work/DingTalk alerts to notify operations for handling.

For the complete compression API calling method, refer to Compression API Guide: REST Interface Design.

Parameter	Recommended Value	Description
broker_url	redis://:password@redis:6379/0	Redis as message broker, independent DB to avoid conflicts
result_backend	redis://:password@redis:6379/1	Result storage uses independent DB, isolated from Broker
task_serializer	json	JSON serialization, cross-language compatible
result_serializer	json	Results also use JSON
accept_content	['json']	Only accept JSON, security hardening
timezone	Asia/Shanghai	Unified timezone
task_acks_late	True	ACK only after task completion, no task loss on crash
worker_prefetch_multiplier	1	Each Worker prefetches only 1 task, avoiding long task starvation
task_time_limit	600	Hard timeout 600s per task
task_soft_time_limit	540	Soft timeout 540s, triggers SoftTimeLimitExceeded
task_reject_on_worker_lost	True	Reject task on Worker abnormal exit, re-queue
result_expires	86400	Results auto-cleaned after 24 hours

4. Queue Configuration Recommendations for Different Scenarios

Different business scenarios have different requirements for throughput, latency, and reliability, requiring differentiated queue configurations. The table below provides recommended configurations for common scenarios.

A general principle: real-time scenarios use high-priority queues + small shards + fast retry, batch scenarios use normal queues + large shards + exponential backoff, and classified scenarios use strict auditing + small shards + multi-level retry. For the complete enterprise batch compression solution, refer to Enterprise Batch Compression Solution: 10000 File Processing in Practice.

Queue Name	Priority	Routing Rule	Typical Tasks
compression_high	9 (highest)	User real-time requests	Single file instant compression
compression_normal	5 (default)	Batch tasks	Batch compression 100–500 files
compression_low	1 (lowest)	Scheduled archiving	Nightly full archive compression
dlq_queue	— (dead letter)	Tasks that failed retry	Manual investigation or compensation handling

5. Frequently Asked Questions (FAQ)

Q1: How to implement async processing for Celery compression tasks?

Use Celery + Redis to build an async task queue: FastAPI receives requests and delivers tasks to the Redis Broker, Celery Workers consume tasks from the Broker and call the Rust compression engine to execute compression, writing results to Backend and MinIO storage. A single task.apply_async executes asynchronously, and status is polled via task.id. With 12 Workers concurrent on a single machine, 10000 files sharded into 100 batches can complete in 40 minutes.

Q2: How to auto-retry failed compression tasks?

Use Celery's autoretry_for parameter to configure auto-retry, setting max_retries=3, retry_backoff=True (exponential backoff, initial 60 seconds), retry_backoff_max=600 seconds, retry_jitter=True (random jitter to avoid avalanches). Tasks that still fail after 3 retries are automatically routed to the dead letter queue dlq_queue for manual or compensation task handling. It's recommended to retry temporary errors (IO timeout/insufficient memory) and send deterministic errors (file corruption/unsupported format) directly to dead letter.

Q3: How to shard 10000 files for batch compression?

Shard by 100 files per shard, totaling 100 subtasks. Batch submit using Celery group or chord, with 12 Workers consuming in parallel, each subtask serially compressing 100 files. Average compression time per file is 3 seconds, about 5 minutes per shard, and 100 shards in parallel complete in about 40 minutes overall. Too small a shard size (e.g., 1 per shard) has high scheduling overhead, too large (e.g., 1000 per shard) has high retry cost on failure — 100 is the empirical optimal value.

Q4: Which is more suitable for compression task queues — Celery or RQ?

Celery is recommended for compression tasks. Celery supports task sharding (group/chord), priority queues, scheduled tasks, task chains, and dead letter queues — complete functionality; RQ is lighter but lacks sharding and priority. Compression scenarios commonly require batch sharding, priority scheduling, and failure retry, all natively supported by Celery. Performance-wise, both are based on Redis with comparable throughput. SmartSlim Network Edition uses a FastAPI + Celery + Redis + MinIO architecture, running stably with 12 concurrent tasks on a single machine.

Metric	Parameter	Measured Value	Description
Total files	10000	—	Mixed formats, average 8MB each
Shard granularity	100 files/shard	100 shard subtasks	Balance scheduling overhead and retry cost
Worker concurrency	12	prefork mode	Single machine 12-core CPU
Single file compression time	—	Average 3.2s	Rust engine medium level
Single shard time	—	About 5.3 min	100 files serial
Overall time	—	About 44 min	100 shards/12 concurrent
Compression ratio	—	67.3%	80GB→26.2GB
Failures	—	17	File corruption, entered dead letter after retry

6. Summary

The standard solution for compression task async processing is the Celery + Redis task queue, with the core being the four-layer decoupling of Producer/Broker/Worker/Backend. For 10000-file batch compression, sharding by 100 files with 12 Workers concurrent, completion in about 40 minutes, compression ratio 60%–70%. Retry strategies should distinguish temporary errors (exponential backoff retry) from deterministic errors (direct to dead letter), combined with 6 monitoring metrics to ensure queue stability.

Remember three points: first, task_acks_late=True ensures no task loss on crash; second, worker_prefetch_multiplier=1 avoids long task starvation; third, the dead letter queue must be configured with monitoring alerts. Choose the right queue architecture and sharding strategy, and both throughput and stability of the compression service can reach a new level.

Error Type	Typical Exception	Handling Strategy	Retry Count	Final Destination
Temporary-IO	ConnectionError, TimeoutError	Exponential backoff retry	3 times	Success or dead letter
Temporary-Resource	MemoryError, OOMKilled	Extended backoff + downgrade	2 times	Success or dead letter
Deterministic-File	FileCorrupted, ParseError	No retry, direct dead letter	0 times	dlq_queue
Deterministic-Format	UnsupportedFormat	No retry, direct dead letter	0 times	dlq_queue
Deterministic-Permission	PermissionDenied	No retry, alert	0 times	dlq_queue + alert

Monitoring Metric	Collection Method	Alert Threshold	Action
Queue backlog	Redis LLEN	>500	Trigger Worker scaling
Task failure rate	Celery events	>5%	Investigate logs + pause submission
Dead letter queue length	Redis LLEN dlq	>10	WeChat Work alert
Worker alive count	Celery inspect	<10	Auto-restart Worker
Average task duration	Flower monitoring	>30s	Check large files + downgrade
CPU utilization	node_exporter	>95%	Throttle + scale

Scenario	Worker Count	Shard Granularity	Priority Queue	Retry Strategy
Personal instant compression	2	No sharding	high	Fast retry 3 times
Enterprise batch archiving	12	100 files/shard	normal/low	Exponential backoff 3 times
Government classified processing	4	50 files/shard	high	Strict retry + audit
E-commerce platform images	16	200 files/shard	normal	Fast retry 2 times
Nightly scheduled archiving	8	500 files/shard	low	Slow backoff 5 times
Real-time video transcoding	24	Single file	high	No retry, alert on failure

FAQ

Q: How to design a Celery-based compression task queue?
A: Architecture: 1) Celery workers — consume compression tasks from Redis/RabbitMQ. 2) Task definitions — each compression job is a Celery task with retry logic. 3) Result backend — store task results in Redis or database. 4) Monitoring — use Flower for real-time worker monitoring. SmartSlim provides a reference Celery configuration with task routing, rate limiting, and priority queues.

Q: How to implement task sharding for large batches?
A: Task sharding splits a large batch into smaller chunks for parallel processing: 1) Group tasks into chunks of 50-100 files. 2) Submit each chunk as a separate Celery task. 3) Use Celery groups to track completion of all chunks. 4) Implement a chord callback for post-processing. SmartSlim's sharding implementation scales linearly with worker count, processing 10,000 files in ~30 minutes with 8 workers.

Q: How to handle failed compression tasks?
A: Failure handling strategy: 1) Automatic retry — Celery's task.retry() with exponential backoff (retry 3 times, delays: 60s, 300s, 900s). 2) Max retries exceeded — move to dead letter queue. 3) Dead letter processing — log failure details, notify admin, store for manual review. 4) Partial success — complete remaining tasks, report failures in final summary. SmartSlim's Celery configuration includes comprehensive failure handling.

Q: What monitoring and alerting should be set up?
A: Monitoring stack: 1) Celery monitoring — Flower dashboard for worker status and task queues. 2) Application metrics — Prometheus metrics for task throughput, success rate, processing time. 3) Log aggregation — ELK stack (Elasticsearch, Logstash, Kibana) for compression logs. 4) Alerting — Grafana alerts for queue depth > 1000, failure rate > 5%, worker downtime. SmartSlim provides a complete monitoring configuration with pre-built Grafana dashboards.

Summary

The key to compression task queue design: celery... lies in identifying the sources of bloat and handling them accordingly. Choose the right compression strategy based on your scenario, prioritizing the largest contributors. SmartSlim can handle all compression steps in one click.

Top 10 Best PDF Compression Tools in 2026 (Including Free)

Uglypear Data — Mon, 10 Aug 2026 06:07:29 +0000

Top 10 Best PDF Compression Tools in 2026 (Including Free)

Original URL: https://www.uglypear.com/en/blog/best-pdf-compression-tools-2026.html

What are the best PDF compression tools in 2026? This article reviews 10 PDF compression tools — including SmartSlim, Adobe Acrobat, and ILovePDF — scoring them across 5 dimensions: compression ratio, quality retention, batch processing, security, and price. Whether you need desktop software, online service, or enterprise solution, this comprehensive comparison helps you make the right choice.

1. Overview of 10 PDF Compression Tools

PDF compression tools on the market fall into four categories: desktop software, online tools, enterprise solutions, and open-source CLI tools. We selected 10 representative products for cross-review, covering free and paid, local and online, personal and enterprise scenarios.

#	Tool Name	Type	Execution	Free?	Target User
1	SmartSlim	Desktop software	Local compression	Free + paid	Personal/Enterprise
2	Adobe Acrobat Pro	Desktop software	Local compression	Paid (subscription)	Professional
3	ILovePDF	Online tool	Cloud upload	Free (limited) + paid	Personal occasional
4	Smallpdf	Online tool	Cloud upload	Free (2/day) + paid	Personal occasional
5	Compress PDF (online)	Online tool	Cloud upload	Free	Personal occasional
6	Nitro PDF Pro	Desktop software	Local compression	Paid (one-time)	Enterprise
7	Foxit PDF Editor	Desktop software	Local compression	Paid (subscription)	Enterprise
8	Wondershare PDFelement	Desktop software	Local compression	Paid (one-time + sub)	SMB
9	PDF24 Creator	Desktop software	Local compression	Free	Personal
10	Ghostscript	CLI tool	Local compression	Free open source	Developer

2. 5-Dimension Score Comparison

We scored the 10 tools across 5 dimensions — compression ratio, quality retention, batch processing, security, and price (5-point scale) — each backed by test data.

Scoring basis: compression ratio based on tested ratio (SmartSlim 92.5% highest); quality retention based on text and image clarity after compression; batch processing based on multi-file simultaneous support; security based on local processing (online tools lose points for requiring file upload); price based on free quota and paid price.

Tool	Compression Ratio	Quality Retention	Batch Processing	Security	Price	Overall
SmartSlim	★★★★★	★★★★★	★★★★★	★★★★★	★★★★☆	24/25
Adobe Acrobat Pro	★★★★☆	★★★★★	★★★★☆	★★★★★	★★☆☆☆	20/25
ILovePDF	★★★☆☆	★★★☆☆	★★☆☆☆	★★☆☆☆	★★★★★	15/25
Smallpdf	★★★☆☆	★★★☆☆	★☆☆☆☆	★★☆☆☆	★★★☆☆	12/25
Compress PDF	★★☆☆☆	★★☆☆☆	★☆☆☆☆	★☆☆☆☆	★★★★★	9/25
Nitro PDF Pro	★★★★☆	★★★★☆	★★★★☆	★★★★★	★★★☆☆	20/25
Foxit PDF Editor	★★★★☆	★★★★☆	★★★★☆	★★★★★	★★★☆☆	20/25
Wondershare PDFelement	★★★☆☆	★★★★☆	★★★☆☆	★★★★☆	★★★☆☆	17/25
PDF24 Creator	★★★☆☆	★★★☆☆	★★★☆☆	★★★★★	★★★★★	18/25
Ghostscript	★★★★☆	★★★☆☆	★★★★☆	★★★★★	★★★★★	21/25

3. Test Data: Same PDF File Compression Comparison

We tested all 10 tools with the same test PDF (80MB, 120 pages, with HD images and scanned pages), comparing compressed size and duration.

Test conclusion: SmartSlim has the highest compression ratio (92.5%), 80MB down to 6MB; Ghostscript is second (86.9%) but requires CLI operation; Adobe Acrobat's 85% ratio preserves quality best. Online tools generally have lower ratios (65%–77%) and require file upload and waiting, with durations 3–5x local tools.

Platform compatibility and feature coverage comparison:

Tool	Original Size	Compressed	Ratio	Quality Assessment	Duration
SmartSlim	80.0MB	6.0MB	92.5%	Text clear, images slightly reduced	8 s
Adobe Acrobat Pro	80.0MB	12.0MB	85.0%	Text clear, images intact	15 s
ILovePDF (high compression)	80.0MB	18.5MB	76.9%	Visible image blur	45 s (incl. upload)
Smallpdf	80.0MB	22.0MB	72.5%	Obvious image blur	60 s (incl. upload)
Compress PDF	80.0MB	28.0MB	65.0%	Weak compression	30 s (incl. upload)
Nitro PDF Pro	80.0MB	13.5MB	83.1%	Good quality retention	18 s
Foxit PDF Editor	80.0MB	14.2MB	82.3%	Good quality retention	20 s
Wondershare PDFelement	80.0MB	16.8MB	79.0%	Medium quality	22 s
PDF24 Creator	80.0MB	20.5MB	74.4%	Medium quality	12 s
Ghostscript	80.0MB	10.5MB	86.9%	Text clear, image loss	6 s

4. Free vs Paid Comparison

Many users wonder whether free tools are sufficient. We compared each tool's free quota and paid price.

SmartSlim Free offers 10 times/day, 10MB per file — fully sufficient for personal daily office use. The annual paid plan is only 98 yuan (unlimited, 1GB per file), far better value than Adobe Acrobat's 1580 yuan/year.

Tool	Windows	macOS	Linux	Xinchuang Compatible	API/SDK	Batch Processing
SmartSlim	Yes	Yes	Yes	Yes (Kylin/UOS)	Yes (C ABI)	Yes (drag-drop/API)
Adobe Acrobat Pro	Yes	Yes	No	No	Limited (JS API)	Yes (limited)
ILovePDF	Web	Web	Web	No	Yes (REST API)	Paid edition
Smallpdf	Web	Web	Web	No	No	No
Nitro PDF Pro	Yes	Yes	No	No	No	Yes
Foxit PDF Editor	Yes	Yes	No	Partial	Limited	Yes
PDF24 Creator	Yes	No	No	No	No	Yes
Ghostscript	Yes	Yes	Yes	Yes	Yes (CLI)	Yes (script)

5. Scenario Recommendations: Different Needs, Different Tools

Different user scenarios emphasize different needs — the table below gives targeted recommendations.

Core principle for choosing a tool: classified documents must be compressed locally (online tools prohibited); enterprise batch uses API; personal daily use, the free edition suffices; developers choose SDK or CLI tools. For more selection dimensions, see File Compression Tool Selection Guide: 7 Evaluation Dimensions and Free Compression Software Review.

Tool	Free Quota	Paid Price	Free Edition Limits	Value Assessment
SmartSlim	Images 5MB / docs 10MB, 10/day	Monthly 9.9 yuan / annual 98 yuan	Single-file size limit	★★★★★ Free enough for daily use
Adobe Acrobat Pro	7-day trial	Annual sub ~1580 yuan	Unusable after trial	★★☆☆☆ Pricey
ILovePDF	2/day, 10MB per file	~50 yuan/month, ~400 yuan/year	Count and size limits	★★★☆☆ Enough occasionally
Smallpdf	2/day, 5MB per file	~75 yuan/month, ~600 yuan/year	Many limits	★★☆☆☆ Free too little
Nitro PDF Pro	14-day trial	One-time ~1200 yuan	Trial watermark	★★★☆☆ One-time payment
Foxit PDF Editor	14-day trial	Annual sub ~980 yuan	Trial watermark	★★★☆☆ Enterprise common
PDF24 Creator	Fully free	No payment	Unlimited	★★★★☆ Free top pick
Ghostscript	Fully free open source	No payment	Unlimited	★★★★☆ Technical user top pick

6. Frequently Asked Questions

Q1: Which PDF compression tool has the highest ratio?

SmartSlim has the highest compression ratio — tested 80MB scanned file compressed to 6MB, a 92.5% ratio. Based on its self-developed Rust compression engine, it automatically identifies PDF content type (text/image/scanned) and dispatches the optimal strategy — image downsampling + format conversion, font subsetting, redundancy cleanup — three steps stacked for deep compression. Adobe Acrobat is second (~85% ratio), Ghostscript third (~80%).

Q2: What free PDF compression tools do you recommend?

Three free recommendations: SmartSlim Free (images 5MB / docs 10MB, 10/day, local compression with privacy security); ILovePDF online (2 free/day, requires file upload); Ghostscript CLI (fully free open source, but requires technical background). For occasional personal use, SmartSlim Free is recommended — data stays on-prem, most secure; for frequent use, upgrade to a paid edition.

Q3: Online PDF compression vs local compression — which is better?

Documents involving sensitive information must use local compression. Online compression requires uploading files to third-party servers, posing data leakage risk — unsuitable for contracts, financial reports, and other classified documents. Local compression (e.g., SmartSlim Desktop) performs all processing on the local machine — data never leaves. Online compression's advantage is no installation and cross-platform, suited for temporary compression of non-sensitive documents.

Q4: What tool for enterprise batch PDF compression?

For enterprises, SmartSlim Server or Network edition is recommended. The Server edition supports single-machine deployment, with API-based automated batch compression, 12 concurrent tasks, daily processing of tens of thousands of documents. The Network edition supports Docker/K8s distributed deployment, 10GB per file, suited for large-scale document archiving. Both support Xinchuang environments (Kylin OS/DM database), with privatized data staying on-prem.

Use Case	Core Need	Recommended Tool	Rationale
Personal daily office	Free, easy, local compression	SmartSlim Free	10 free/day, drag-and-drop, data stays on-prem
Classified document compression	Absolute security, local processing	SmartSlim Desktop	Local compression, 5-level security
Enterprise batch archiving	API automation, high concurrency	SmartSlim Server	12 concurrent, API integration, 10k/day
Professional PDF editing	Edit + compress integrated	Adobe Acrobat Pro	Most feature-complete, edit + compress in one
Temporary one-time compression	No install, fast	ILovePDF online	No install, drag-drop upload
Developer integration	CLI/SDK	Ghostscript/SmartSlim SDK	Ghostscript free open source, SDK easier to integrate
Xinchuang environment deployment	Domestic compatibility	SmartSlim Server	Supports Kylin OS/DM database/Phytium CPU
Zero budget	Fully free	PDF24 Creator	Fully free, unlimited, local compression

7. Summary

The top PDF compression tool in 2026 is SmartSlim — 92.5% compression ratio, the highest in the industry; local compression with data staying on-prem; the free edition meets daily needs; annual paid only 98 yuan, outstanding value. Adobe Acrobat suits professional PDF editing needs; ILovePDF suits temporary one-time compression; Ghostscript suits developer CLI scenarios. Three principles for choosing a tool: classified documents use local; batch processing uses API; personal daily use uses the free edition.

SmartSlim is built on the Rust compression engine, supports 10 categories and 40+ formats including PDF/images/video/Office/OFD, and offers four forms — Desktop/Server/Network/SDK — covering all scenarios from personal to enterprise.

FAQ

Q: What is the best PDF compression tool in 2026?
A: SmartSlim is the top recommendation for 2026 — it offers the best compression ratio (80-95% reduction), maintains high quality, supports batch processing, runs locally (no privacy concerns), and is available in free and paid versions. For enterprise users, SmartSlim Enterprise provides SDK integration, API access, and deployment options. Adobe Acrobat Pro is good for individual use but expensive. Online tools like ILovePDF are convenient but have file size limits and privacy risks.

Q: Are free PDF compression tools effective?
A: Yes, several free tools are effective: SmartSlim Free (best compression ratio, local processing), Smallpdf (convenient, 2 files/day free), and PDF24 Tools (feature-rich, desktop version). Free tools can typically reduce PDF size by 50-80%. For sensitive documents, always use local processing tools. SmartSlim Free offers unlimited compression without file size limits, making it the best free option for most users.

Q: How do online PDF compression tools compare to desktop software?
A: Online tools: convenient, no installation, accessible from any device. Limitations: file size limits (typically 50-100MB), privacy concerns (files uploaded to third-party servers), internet dependency, and slower processing. Desktop software: no file size limits, 100% local processing (no privacy risks), faster processing, batch processing, and more advanced features. SmartSlim offers both — desktop app for daily use and cloud API for automated workflows.

Q: What security considerations are important for PDF compression?
A: Key security considerations: 1) Never upload sensitive documents to unknown online tools. 2) Use local processing for confidential documents (contracts, financial reports, legal documents). 3) Verify that the tool properly handles encrypted PDFs. 4) Check if the tool modifies digital signatures. 5) Ensure the tool maintains PDF/A compliance when required. SmartSlim processes all files locally without any network transmission, ensuring complete data security.

Summary

The key to top 10 best pdf compression... lies in identifying the sources of bloat and handling them accordingly. Choose the right compression strategy based on your scenario, prioritizing the largest contributors. SmartSlim can handle all compression steps in one click.

AVIF Format Guide: The Next-Generation Image Compression STD

Uglypear Data — Thu, 06 Aug 2026 13:44:48 +0000

Original URL: https://www.uglypear.com/en/blog/avif-format-guide.html

What is AVIF? This article provides a complete guide to the AVIF image format — based on AV1 encoding, 50% smaller than JPEG and 20% smaller than WebP, with a 4-format comparison test of AVIF vs WebP vs JPEG vs PNG and a browser compatibility migration guide. AVIF represents the next generation of image compression, offering superior compression efficiency for modern web applications.

1. What is AVIF

AVIF (AV1 Image File Format) is an image format released by the Alliance for Open Media (AOMedia) in 2019, based on the intra-frame coding technology of the AV1 video encoder. AV1 itself is an open-source, royalty-free coding standard designed for next-generation video (8K/120fps), and its intra-frame coding tools are equally applicable to static image compression. AVIF essentially packages AV1's intra-frame coding into a standalone image container format.

The core advancements of AVIF over the previous-generation WebP include: support for 12-bit color depth and HDR (High Dynamic Range), while WebP only supports 8-bit SDR; higher compression ratios (10%–30% smaller at equivalent quality); and support for richer coding tools (directional prediction, more flexible block partitioning). The tradeoff is slower encoding speed — AVIF encoding takes about 3–5 times longer than WebP.

Feature	AVIF	WebP	JPEG	PNG
Release Year	2019	2010	1992	1996
Underlying Codec	AV1 Intra	VP8 Intra	DCT	DEFLATE
Lossy Compression	Yes	Yes	Yes	No
Lossless Compression	Yes	Yes	No	Yes
Alpha Channel	Yes	Yes	No	Yes
HDR/Wide Color Gamut	Yes (12-bit)	No (8-bit)	No (8-bit)	No (8/16-bit)
Animation Support	Yes	Yes	No	APNG extension
Licensing	Free & Open Source	Free & Open Source	Free	Free

2. AVIF Technical Principles Explained

AVIF's compression pipeline consists of three core stages: partition prediction, transform coding, and loop filtering. Each stage inherits the advanced tools from AV1 video coding, which is the fundamental reason AVIF's compression efficiency far exceeds that of JPEG.

AV1 divides the image into coding blocks (up to 128×128), and each block can be recursively subdivided into smaller sub-blocks (down to 4×4), adaptively selecting partitioning based on image content. Predictive coding uses already-reconstructed pixels to predict the current block, encoding only the residual (the difference between actual and predicted values).

The residual after prediction is transformed from the spatial domain to the frequency domain using transform coding, and high-frequency coefficients are quantized to achieve compression. AV1 supports multiple transform types, offering more flexibility than JPEG's single DCT.

The reconstructed image after quantization produces blocking artifacts and ringing artifacts, which AV1 repairs using multi-level loop filtering. This is the key reason AVIF's image quality at low bitrates is significantly better than JPEG.

For a comprehensive overview of image compression, refer to the Image Compression Guide: JPG/PNG/WebP Format Comparison.

Prediction Type	Principle	Applicable Regions	vs JPEG
Intra Prediction	Predicts using adjacent encoded pixels	Smooth areas/gradients	JPEG has no prediction, encodes everything
Directional Prediction	56 directional angle predictions	Edges/texture directions	JPEG has no directional prediction
Palette Prediction	Uses color palette indexing	Text/icons/animation	JPEG lacks this tool
Block Copy	Copies other regions within the image	Repeated textures	JPEG lacks this tool

3. Practical Case: Four-Format Comparison Test

We compare the file size, image quality, and encoding time of AVIF, WebP, JPEG, and PNG using the same test image. Test sample: 1920×1080 natural landscape photo, 24-bit true color, original BMP approximately 5.93MB.

Document characteristics: 1920×1080, 24-bit color, containing sky gradients, foliage textures, and building edges, original BMP 5.93MB.

File size and quality comparison (target SSIM 0.92):

Results: At equivalent quality (SSIM 0.92+), AVIF is only 210KB — 50% smaller than JPEG and 29.5% smaller than WebP. PNG has the highest lossless quality but the largest file size (4.82MB), making it unsuitable for photo content. AVIF performs particularly well in foliage texture and sky gradient areas — JPEG shows visible color blocking at low bitrates, while AVIF has almost no visible artifacts.

AVIF's encoding time is 15 times that of JPEG and 4 times that of WebP, because AV1's partition prediction and transform selection require extensive rate-distortion optimization calculations. However, AVIF's decoding speed is only 2 times slower than JPEG — minimal impact on the browsing experience. It is recommended to pre-generate AVIF images on the server side (encode at build time) rather than encoding in real time.

Transform Type	Block Size	Applicable Scenarios	Advantage
DCT	4×4 to 64×64	General purpose	Classic transform, energy compaction
ADST (Asymmetric Discrete Sine Transform)	4×4 to 32×32	Directional edges	Better for edge residuals
Identity Transform	4×4 to 32×32	Lossless/low-loss	Skips transform, direct quantization
Recursive Transform	Combined	Complex textures	Two-level transform improves energy compaction

4. Browser Compatibility and Migration Strategy

AVIF's browser compatibility has essentially matured after 2023, but fallback solutions are still needed for older browsers. The table below lists the support status of major browsers.

Global browser support for AVIF is approximately 96%, with only IE and very old browser versions lacking support. The recommended migration strategy uses the HTML5 picture tag for progressive enhancement.

Recommended picture tag syntax:

<picture>
<source srcset="photo.avif" type="image/avif">
<source srcset="photo.webp" type="image/webp">
<img src="photo.jpg" alt="Landscape photo">
</picture>

The browser selects the first supported format by priority: if AVIF is supported, it uses AVIF (smallest); otherwise, it tries WebP, and finally falls back to JPEG. This approach enjoys AVIF's compression advantages while ensuring normal display on older browsers. For a complete web image optimization solution, refer to Web Image Optimization: 50% Performance Improvement in Practice.

Filter Stage	Function	Processing Order	Effect
CDEF (Constrained Directional Enhancement Filter)	Removes block boundary artifacts	Stage 1	Smooths block boundaries
LR (Loop Restoration Filter)	Wiener filter/Self-guided filter	Stage 2	Restores detail textures
Super-resolution Reconstruction	Reconstructs from low-resolution frames	Stage 3	Enhances high-frequency details

5. Format Selection Guide for Different Scenarios

While AVIF is excellent, it's not suitable for all scenarios. Format selection for different scenarios requires comprehensive consideration of compression ratio, compatibility, encoding speed, and feature requirements.

A general principle: for static content pre-generation, choose AVIF (optimal size); for real-time generation, choose JPEG/WebP (speed priority); for lossless requirements, choose PNG; for HDR content, choose AVIF (the only option). When using SmartSlim for batch conversion, you can automatically generate multiple formats (AVIF+WebP+JPEG), covering all compatibility scenarios in one pass.

Format	Parameters	File Size	vs Original	SSIM Quality	vs JPEG
PNG (Lossless)	DEFLATE Maximum	4.82MB	-18.7%	1.000	+938%
JPEG	Quality 85	420KB	-92.9%	0.921	Baseline
WebP	Quality 80	298KB	-95.0%	0.923	-29.0%
AVIF	Quality 55 (CRF)	210KB	-96.5%	0.925	-50.0%

6. Frequently Asked Questions (FAQ)

Q1: What is the AVIF format?

AVIF (AV1 Image File Format) is an image format based on AV1 video coding, released by the Alliance for Open Media (AOMedia) in 2019. AVIF leverages AV1's intra-frame coding technology to compress static images, achieving 50% smaller files than JPEG and 20% smaller than WebP, while supporting lossless compression, HDR, wide color gamut, and alpha channels. AVIF is an open-source, royalty-free next-generation image compression standard, natively supported by major browsers including Chrome and Firefox.

Q2: How much smaller is AVIF compared to JPEG?

At equivalent quality (SSIM 0.9 or above), AVIF is approximately 50% smaller than JPEG. For example, a 1920×1080 photo: JPEG at quality 85 is about 420KB, while AVIF at equivalent quality is about 210KB. Compared to WebP, AVIF is about 20% smaller at medium quality, with even greater advantages at high quality (30%+ smaller). AVIF's compression advantage comes from AV1's more advanced predictive coding and transform tools, performing particularly well in areas with complex textures.

Q3: How is AVIF's browser compatibility?

As of 2026, AVIF is supported by major browsers: Chrome 85+ (September 2020), Firefox 93+ (October 2021), Edge 92+ (2021), and Safari 16.4+ (March 2023). Global browser support rate is approximately 96%. Older browsers that don't support AVIF can be provided with JPEG/WebP fallback via the picture tag. It's recommended to use

<picture><source srcset='img.avif' type='image/avif'><img src='img.jpg'></picture>

for progressive enhancement.

Q4: Which is better, AVIF or WebP?

Overall, AVIF is superior to WebP. Compression ratio: AVIF is 10%–30% smaller than WebP; Features: AVIF supports HDR and 12-bit color depth, while WebP only supports 8-bit SDR; Encoding speed: WebP is faster than AVIF (AVIF encoding is slower); Compatibility: WebP support rate is about 98%, AVIF about 96%. If you pursue ultimate compression ratio and HDR support, choose AVIF; if you prioritize encoding speed and broad compatibility, choose WebP. It's recommended to provide both formats simultaneously using the picture tag.

Format	Encoding Time	Decoding Time	Speed Rating	Applicable Scenarios
JPEG	18ms	8ms	Extremely fast	Real-time encoding
WebP	65ms	15ms	Fast	Web real-time
AVIF	280ms	22ms	Slow (encoding)/Fast (decoding)	Pre-generation
PNG	120ms	12ms	Medium	Lossless scenarios

7. Summary

AVIF is the next-generation image compression standard, based on AV1 intra-frame coding, 50% smaller than JPEG and 20% smaller than WebP, while also supporting HDR, 12-bit color depth, and alpha channels. Its technical leadership comes from partition prediction (56 directional predictions), multiple transform coding types (DCT/ADST/Identity Transform), and multi-level loop filtering (CDEF+LR). The tradeoff is slower encoding speed (about 4 times that of WebP), making it suitable for pre-generation scenarios rather than real-time encoding.

The recommended migration strategy is the picture tag multi-format approach: AVIF first, WebP second, JPEG fallback, balancing compression ratio and 100% compatibility. Remember three points: first, AVIF encoding is slow but decoding is fast, suitable for server-side pre-generation; second, HDR content can only use AVIF (WebP/JPEG don't support it); third, use the picture tag to provide all three formats simultaneously, letting the browser automatically select the optimal one.

Browser	Supported Version	Release Date	Global Share	Support Status
Chrome	85+	September 2020	65%	Native support
Safari	16.4+	March 2023	18%	Native support
Edge	92+	2021	5%	Native support
Firefox	93+	October 2021	3%	Native support
Opera	71+	2020	2%	Native support
IE	Not supported	—	<1%	Fallback required

Migration Approach	Implementation	Compatibility	Maintenance Cost	Recommendation
picture tag	Multi-format source + img fallback	100%	Low	★★★★★
AVIF only	Use avif directly	96%	Very low	★★★☆☆
Accept header negotiation	Server returns based on Accept	100%	Medium	★★★★☆
JPEG only	No migration	100%	Very low	★☆☆☆☆

Scenario	Recommended Format	Reason	Size Reference
E-commerce product images	AVIF (primary)+JPEG (fallback)	Pursues ultimate size, improves loading speed	JPEG 420KB→AVIF 210KB
News content	WebP (primary)+JPEG (fallback)	Fast encoding, good compatibility	JPEG 420KB→WebP 298KB
HDR photography	AVIF	Only mainstream format supporting 12-bit HDR	Lossless HDR only available with AVIF
UI icons/screenshots	PNG	Lossless requirement, sharp edges	Size not a concern
Social media	AVIF or WebP	Small size, fast sharing	Both work
Email embedding	JPEG	Email client compatibility priority	AVIF not recommended (poor compatibility)
Video thumbnails	AVIF	Natural pairing with AV1 video	50% smaller than JPEG
Real-time generation (CAPTCHA, etc.)	JPEG	Encoding speed priority	18ms vs AVIF 280ms

FAQ

Q: What is AVIF and why is it better than JPEG?
A: AVIF (AV1 Image File Format) is a next-generation image format based on the AV1 video codec. It achieves 50% smaller files than JPEG at equivalent visual quality, and 20% smaller than WebP. It supports HDR (High Dynamic Range), wide color gamut (BT.2020), 10/12-bit color depth, transparency, and lossless compression. AVIF is the successor to JPEG, offering dramatically better compression efficiency for modern web and mobile applications.

Q: How does AVIF compare to WebP?
A: AVIF is 20-30% more efficient than WebP at equivalent quality. For a typical photo: WebP at quality 80 produces a 40KB file. AVIF at equivalent quality produces a 30-32KB file. AVIF also supports HDR and 10-bit color, which WebP does not. However, WebP has broader browser support (95%+ vs 90%+ for AVIF). AVIF encoding is also slower than WebP. For new projects, AVIF is recommended for forward compatibility.

Q: What is the browser compatibility of AVIF?
A: AVIF is supported by: Chrome 85+ (2020), Firefox 93+ (2021), Opera 71+ (2020), Safari 16.1+ (2022). Edge 121+ (2024). Overall support: ~90% of global browser users. For older browsers, use the element with fallback to WebP or JPEG. SmartSlim automatically generates AVIF with fallbacks, ensuring compatibility across all browsers while delivering AVIF to modern browsers.

Q: How to migrate existing images to AVIF?
A: Migration steps: 1) Use SmartSlim or avifenc to convert JPEG/PNG to AVIF. 2) Update HTML with elements:

<picture><source srcset="image.avif" type="image/avif"><source srcset="image.webp" type="image/webp"><img src="image.jpg" alt=""></picture>

. 3) Test on all browsers. 4) For CDN delivery, enable automatic AVIF conversion. SmartSlim's batch conversion tool handles the entire migration process, generating AVIF + WebP + JPEG variants with responsive markup.

Summary

The key to avif format guide: the next-generation... lies in identifying the sources of bloat and handling them accordingly. Choose the right compression strategy based on your scenario, prioritizing the largest contributors. SmartSlim can handle all compression steps in one click.

Word Document Too Large? 3 Compression Methods Tested

Uglypear Data — Wed, 05 Aug 2026 07:21:20 +0000

Bottom line first: Word documents become too large mainly because embedded images and embedded fonts account for the vast majority of their size. The solution is a three-step approach: compress embedded images (downsample DPI + convert to JPEG), apply font subsetting (keep only the characters actually used), and save as .docx to clear redundancy. A 50MB tender document packed with high-resolution images can be compressed to under 5MB with this workflow, with virtually no loss of text or chart clarity. Below, we start from what causes document bloat, compare 3 tested methods, and include a complete case study.

If you're not yet familiar with the overall concept of file compression, we recommend reading the Complete Guide to File Compression first.

1. Why Are Word Documents So Large?

A Word document (.docx/.doc) is essentially a container that packages text, images, fonts, styles, revision history, and more together. Size bloat almost always comes from non-text content. Understanding the sources of size is the key to targeted compression.

Size Source	Typical Share	Cause	Compression Potential
Embedded images	60%–85%	High-res originals pasted directly, uncompressed; multiple images stacked	★★★★★
Embedded fonts	10%–25%	Full Chinese font files embedded (10–20MB each)	★★★★☆
Revisions and comments	3%–10%	Revision history and comments left from multi-user collaboration	★★★☆☆
Document redundancy	2%–8%	Redundant XML and obsolete styles from repeated save-as operations	★★☆☆☆
Pure text content	1%–5%	The body text itself	★☆☆☆☆

As the table above shows, embedded images and embedded fonts together account for over 80%, making them the main battleground for compression. A 50MB tender document might contain 40MB of images and 8MB of fonts, with less than 2MB of pure text — once you compress the images and fonts, the document naturally shrinks.

2. Three Compression Methods Tested and Compared

Based on the size sources of Word documents, there are 3 mainstream compression methods. Each method differs in principle, applicable scenarios, and effect. The table below gives an overall comparison first, followed by detailed explanations.

Method	Principle	Applicable Scenario	Expected Compression Rate	Clarity Impact
Embedded image compression	Downsample DPI + format conversion	Image-heavy documents (tenders/reports)	70%–90%	Virtually no difference on screen
Font subsetting	Keep only used characters	Documents with embedded Chinese fonts	80%–95%	No impact
Save-as optimization	Clear redundancy + format conversion	Old documents / repeatedly edited files	10%–30%	No impact

1. Embedded Image Compression (Highest Return)

Embedded images are the number one source of Word document size. The compression method has two steps: downsampling DPI and format conversion. DPI (dots per inch) determines image pixel density — screen reading only needs 96–150DPI, yet many documents paste in 300DPI or even 600DPI originals. Format conversion means turning lossless formats (PNG/BMP) into lossy formats (JPEG), drastically reducing size with virtually no perceptible difference to the naked eye.

In practice: open the Word document with SmartSlim, and the engine automatically identifies the embedded image types and assigns compression strategies — scanned documents are downsampled to 150DPI and converted to grayscale JPEG, photos are downsampled to 200DPI and converted to color JPEG at quality 80, and wireframe/screenshots stay as PNG but undergo PNG optimization. This type-based dispatching works better than uniform parameters — scanned documents compress by 90% while screenshots remain virtually untouched.

Image Type	Original Parameters	Compression Parameters	Size Reduction
High-res photo	300DPI, PNG, 8MB	200DPI, JPEG q80, 0.8MB	90%
Scanned document	600DPI, BMP, 15MB	150DPI, JPEG q72 grayscale, 0.5MB	97%
Wireframe/screenshot	96DPI, PNG, 2MB	96DPI, PNG optimized, 1.2MB	40%
Chart/flowchart	200DPI, PNG, 3MB	150DPI, JPEG q85, 0.6MB	80%

For more detailed principles on image compression, see the Image Compression Guide: JPG/PNG/WebP Format Comparison.

2. Font Subsetting (Essential for Chinese Documents)

Many Word documents embed complete font files to ensure consistent display across devices. A complete Chinese font (such as Source Han Serif) easily reaches 10–20MB, and embedding both Regular and Bold weights starts at 30MB. But the document may actually use only a thousand or two distinct characters — font subsetting keeps only the characters that are used and discards the rest.

The size after font subsetting depends on character usage. A 50-page tender typically uses 800–1500 distinct characters; the full Source Han Serif file is 18MB, but after subsetting it drops to just 0.3–0.8MB, a reduction of over 95%. This operation has zero impact on display — the retained characters are exactly the same as before; it simply removes the characters that weren't used.

Font	Full File	After Subsetting (1200 characters)	Reduction
Source Han Serif Regular	18.2MB	0.42MB	97.7%
Source Han Sans Regular	16.8MB	0.38MB	97.7%
Microsoft YaHei Regular	15.6MB	0.35MB	97.8%
Arial Regular	0.8MB	0.06MB	92.5%

3. Save-As Optimization (Essential Finishing Step)

After a Word document has been edited, revised, and saved multiple times, it accumulates redundant XML, obsolete style definitions, old revision history, and other junk data. Saving a .doc as .docx is itself an effective cleanup — .docx is based on the Open XML zip compression structure, which is typically 30%–50% smaller than the .doc binary format. When saving, Word rebuilds the document structure and automatically clears some redundancy.

Going further, when processed by SmartSlim, the Rust compression engine unpacks the docx zip archive, cleans up obsolete resources (unreferenced images, duplicate style definitions, old revision records), and repackages the compressed file. This step typically squeezes out another 10%–15% of size, with no impact on display.

3. Case Study: Compressing a 50MB Tender to 4.6MB

This is a construction engineering tender in .docx format, with an original size of 50.3MB. The document contains 48 pages of body text, 32 high-resolution site photos (1–2MB each), 8 CAD-exported drawings, and embedded Source Han Serif Regular+Bold. It needs to be sent via WeCom (20MB limit), with a target of under 5MB.

Document characteristics: 48 pages A4, 32 photos at 300DPI PNG (42MB total), 8 CAD drawings at 200DPI PNG (6MB total), embedded fonts 28MB, with extensive revision history from 5 rounds of edits.

Execution parameters and size changes:

Step	Operation	Key Parameters	Size Change
1	Photo compression	300DPI→200DPI, PNG→JPEG q80	50.3→12.1MB
2	Drawing compression	200DPI→150DPI, PNG→JPEG q85	12.1→8.3MB
3	Font subsetting	Keep 1342 used characters	8.3→5.1MB
4	Clean redundancy + repackage	Clear revision history + obsolete styles	5.1→4.6MB

Result: 4.6MB, a compression ratio of about 10.9:1, well below the WeCom 20MB limit. The photos show virtually no perceptible difference on screen, the CAD drawing lines remain clearly distinguishable, the text layout is unchanged, and the fonts display normally. The entire process was completed locally — the tender never passed through any third-party server, meeting the confidentiality requirements for business documents.

4. Compression Recommendations for Different Scenarios

Different types of Word documents have different size sources and compression priorities. The table below offers recommendations for common scenarios.

Document Type	Typical Size	Main Size Source	Recommended Strategy
Tender / project report	30–80MB	High-res site photos	Downsample photos + convert to JPEG, font subsetting
Academic paper	5–20MB	Charts + embedded fonts	PNG optimization for charts, font subsetting
Product manual	10–50MB	Product images + layout styles	Tiered image compression, clean redundant styles
Contract / agreement	2–10MB	Scanned signature pages	Downsample scans + grayscale JPEG
Meeting minutes	1–5MB	Mostly plain text	Save as .docx, clean revision history
Training materials	20–100MB	Screenshots + animated screenshots	PNG optimization for screenshots, convert large images to JPEG

A general principle: for image-heavy documents, compress images first (highest return); for documents with embedded fonts, font subsetting is a must; for old documents, save as .docx first to clear redundancy. With all three steps combined, the vast majority of Word documents can be compressed to 10%–20% of their original size.

If you need to compare the effectiveness of different compression tools, see the File Compression Tool Selection Guide: 7 Dimensions Evaluated.

5. FAQ

Q1: How do I compress a Word document to under 5MB?

First use a compression tool to extract embedded images and downsample them to 150DPI, convert to JPEG at quality 80, then apply font subsetting (keep only used characters), and finally save as .docx to clear redundancy. A 50MB tender with lots of high-res images can be compressed to under 5MB in three steps, with virtually no loss of text or chart clarity. The key is to prioritize image compression (the largest share), then do font subsetting (highly effective for Chinese documents), and finish with redundancy cleanup.

Q2: Which is smaller, .docx or .doc?

.docx is typically 30%–50% smaller than .doc. .docx is based on the Open XML zip compression structure and is compressed natively; .doc is a binary format that does not compress embedded resources. Saving the same document as .docx noticeably reduces the size, so we recommend using .docx uniformly. In addition, .docx is an open standard with better compatibility, and various compression tools can parse and process it.

Q3: What if images become blurry after Word compression?

Image blur is usually caused by overly aggressive downsampling. 150DPI is sufficient for screen reading, while printing requires 200–300DPI. JPEG quality should be set to 80 or above; below 70, noticeable color blocking appears. If the document contains wireframes or screenshots, use PNG instead of JPEG to avoid edge artifacts. SmartSlim automatically identifies image types and dispatches strategies — photos convert to JPEG, screenshots stay as PNG — avoiding the blur caused by a one-size-fits-all approach.

Q4: Is there a way to batch compress Word documents?

You can use SmartSlim desktop edition for batch drag-and-drop processing, supporting 5 Word formats including .docx/.doc/.docm/.odt/.rtf, and batch compress hundreds of files at once. Enterprise users can use the server edition API for automated batch processing, with parallel compression via a Celery task queue — 12 concurrent tasks per machine and a daily throughput of tens of thousands of documents.

Summary

Word documents become too large because embedded images and embedded fonts account for over 80% of the size — the solution is targeted compression. The logic of the 3 methods is clear: image compression offers the highest return (downsample DPI + convert to JPEG, 70%–90% reduction), font subsetting is highly effective for Chinese documents (keep only used characters, 80%–95% reduction), and save-as optimization finishes the job by clearing redundancy (10%–30% reduction). With all three steps combined, a 50MB tender can be reliably compressed to under 5MB.

Remember three points: first, diagnose the size source before acting — compress images first if there are many, and do font subsetting if fonts are large; second, dispatch strategies by image type — convert photos to JPEG and keep screenshots as PNG, avoiding a one-size-fits-all approach; third, use local compression for sensitive documents and don't upload them to online tools. With the right tools and methods, slimming down Word documents is actually quite simple.

About SmartSlim

SmartSlim is the core product of Guangzhou Uglypear Data Technology Co., Ltd. Built on a self-developed Rust compression engine, it provides efficient and secure file compression solutions for individual and enterprise users. It supports 10 major categories and 40+ formats, including PDF, images, video, Office (Word/Excel/PPT), and OFD, covering mainstream document, image, and audio-video file types.

Core advantages:

Local compression, data never leaves your domain: All compression is performed locally, so sensitive files never need to be uploaded to third-party servers, meeting the confidentiality and compliance requirements of business, government, and financial scenarios.
Type-based compression strategy dispatching: The engine automatically identifies file types and matches optimal compression parameters — photos convert to JPEG, screenshots stay as PNG, fonts undergo subsetting — avoiding the quality loss caused by a one-size-fits-all approach.
Batch processing and automation: The desktop edition supports drag-and-drop batch compression, processing hundreds of files at once; the server edition provides an API for automated batch processing via a Celery task queue, with 12 concurrent tasks per machine and a daily throughput of tens of thousands of documents.
Multi-form product matrix: Offers desktop, server, network editions, and a Rust compression SDK, adapting to personal office, team collaboration, and enterprise integration scenarios.

Try SmartSlim now and make file compression simpler and more secure:

Free desktop download: https://www.uglypear.com/en/download.html
Enterprise solution inquiry: https://www.uglypear.com/en/contact.html
View SDK documentation: https://www.uglypear.com/en/products/sdk.html

How to Evaluate File Compression Tools: 7-Dimension Framework

Uglypear Data — Tue, 04 Aug 2026 03:26:29 +0000

The Selection Dilemma: Too Many Tools, Too Few Criteria

There are hundreds of file compression tools on the market. From command-line utilities like gzip, xz, and 7-Zip, to graphical applications like WinRAR and Bandizip, to various online compression services and specialized transcoding tools, each one claims to be "faster, smaller, better." But when you face a concrete need—halving the size of a batch of PDFs, or transcoding hundreds of videos to save storage—you discover that no single tool is optimal in every scenario.

The root cause is that a compression tool's capabilities are multidimensional. Compression ratio and speed are often at odds, quality retention and size control require trade-offs, and local processing versus cloud services each have their own costs. If you focus solely on "compression ratio," you may end up with a tool that performs poorly in real-world use.

This article proposes a 7-dimension evaluation framework to help you structurally assess any compression tool. Whether you are evaluating an open-source CLI, a commercial GUI, or a SaaS service, you can use the same yardstick and make a selection decision that fits your actual scenario.

Prerequisite: Archive vs. Content Compression

Before diving into the 7-dimension evaluation, one fundamental distinction must be clarified: compression tools fall into two major categories, with completely different working principles, applicable scenarios, and evaluation priorities.

Archive compression tools (e.g., 7-Zip, gzip, Zstandard): Package files at the container level and eliminate redundancy. Lossless and reversible, but largely ineffective on already-compressed files.
Content compression tools (e.g., FFmpeg, ImageMagick, Ghostscript): Dive into the internal structure of files for re-encoding. Mostly lossy and irreversible, but can dramatically reduce file size at the source.

This distinction is crucial—stuffing a 50MB PDF into a 7-Zip archive might only shrink it to 49MB, while using Ghostscript to downsample internal images can compress it to 6.8MB. Their "compression ratios" are not on the same level at all, and direct horizontal comparison can be seriously misleading.

For a detailed explanation of the essential difference between the two types of compression, process comparison, and real-world examples, see Why Zipping Files Doesn't Actually Compress Them. The 7-dimension framework in this article applies to both types of tools, but when comparing specific tools below, their category will be clearly labeled to avoid confusion from cross-category comparison.

Framework Overview

The flowchart below shows the relationships among the 7 evaluation dimensions and how they map to different usage scenarios:

Dimension 1: Compression Ratio

Evaluation Criteria

Compression ratio is the most intuitive metric, usually expressed as "ratio" (original size / compressed size) or "space savings" (1 − compressed size / original size). But a single number can be misleading, because compressibility varies dramatically across file types:

Text and source code: High redundancy, highly compressible. Lossless algorithms typically achieve 3x to 10x ratios.
Uncompressed bitmaps (BMP, RAW): Strong inter-pixel correlation, large compression potential.
Already-compressed data (JPEG, MP4, ZIP): Statistical redundancy already removed; re-compression yields almost nothing.
Mixed-content archives: Contain both text and already-compressed media; overall ratio depends on the proportion of each part.

Therefore, when evaluating compression ratio, you must test by file type separately, rather than relying on a tool's claimed "average ratio."

Tool Comparison (Archive Compression Tools)

The table below covers archive compression tools only. The "compression ratio" of content compression tools depends on the target format and quality parameters, which are not on the same dimension and cannot be directly compared.

Tool/Algorithm	Text Ratio	Bitmap Ratio	Already-compressed Media	Notes
gzip (DEFLATE)	3x ~ 5x	2x ~ 4x	~1x	Fast, best compatibility
xz (LZMA2)	5x ~ 8x	3x ~ 5x	~1x	High ratio, slow
7-Zip (LZMA2)	5x ~ 8x	3x ~ 5x	~1x	Supports solid compression
Zstandard	4x ~ 6x	3x ~ 4x	~1x	Best speed/ratio balance
Brotli	4x ~ 7x	3x ~ 4x	~1x	Optimized for web text

It must be emphasized that for already-compressed media like JPEG or H.264, lossless archive tools (gzip, 7z) cannot compress them further. To reduce the size of such files, you must turn to lossy re-encoding tools designed for that media type.

Recommendations

For text, source code, and logs, prioritize LZMA2-based tools (7-Zip, xz) for the highest ratio.
For a balance of speed and ratio, Zstandard is the modern default, offering adjustable levels from 1 to 22.
To shrink already-compressed media, do not rely on archive tools—use lossy re-encoding solutions for the corresponding type.

Dimension 2: Compression Speed

Evaluation Criteria

Compression speed refers to the amount of data processed per unit of time, commonly expressed in MB/s. But two details are often overlooked:

Speed is strongly correlated with compression level. The same algorithm may be 10x faster at its lowest level than at its highest, while the ratio improvement is only a few percentage points. Evaluation must fix the compression level.
Single-threaded vs. multi-threaded differences. Modern tools like Zstandard, LZMA2, and xz all support multi-threading, and throughput can multiply on multi-core machines.

Time cost is a key selection consideration. For one-time archiving, slower is acceptable; but for real-time transmission or CI/CD pipelines, speed is often more important than the maximum compression ratio.

Tool Comparison

The table below shows approximate throughput under typical conditions (text data, mid-range compression level) (archive compression tools only):

Tool	Level	Single-thread Throughput	Multi-thread Support	Decompression Speed
gzip	-6	~50 MB/s	Limited (pigz)	~200 MB/s
xz	-6	~10 MB/s	Yes	~50 MB/s
Zstandard	-3	~400 MB/s	Yes	~1000 MB/s
Zstandard	-19	~10 MB/s	Yes	~1000 MB/s
Brotli	-6	~70 MB/s	Limited	~400 MB/s

A notable phenomenon: Zstandard at level 3 approaches gzip's compression ratio at level 6, but is nearly 10x faster. This is the core advantage of modern algorithms over classic DEFLATE.

Recommendations

For interactive operations or frequently invoked scenarios, choose high-throughput tools (low-level Zstandard, gzip).
For offline archiving or cold storage preparation, slower high-compression tools (xz, high-level Zstandard) are acceptable.
Always enable multi-threading and explicitly record the core count and level in benchmarks.

Dimension 3: Quality Retention

Evaluation Criteria

For lossy compression tools, "compression ratio" and "quality loss" are two sides of the same coin. When evaluating quality retention, focus on:

Granularity of adjustable quality parameters: Whether fine-grained quality control is available (e.g., CRF 1~51, or quality 1~100).
Objective quality metrics: PSNR, SSIM, VMAF, and similar metrics can quantify the degree of quality loss.
Subjective quality performance: Some encoders produce better subjective quality at low bitrates than others, even when objective metrics are similar.
Two-pass encoding support: Enables more precise control of target bitrate or file size.

Tool Comparison

Tool Type	Typical Tools	Quality Control	Subjective Quality
Lossy image	libjpeg / mozjpeg	quality 1~100	mozjpeg smaller at same quality
Modern image formats	libwebp / libavif	quality 1~100	AVIF clearly better than JPEG at low quality
Video encoding	x264 / x265	CRF 18~28	x265 smaller at same CRF but slower
Modern video encoding	libvpx / SVT-AV1	CRF + bitrate constraint	AV1 highest ratio, extremely slow encoding
Audio encoding	libmp3lame / libopus	bitrate / VBR	Opus significantly better than MP3 at low bitrates

Taking video as an example, ffmpeg with libx265 at CRF 24 typically reduces file size by 30%~50% compared to libx264 at CRF 24, while the VMAF score difference is within 2 points and is subjectively almost imperceptible.

Recommendations

Image scenarios: For compatibility priority, use mozjpeg; for maximum compression ratio where the environment supports it, use AVIF.
Video scenarios: For distribution and archiving, choose x265 or SVT-AV1; for real-time communication, choose VP9 or AV1 real-time mode.
Audio scenarios: Use Opus for speech; AAC for music distribution; consider MP3 only for compatibility needs.
Always combine objective metrics (VMAF, SSIM) with subjective blind testing—do not rely on bitrate alone.

Dimension 4: Format Support

Evaluation Criteria

A compression tool's value depends heavily on how many formats it can handle. Evaluation focuses on two layers:

Archive formats: Support for containers and compression algorithms like ZIP, 7Z, TAR, GZ, XZ, ZST.
Content formats: PDF, images (JPEG/PNG/WebP/AVIF), video (MP4/MKV/MOV), documents (Office), audio (MP3/AAC/FLAC/Opus).

Many tools are strong in only one layer. For example, 7-Zip excels at archive formats but does almost no content re-encoding; FFmpeg excels at audio/video transcoding but does not handle archiving. True "all-in-one" tools are rare, and in most cases a combination is needed.

Tool Comparison

Tool	Category	Archive Formats	Image Re-encoding	Video Transcoding	Audio Transcoding	PDF Processing
7-Zip	Archive	Rich	No	No	No	No
FFmpeg	Content	No	Yes	Yes	Yes	No
ImageMagick	Content	No	Yes	Limited	Limited	No
Ghostscript	Content	No	No	No	No	Yes
qpdf / cpdf	Content	No	No	No	No	Yes
Online services	Mixed	Limited	Yes	Yes	Yes	Yes

Recommendations

For single file-type scenarios, choose a specialized tool for that domain (e.g., FFmpeg for video, Ghostscript for PDF).
For diverse file-type scenarios, build a toolchain: 7-Zip for archiving, FFmpeg/ImageMagick for media, Ghostscript for PDF.
Online services suit ad-hoc needs but are not suitable for large volumes or sensitive files—see Dimension 5.

Dimension 5: Security and Privacy

Evaluation Criteria

Compression involves data reading, processing, and storage. The security dimension has three layers:

Data processing location: Local processing vs. cloud upload. Local processing never leaves the device, offering the highest privacy.
Transmission and storage encryption: Whether archives support encryption (AES-256), whether file names can be encrypted.
Supply chain security: Whether the tool is open source, auditable, and has a known vulnerability history.

For enterprises or scenarios involving sensitive information, this dimension is often a veto factor.

Local Processing vs. Online Upload

Processing Method	Privacy Risk	Speed	Applicable Scenarios
Pure local CLI/GUI	Very low	Depends on hardware	Sensitive files, large batches
Online service (upload)	High	Fast (cloud compute)	Public assets, ad-hoc needs
Hybrid (encrypt locally, then upload)	Medium	Medium	Teams needing collaboration

Online services typically store files temporarily on servers. Some claim "deleted after processing," but this is hard for users to verify. For sensitive files like contracts, ID documents, and medical images, local processing should be mandatory.

Encryption Capability Comparison

7-Zip: Supports AES-256 encryption with optional filename encryption; the de facto standard for archive encryption.
ZIP: Traditional ZIP encryption (ZipCrypto) is insecure; use WinZip's AES extension instead.
GPG: Asymmetric encryption, suitable for file transfer encryption, but not a compression tool.

Recommendations

For classified or compliance scenarios, mandate local open-source tools (7-Zip, GPG).
When encrypted archiving is needed, use 7-Zip's AES-256 with filename encryption.
Stay vigilant about online services; use them only for publicly shareable assets.

Dimension 6: Usability

Evaluation Criteria

Usability determines whether a tool can truly be adopted. Key evaluation points include:

Interaction form: CLI (command line) vs. GUI (graphical interface) vs. API (programming interface).
Batch processing capability: Whether directory recursion, wildcards, and task queues are supported.
Automation integration: How easily it embeds into scripts, CI/CD pipelines, and scheduled tasks.
Observability: Whether progress, logs, and error codes are provided for troubleshooting.

CLI tools are powerful but have a steep learning curve; GUI tools are easy to pick up but hard to automate. The ideal choice depends on the user's technical background and frequency of use.

CLI vs. GUI Comparison

Dimension	CLI Tools	GUI Tools
Learning cost	High	Low
Batch processing	Strong (script-driven)	Implementation-dependent
Automation	Natively supported	Usually unsupported
Parameter granularity	High	Limited by interface
Suited for	Developers, ops	General office users

Recommendations

Developers and ops: Use CLI tools primarily (FFmpeg, 7-Zip, zstd), paired with shell scripts for automation.
General office users: Choose GUI tools that support batch drag-and-drop (e.g., Bandizip, HandBrake's graphical interface).
Enterprise workflows: Choose solutions that provide APIs or SDKs for easy integration into existing systems.

Dimension 7: Cost and Licensing

Evaluation Criteria

Cost is not just the purchase price; it also includes:

License type: Open source (GPL, BSD, MIT, LGPL) vs. commercial proprietary vs. free but closed source.
Commercial restrictions: Some tools are free for personal use but require payment for commercial use (e.g., WinRAR).
Patent risk: Some codec formats (e.g., H.264, HEVC) involve patent licensing fees; enterprises distributing them must be careful.
Maintenance and support: Open-source tools rely mainly on community support; commercial tools usually offer SLAs.

Tool Licensing Comparison

Tool	Category	License	Commercial Use	Patent Risk
7-Zip	Archive	LGPL	Free	Low
WinRAR	Archive	Proprietary	Commercial requires payment	None
Zstandard	Archive	BSD	Free	None
FFmpeg	Content	LGPL/GPL (configurable)	Watch codec licenses	H.264/HEVC involve patents
x264/x265	Content	GPL	Commercial requires license	HEVC patent pool complex
SVT-AV1	Content	BSD	Free	AV1 patents open

Patent risk is the most easily overlooked trap in enterprise selection. For example, using x265 to encode and distribute H.265 video in a product may require paying licensing fees to patent pools like MPEG LA and HEVC Advance. AV1, although slow to encode, is safer on the patent front.

Recommendations

Personal use: Open-source tools meet the vast majority of needs.
Enterprise commercial use: Prefer BSD/MIT-licensed tools to avoid GPL contagion; when distributing video, evaluate HEVC patent costs and switch to AV1 if necessary.
Teams needing official support: Evaluate the total cost of ownership (TCO) of commercial solutions, including license fees and operational costs.

Comprehensive Scoring Matrix

By quantifying the 7 dimensions on a 1~5 scale, you can build a comprehensive scoring matrix. The table below uses several typical tool categories as examples to show how to use this matrix for horizontal comparison (higher scores are better):

Dimension	7-Zip	FFmpeg	Zstandard	Online Service	Commercial GUI
Tool Category	Archive	Content	Archive	Mixed	Mixed
Compression ratio	5	4 (media)	4	3	3
Compression speed	3	4	5	4	3
Quality retention	N/A	5	N/A	3	3
Format support	4 (archive)	5 (media)	2	4	4
Security & privacy	5	5	5	2	3
Usability	3 (CLI)/4 (GUI)	2 (CLI)	3	5	5
Cost & licensing	5	4	5	2	2
Weighted total	25	29	24	23	23

Cross-Category Comparison Note: The scores for 7-Zip (archive) and FFmpeg (content) in the table above are not directly comparable—they solve problems at different levels. 7-Zip's "compression ratio 5" refers to its lossless archiving capability for uncompressed data, while FFmpeg's "compression ratio 4" refers to its lossy re-encoding capability for audio/video. The weighted total is only for demonstrating the scoring method; in practice, first determine which type of tool you need, then compare within the same category.

Note: The weighted total is only an example. In practice, weight each dimension according to the scenario. For example, enterprise batch processing should give high weight to "security and privacy," while personal office should give high weight to "usability."

Recommended Tools by Scenario

Scenario 1: Personal Office

Requirements: Diverse file types (PDF, images, Office documents), low frequency, low automation requirements, high usability requirements.

Recommended combination:

General archiving: 7-Zip (free, comprehensive formats, strong encryption)
PDF compression: Ghostscript command line or graphical tools built on it
Image compression: Local tools based on mozjpeg or libwebp
Video compression: HandBrake (a graphical front-end for FFmpeg with friendly presets)

Scenario 2: Developer

Requirements: Needs to embed in scripts and CI/CD, pursues controllability and reproducibility, high requirements for CLI and API friendliness.

Recommended combination:

Text and archiving: Zstandard (balanced speed and ratio, multi-threaded) or xz (extreme ratio)
Image processing: ImageMagick + mozjpeg
Video transcoding: FFmpeg + libx264/libx265/SVT-AV1, choosing the encoder as needed
PDF processing: Ghostscript, qpdf, cpdf
Automation: Orchestrate with shell or Python, recording logs and exit codes

Scenario 3: Enterprise Batch Processing

Requirements: Large data volumes, strict compliance requirements, need for audit and access control, may involve patents and licensing.

Recommended combination:

Archiving and encryption: 7-Zip (AES-256) + internal audit process
Media transcoding: Self-built FFmpeg cluster, prioritizing SVT-AV1 to avoid HEVC patent risk
PDF processing: Ghostscript cluster + file permission management
Monitoring: Integrate into a task queue (e.g., Celery, Airflow), recording each task's processing time, compression ratio, and quality metrics
Security: All processing done locally or in a private cloud; uploading to public network services is prohibited

FAQ

Q1: Can free open-source tools really meet enterprise needs?

In most scenarios, yes. Tools like 7-Zip, FFmpeg, Zstandard, and Ghostscript are already widely adopted in enterprise production environments. The key point is that open-source tools are usually powerful enough, but lack official SLAs and commercial support. Enterprises must bear the operational and troubleshooting costs themselves and evaluate the patent licensing risks of relevant codecs. For industries with extremely high compliance requirements (such as finance and healthcare), it is recommended to build internal audit and rollback mechanisms on top of open-source tools.

Q2: Can online compression services be used for work files?

It is not recommended for sensitive work files. Online services carry data leakage risks during transmission and processing, and the promise of "deleted after processing" is hard to verify. Even for public assets, frequently uploading large files consumes bandwidth and time. For work files, local processing should be mandatory; only consider online services when files are explicitly public and the need is a one-off.

Q3: Is a higher compression level always better?

No. The higher the compression level, the slower the speed, while the marginal gain in compression ratio diminishes. Taking Zstandard as an example, going from level 3 to level 19 may improve the ratio by only 10%~15%, but compression speed drops by tens of times. When selecting, find the inflection point on the "speed-ratio" curve according to the scenario, rather than blindly pursuing the highest level. Typically, levels 3~6 cover most everyday needs.

Q4: How do you evaluate the quality of a new tool?

Score it on each of the 7 dimensions in this article, and prepare a test set covering multiple file types for benchmarking. Focus on recording: compression ratio for each file type, compression and decompression throughput, quality metrics for lossy compression (PSNR/SSIM/VMAF), multi-threading support, and licensing and patent status. Only by horizontally comparing existing tools with the same test set can you reach an objective conclusion.

Summary

There is no silver bullet for file compression tool selection. A tool that leads in compression ratio may lag in speed or usability; a free open-source tool may hide patent risks; a convenient online service may introduce unacceptable privacy concerns.

The value of the 7-dimension evaluation framework is that it forces you to decompose the vague notion of "good or bad" into quantifiable, comparable concrete metrics. Compression ratio, compression speed, quality retention, format support, security, usability, cost and licensing—each dimension corresponds to a real engineering constraint, and ignoring any one of them can lead to a selection mistake.

In practice, first clarify the usage scenario and constraints (file types, data volume, compliance requirements, frequency of use), then assign weights to the 7 dimensions, and finally score horizontally using a uniform test set. This structured approach is more reliable than relying on vendor marketing or a single review article, and it is easier to build consensus within a team. Once you master this framework, you can quickly judge whether any new tool is worth adopting—rather than being led by marketing copy.

Batch Compress 1000 Images with Python Pillow: Complete Guide

Uglypear Data — Mon, 03 Aug 2026 05:49:52 +0000

Why Batch Compress Images

You have 1000 high-resolution photos exported from a camera, each around 8MB, and the entire directory consumes nearly 8GB. If the goal is uploading to a website, embedding into a presentation, or sending via email, that volume is entirely unacceptable. Compressing them one by one in a graphical editor is both inefficient and error-prone—this is exactly where scripted batch processing shines.

Python's Pillow library is the de facto standard for image processing. It provides read and write support for mainstream formats including JPEG, PNG, WebP, BMP, and TIFF, along with fine-grained control over quality factors, sampling rates, and color modes. With a few dozen lines of Python, you can traverse an entire directory tree, apply a uniform compression strategy, and preserve the original folder structure in the output.

This article starts from Pillow's basic usage and progressively builds a complete, runnable, multithreaded batch compression script. It also covers practical engineering details such as EXIF preservation, format conversion, and intelligent parameter selection.

Pillow Installation and Basic Usage

Installation

Pillow is the actively maintained fork of PIL (Python Imaging Library), but the import name remains PIL. Install it with pip:

pip install Pillow

If you need to handle a large volume of images or want faster decoding, install the full version with plugin support:

pip install "Pillow[all]"

After installation, verify the version:

from PIL import Image, __version__
print(__version__)  # e.g., 10.4.0

Basic Operations

The core entry point of Pillow is the Image class. The three most common operations are open, modify, and save:

from PIL import Image

# Open an image (lazy loading; pixel data is read on first access)
img = Image.open("photo.jpg")
print(img.format, img.size, img.mode)  # JPEG (4000, 3000) RGB

# Get a pixel value
pixel = img.getpixel((100, 100))

# Resize
resized = img.resize((1920, 1080))

# Save
resized.save("photo_small.jpg", quality=85)

A few details are worth noting: the object returned by Image.open() holds a file reference that is not released until explicitly closed or the with block exits. In batch processing, always use a context manager or call close() explicitly, otherwise file handles will leak.

Single Image Compression Examples

The foundation of batch compression is single-image processing. We first implement a function that supports JPEG quality adjustment, resolution downscaling, and format conversion.

JPEG Quality Adjustment

The quality parameter of JPEG ranges from 1 to 100 and directly controls the scaling of the quantization table. It is the primary knob for balancing file size and image quality:

from PIL import Image

def compress_jpeg(input_path, output_path, quality=75):
    """Compress a JPEG with the specified quality factor"""
    with Image.open(input_path) as img:
        # JPEG does not support transparency; RGBA/P modes must be converted first
        if img.mode in ("RGBA", "P", "LA"):
            img = img.convert("RGB")
        elif img.mode != "RGB":
            img = img.convert("RGB")
        img.save(
            output_path,
            format="JPEG",
            quality=quality,
            optimize=True,       # Enable Huffman encoding optimization
            progressive=True,    # Generate progressive JPEG; slightly larger but better loading experience
        )

# Compare different quality factors
compress_jpeg("photo.jpg", "q95.jpg", quality=95)
compress_jpeg("photo.jpg", "q75.jpg", quality=75)
compress_jpeg("photo.jpg", "q50.jpg", quality=50)

optimize=True makes Pillow scan the data an extra time during saving to rebuild an optimal Huffman encoding table. This typically reduces the file size by another 2-5% at the cost of slightly slower saving.

Resolution Adjustment

For web display, a 4000x3000 original is far larger than necessary. Using thumbnail() to downscale proportionally is the most effective approach—halving the resolution reduces the file size by roughly 75%:

from PIL import Image

def resize_image(input_path, output_path, max_size=1920, quality=80):
    """Scale the longest edge to max_size while preserving the aspect ratio"""
    with Image.open(input_path) as img:
        # thumbnail modifies in place and never exceeds the specified dimensions
        # Image.Resampling.LANCZOS is the best-quality downsampling algorithm
        img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
        if img.mode != "RGB":
            img = img.convert("RGB")
        img.save(output_path, format="JPEG", quality=quality, optimize=True)

resize_image("photo.jpg", "photo_1920.jpg", max_size=1920, quality=80)

The difference between thumbnail() and resize() is that thumbnail() preserves the aspect ratio and never enlarges the image—the final dimensions are always less than or equal to the specified values. resize() forces the image to the exact target dimensions. In batch processing, you almost always want thumbnail().

Format Conversion

Converting PNG to WebP is a common optimization—WebP is 26% smaller than PNG and 25-35% smaller than JPEG at equivalent quality, while also supporting transparency:

from PIL import Image

def convert_to_webp(input_path, output_path, quality=80, lossless=False):
    """Convert any format to WebP"""
    with Image.open(input_path) as img:
        img.save(
            output_path,
            format="WebP",
            quality=quality,
            lossless=lossless,   # When True, uses lossless mode
            method=6,            # Compression effort 0-6; 6 is slowest but smallest
        )

convert_to_webp("icon.png", "icon.webp", quality=85)
convert_to_webp("logo.png", "logo_lossless.webp", lossless=True)

The method parameter controls compression effort, ranging from 0 to 6. Higher values yield better compression ratios but take longer. For batch processing where ultimate file size matters, set it to 6; for speed, 4 is a good balance.

Complete Batch Compression Script

Now we assemble the single-image logic into a complete batch script. This script traverses all images in the input directory, preserves the original folder structure in the output, and displays processing progress along with the overall compression results.

The flowchart below illustrates the overall batch processing workflow:

The complete script:

"""
batch_compress.py - Batch image compression script
Usage: python batch_compress.py <input_dir> <output_dir> [--quality 75] [--max-size 1920]
"""
import os
import sys
import argparse
from pathlib import Path
from PIL import Image

SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".tiff", ".tif"}


def compress_single(input_path, output_path, quality=75, max_size=1920,
                    target_format=None, keep_exif=True):
    """Compress a single image"""
    with Image.open(input_path) as img:
        # 1. Extract EXIF data (capture time, camera model, GPS, etc.)
        exif_data = img.info.get("exif", b"") if keep_exif else b""

        # 2. Color mode conversion: JPEG does not support transparency
        out_format = (target_format or img.format or "JPEG").upper()
        if out_format in ("JPEG", "JPG") and img.mode in ("RGBA", "P", "LA"):
            # Composite onto a white background to avoid black regions after conversion
            background = Image.new("RGB", img.size, (255, 255, 255))
            if img.mode == "P":
                img = img.convert("RGBA")
            background.paste(img, mask=img.split()[-1] if img.mode == "RGBA" else None)
            img = background
        elif img.mode not in ("RGB", "RGBA", "L"):
            img = img.convert("RGB")

        # 3. Resolution adjustment: only downscale, never upscale
        if max_size and max(img.size) > max_size:
            img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)

        # 4. Save
        save_kwargs = {"optimize": True}
        if out_format in ("JPEG", "JPG"):
            save_kwargs["quality"] = quality
            save_kwargs["progressive"] = True
            if exif_data:
                save_kwargs["exif"] = exif_data
        elif out_format == "WEBP":
            save_kwargs["quality"] = quality
            save_kwargs["method"] = 6
        elif out_format == "PNG":
            save_kwargs["optimize"] = True

        img.save(output_path, format=out_format, **save_kwargs)


def batch_compress(input_dir, output_dir, quality=75, max_size=1920,
                   target_format=None, keep_exif=True):
    """Batch compress an entire directory"""
    input_dir = Path(input_dir)
    output_dir = Path(output_dir)

    # Collect all files to process
    tasks = []
    for root, _, files in os.walk(input_dir):
        for filename in files:
            if Path(filename).suffix.lower() in SUPPORTED_FORMATS:
                src = Path(root) / filename
                rel = src.relative_to(input_dir)
                dst = output_dir / rel
                # Adjust extension based on target format
                if target_format:
                    dst = dst.with_suffix(f".{target_format.lower()}")
                tasks.append((src, dst))

    total = len(tasks)
    if total == 0:
        print("No supported image files found")
        return

    print(f"Found {total} images. Starting compression...\n")

    processed = 0
    total_original = 0
    total_compressed = 0
    errors = []

    for src, dst in tasks:
        # Ensure output directory exists
        dst.parent.mkdir(parents=True, exist_ok=True)

        original_size = src.stat().st_size
        try:
            compress_single(src, dst, quality, max_size, target_format, keep_exif)
            new_size = dst.stat().st_size
            total_original += original_size
            total_compressed += new_size
        except Exception as e:
            errors.append((src, str(e)))
            new_size = original_size  # Count as original size on failure

        processed += 1
        pct = processed / total * 100
        saved = (1 - new_size / original_size) * 100 if original_size else 0
        print(f"[{pct:5.1f}%] ({processed}/{total}) {src.relative_to(input_dir)}  "
              f"{original_size // 1024}KB -> {new_size // 1024}KB  (-{saved:.1f}%)")

    # Summary report
    print(f"\n{'=' * 50}")
    print(f"Done: {processed} succeeded, {len(errors)} failed")
    if total_original > 0:
        ratio = total_original / total_compressed if total_compressed else 0
        saved_mb = (total_original - total_compressed) / 1024 / 1024
        print(f"Original size: {total_original / 1024 / 1024:.2f} MB")
        print(f"Compressed size: {total_compressed / 1024 / 1024:.2f} MB")
        print(f"Space saved: {saved_mb:.2f} MB (ratio {ratio:.2f}x)")

    for src, err in errors:
        print(f"  Failed: {src} - {err}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Batch image compression tool")
    parser.add_argument("input_dir", help="Input image directory")
    parser.add_argument("output_dir", help="Output directory")
    parser.add_argument("--quality", type=int, default=75, help="JPEG/WebP quality factor (1-100)")
    parser.add_argument("--max-size", type=int, default=1920, help="Maximum longest edge in pixels")
    parser.add_argument("--format", default=None, help="Target format (JPEG/WebP/PNG)")
    parser.add_argument("--no-exif", action="store_true", help="Discard EXIF data")
    args = parser.parse_args()

    batch_compress(
        args.input_dir,
        args.output_dir,
        quality=args.quality,
        max_size=args.max_size,
        target_format=args.format,
        keep_exif=not args.no_exif,
    )

Usage examples:

# Basic usage
python batch_compress.py ./photos ./output

# Convert to WebP, quality 80, max longest edge 2560
python batch_compress.py ./photos ./output --quality 80 --max-size 2560 --format webp

This script has three core design principles: it uses os.walk() to recursively traverse and preserve the directory structure; it uses relative_to() to compute relative paths ensuring the output layout matches the input; and it uses Path.stat() to obtain file sizes for calculating the actual compression effect.

Advanced Techniques

EXIF Information Preservation

EXIF data contains capture time, GPS coordinates, camera parameters, aperture, shutter speed, and other metadata. By default, img.save() discards this information. To preserve it, you must extract it manually and pass it when saving:

from PIL import Image

def compress_with_exif(input_path, output_path, quality=75):
    with Image.open(input_path) as img:
        exif = img.info.get("exif", b"")  # Extract raw EXIF bytes
        if img.mode != "RGB":
            img = img.convert("RGB")
        img.save(
            output_path,
            format="JPEG",
            quality=quality,
            exif=exif if exif else None,  # Pass None when there is no EXIF
            optimize=True,
        )

If you only need to retain certain EXIF fields (for example, only the orientation tag), use the getexif() method to pick items individually:

from PIL import ExifTags

def keep_orientation_only(input_path, output_path, quality=75):
    with Image.open(input_path) as img:
        exif = img.getexif()
        orientation = exif.get(0x0112)  # Orientation tag
        # Rotate the image based on the orientation tag
        if orientation:
            from PIL import ImageOps
            img = ImageOps.exif_transpose(img)
        img.save(output_path, format="JPEG", quality=quality, optimize=True)

ImageOps.exif_transpose() automatically rotates the image according to the EXIF orientation tag. This is especially useful when processing photos taken with smartphones—many phone sensors are mounted in landscape orientation and rely on the EXIF orientation tag to indicate the correct display direction.

PNG to WebP

When converting PNG to WebP, you need to choose between lossy and lossless modes based on the image characteristics. For images with text and sharp lines (icons, UI elements), lossless WebP preserves crisp edges. For photographic content, lossy WebP offers a significant size advantage:

from PIL import Image

def png_to_webp(input_path, output_path, lossless_threshold=0.3):
    """Intelligently choose lossy or lossless WebP based on image characteristics"""
    with Image.open(input_path) as img:
        # Count colors to determine if this is a simple graphic
        colors = img.getcolors(maxcolors=65536)
        is_simple = colors is not None and len(colors) < 256

        if is_simple:
            # Few colors, sharp edges: lossless mode
            img.save(output_path, format="WebP", lossless=True, method=6)
            print(f"Lossless WebP (color count: {len(colors)})")
        else:
            # Rich colors, photographic: lossy mode
            if img.mode != "RGB":
                img = img.convert("RGB")
            img.save(output_path, format="WebP", quality=82, method=6)
            print("Lossy WebP (photographic)")

getcolors() returns a list of colors or None (when the color count exceeds maxcolors). A small number of colors usually indicates a simple graphic such as an icon or screenshot, which is better suited for lossless compression.

Smart Parameter Selection

Different images are best suited for different compression strategies. The following function automatically selects the format and parameters based on resolution, transparency, and color count:

from PIL import Image

def smart_compress(input_path, output_path):
    """Automatically select the optimal compression strategy based on image characteristics"""
    with Image.open(input_path) as img:
        width, height = img.size
        pixel_count = width * height
        mode = img.mode
        has_transparency = mode in ("RGBA", "LA") or (
            mode == "P" and "transparency" in img.info
        )

        # Strategy 1: Has transparency -> WebP (balances transparency and size)
        if has_transparency:
            if pixel_count > 2_000_000:
                q = 80
            else:
                q = 88
            img.save(output_path, format="WebP", quality=q, method=6)
            return "Lossy WebP (transparent)"

        # Strategy 2: Very large photo -> downscale + JPEG medium quality
        if pixel_count > 8_000_000 or width > 4000:
            img = img.convert("RGB")
            img.thumbnail((2560, 2560), Image.Resampling.LANCZOS)
            img.save(output_path, format="JPEG", quality=75, optimize=True)
            return "JPEG (large image downscaled)"

        # Strategy 3: Normal photo -> JPEG higher quality
        if pixel_count > 500_000:
            img = img.convert("RGB")
            img.save(output_path, format="JPEG", quality=82, optimize=True)
            return "JPEG (high quality)"

        # Strategy 4: Small image -> keep as PNG
        img.save(output_path, format="PNG", optimize=True)
        return "PNG (small image preserved)"

The core logic of this strategy is: images with transparency go to WebP, large images are downscaled before compression, normal photos use JPEG, and small images use PNG for fidelity. You can adjust the thresholds based on your actual requirements.

Performance Optimization

Multithreaded Processing

Image compression is a task that mixes I/O-bound and CPU-bound work. Disk read/write and encoding each account for a portion of the time. Python's GIL limits pure CPU parallelism, but Pillow releases the GIL when calling the underlying C libraries for JPEG/WebP encoding, so multithreading can bring substantial speedups:

import os
from pathlib import Path
from PIL import Image
from concurrent.futures import ThreadPoolExecutor, as_completed
import threading

SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".tiff"}

_print_lock = threading.Lock()


def compress_single(input_path, output_path, quality=75, max_size=1920):
    with Image.open(input_path) as img:
        exif = img.info.get("exif", b"")
        if img.mode != "RGB":
            img = img.convert("RGB")
        if max_size and max(img.size) > max_size:
            img.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
        kwargs = {"quality": quality, "optimize": True, "progressive": True}
        if exif:
            kwargs["exif"] = exif
        img.save(output_path, format="JPEG", **kwargs)


def batch_compress_threaded(input_dir, output_dir, quality=75, max_size=1920,
                            workers=4):
    input_dir = Path(input_dir)
    output_dir = Path(output_dir)

    # Collect tasks
    tasks = []
    for root, _, files in os.walk(input_dir):
        for f in files:
            if Path(f).suffix.lower() in SUPPORTED_FORMATS:
                src = Path(root) / f
                rel = src.relative_to(input_dir)
                dst = output_dir / rel
                dst.parent.mkdir(parents=True, exist_ok=True)
                tasks.append((src, dst))

    total = len(tasks)
    print(f"{total} images total, processing with {workers} threads\n")

    completed = 0
    total_saved = 0

    with ThreadPoolExecutor(max_workers=workers) as executor:
        futures = {
            executor.submit(compress_single, src, dst, quality, max_size): (src, dst)
            for src, dst in tasks
        }
        for future in as_completed(futures):
            src, dst = futures[future]
            try:
                future.result()
                saved = src.stat().st_size - dst.stat().st_size
                total_saved += saved
            except Exception as e:
                with _print_lock:
                    print(f"Failed: {src.name} - {e}")
            completed += 1
            with _print_lock:
                print(f"[{completed / total * 100:5.1f}%] ({completed}/{total}) {src.name}")

    print(f"\nDone. Space saved: {total_saved / 1024 / 1024:.2f} MB")


if __name__ == "__main__":
    batch_compress_threaded("./photos", "./output", quality=75, max_size=1920, workers=4)

The empirical rule for thread count is the number of CPU logical cores. On an 8-core machine, 4-6 threads typically achieve near-optimal speedup. Too many threads can actually degrade performance due to disk I/O contention and memory pressure.

If you need to go further with parallelism, you can use ProcessPoolExecutor to bypass the GIL. The tradeoff is that processes cannot share memory, each process must independently load the Pillow library, and startup overhead is higher.

Memory Management

Memory consumption becomes a concern when processing high-resolution images. A single 6000x4000 RGB image occupies about 72MB of memory after decoding (6000x4000x3 bytes). If 10 are open simultaneously, that is 720MB.

Several key memory management principles:

from PIL import Image
import gc

# 1. Always use the with statement to ensure file handles and pixel buffers are released promptly
def safe_compress(input_path, output_path, quality=75):
    with Image.open(input_path) as img:
        # 2. Copy the image if you need to modify it, avoiding operations on the original
        work = img.copy()
    # img is released; work is still in scope
    if work.mode != "RGB":
        work = work.convert("RGB")
    work.save(output_path, format="JPEG", quality=quality, optimize=True)
    work.close()  # 3. Close explicitly

# 4. Trigger garbage collection periodically during batch processing
def batch_with_gc(tasks, interval=100):
    for i, task in enumerate(tasks):
        process(task)
        if i % interval == 0:
            gc.collect()

Additionally, Image.MAX_IMAGE_PIXELS defaults to a limit of approximately 178 million pixels (about 8900x8900). Exceeding this raises a DecompressionBombError. If you genuinely need to process very large images, you can raise this threshold, but be mindful of memory consumption:

Image.MAX_IMAGE_PIXELS = None  # Disable the limit (use with caution)

Compression Results Comparison

The table below shows measured data from a set of 50 smartphone photos (original average 6.5MB, 4000x3000 pixels). All tests ran on the same hardware, with resolution uniformly downscaled to a longest edge of 1920:

Quality Factor	Format	Average Size	Ratio	Subjective Quality	Use Case
Original	JPEG	6.5 MB	1.0x	Baseline	—
95	JPEG	1.8 MB	3.6x	Virtually identical	High-quality archival
85	JPEG	0.92 MB	7.1x	Imperceptible difference	Website hero images
75	JPEG	0.58 MB	11.2x	Slight artifacts on close inspection	Thumbnails, listing pages
65	JPEG	0.42 MB	15.5x	Visible blocky artifacts	Not recommended
50	JPEG	0.28 MB	23.2x	Obvious distortion	Tiny previews only
80	WebP	0.51 MB	12.7x	Comparable to JPEG 85	Modern websites
Lossless	WebP	4.2 MB	1.5x	Perfectly lossless	When transparency is needed

Several patterns emerge from the data:

Quality 85 is the cost-effectiveness inflection point: it saves nearly half the size compared to 95, with quality differences that are virtually imperceptible to the eye.
Below quality 70, returns diminish rapidly: file size continues to decrease, but quality degradation accelerates noticeably.
WebP quality factors are not directly equivalent to JPEG: WebP 80 produces quality comparable to JPEG 85 but at a smaller size.
Lossless WebP only makes sense when transparency is required; for photographs, the size advantage is minimal.

FAQ

Q1: Why do images appear upside down or sideways after compression?

This happens because the EXIF orientation tag from the smartphone is discarded during compression, while the actual pixel data was stored in landscape orientation. The solution is to call ImageOps.exif_transpose(img) before saving. It rotates the pixel data according to the EXIF orientation tag so the image is correctly oriented before being saved. This way, even if downstream software does not read the EXIF orientation information, the image will display correctly.

Q2: Why does memory usage keep growing when processing 1000 images?

The most common cause is failing to close image objects promptly. Always use the with Image.open(...) as img: context manager to ensure each image is released immediately after processing. Second, in the multithreaded version, if all tasks are submitted at once, the future objects hold references to results until they are consumed, causing memory to accumulate. The solution is to submit tasks in batches using chunksize, or switch to a producer-consumer queue pattern to control concurrency. Finally, calling gc.collect() periodically can reclaim memory held by circular references.

Q3: Why do transparent areas turn black after converting PNG to JPEG?

The JPEG format does not support transparency. When Pillow converts RGBA to RGB by default, it fills the transparent background with black. The solution is to first create a white-background RGB image, then use paste() to overlay the original image with the alpha channel as a mask:

background = Image.new("RGB", img.size, (255, 255, 255))
background.paste(img, mask=img.split()[3])  # The 4th channel is alpha

Alternatively, convert directly to WebP, which natively supports transparency without any extra handling.

Q4: Why doesn't multithreading deliver a linear speedup?

Image compression involves both disk I/O and CPU encoding. Once the thread count increases beyond a certain point, disk bandwidth becomes the bottleneck—the CPU cannot go faster than the data can be read and written. Additionally, decoded pixel data for high-resolution images is large, and processing multiple large images simultaneously in different threads leads to memory bandwidth contention and reduced cache hit rates. In practice, 4-8 threads typically yield a 2-4x speedup, with diminishing returns beyond that. For mechanical hard drives, 4 threads is recommended; for NVMe SSDs, the count can be increased somewhat.

Summary

Batch compressing images with Python Pillow can be broken down into three layers:

Foundation layer: Master the three methods Image.open(), thumbnail(), and save(), combined with the quality and optimize parameters, and you can compress a single image. This is the building block for all batch processing.
Engineering layer: Use os.walk() to traverse directories, Path.relative_to() to preserve structure, and concurrent.futures for parallel processing to scale single-image logic to thousands of images. This layer focuses on correctness, robustness, and throughput.
Optimization layer: Intelligently select formats and parameters based on image characteristics—transparent images go to WebP, large images are downscaled first, small images stay as PNG. Details like EXIF preservation, orientation correction, and memory management determine whether the script can run reliably in a production environment.

There is an empirical rule for quality factor selection: 85 is the threshold of visually lossless quality, 75 is the balance point between size and quality, and below 65 is not recommended for display purposes. WebP is now fully supported across modern browsers and should be the preferred format for new projects.

The value of this script lies not in replacing professional tools but in its customizability. You can adjust every stage according to your actual needs—configure different parameters for specific directories, automatically upload to a CDN after compression, or integrate it into a CI/CD pipeline. Understanding the principles behind each line of code gives you the foundation to handle any image processing scenario.

Is Using Online Compression Tools a GDPR/PIPL Compliance Risk?

Uglypear Data — Sat, 01 Aug 2026 03:38:41 +0000

Where Compliance Begins

Since China's Data Security Law (DSL) and Personal Information Protection Law (PIPL) took effect in 2021, and with the EU's General Data Protection Regulation (GDPR) in force since 2018, enterprises face unprecedented compliance scrutiny over how they process files. A contract, a financial report, a customer list—each can cross a legal red line during transmission, storage, compression, or conversion.

File compression and format conversion are routine operations, but legally they all qualify as "data processing activities." The moment data is created, edited, transmitted, or destroyed, it falls under the DSL's scope; the moment a file contains personal information, it also falls under the PIPL's scope (and the GDPR's scope if it concerns EU data subjects). So the answer to "is using an online compression tool a compliance violation?" does not depend on the tool itself. It depends on three variables: the classification level of the file, where the data flows during processing, and whether the enterprise has fulfilled its compliance obligations.

This article examines the legal requirements, maps out the compliance risks of enterprise file processing, and answers where the compliance boundary of online compression tools lies—across both the Chinese (DSL/PIPL) and international (GDPR) regulatory contexts.

I. Core Requirements of the DSL, PIPL, and GDPR

The Data Security Law: Two Core Obligations

China's Data Security Law, effective September 1, 2021, establishes the foundational framework for data security. For enterprise file processing, the core requirements concentrate on two levels:

The first is data classification and grading. The law requires the state to establish a data classification and grading protection system, and enterprises must classify and grade their own data accordingly. Files must be divided into different levels based on importance and the severity of harm if leaked. Public, internal, sensitive, and core classified information should each receive protection of different intensity. This directly affects tool selection—files of different levels must be matched with processing methods of corresponding security.

The second is full-lifecycle secure processing. File processing (creation, editing, compression, conversion, transmission, destruction) all constitute data processing activities and must be brought under compliance management. Enterprises must ensure tools themselves do not cause data leakage, and "whether the data flow is controllable" must be the primary evaluation factor.

The PIPL: Four Key Constraints

The Personal Information Protection Law, effective November 1, 2021, sets explicit requirements for the collection, storage, use, transmission, and deletion of personal information. Many files enterprises handle contain personal information: employee resumes with ID numbers, customer contracts with contact details, financial statements with bank accounts. The PIPL imposes four main constraints:

Minimum necessity: Processing personal information shall be limited to the minimum scope necessary to achieve the processing purpose.
Notice and consent: Collecting and using personal information requires informing the individual and obtaining consent.
Cross-border transfer restrictions: Providing personal information overseas requires meeting statutory conditions such as security assessment, certification, or signing a standard contract.
Deletion obligation: Once the processing purpose is achieved, personal information should be proactively deleted.

The GDPR: Parallel Obligations

For organizations subject to the GDPR, the obligations overlap significantly with the PIPL, though the legal mechanisms differ:

Lawfulness, fairness, and transparency (Article 5) mirrors the PIPL's notice-and-consent framework.
Data minimization (Article 5(1)(c)) is the GDPR equivalent of minimum necessity: collect only what is adequate and necessary.
International data transfers (Chapter V) require safeguards such as Standard Contractual Clauses (SCCs), Binding Corporate Rules, or adequacy decisions—functionally similar to the PIPL's cross-border conditions.
Records of processing activities (Article 30) require controllers and processors to maintain documentation of processing operations, analogous to the DSL's log retention requirement.

The practical takeaway is that an enterprise operating across jurisdictions faces a converging set of obligations: classify data, minimize collection, control cross-border flows, and keep auditable records. The most easily overlooked gap is uploading files containing personal information to online tools hosted on overseas servers. A seemingly ordinary "online compression" operation may already constitute a cross-border transfer, triggering statutory obligations under both the PIPL and the GDPR.

II. The Data Classification System and Its Impact on File Processing

Data classification is not only a legal requirement; it is the practical starting point for file processing compliance. Without classification there is no differentiated protection; without differentiated protection, there is no way to judge whether an online compression tool may be used.

Classification Logic

Drawing on industry practice, enterprise files can generally be divided into four levels:

Public level: Information already published or that can be published, such as website announcements. Processing with online tools poses no compliance risk.
Internal level: Information used internally and not disclosed externally, such as internal notices. Using online tools requires evaluating the provider's qualifications.
Sensitive level: Files containing personal information, trade secrets, or important data, such as customer contracts and financial statements. These must be processed locally; uploading to public-network tools is prohibited.
Core level: Files involving state secrets or core enterprise assets, such as classified documents. These must be processed in an isolated environment using tools that have passed specialized certification.

Data Classification Processing Flow

The diagram below shows the complete classification flow from file identification to selection of a processing method:

The core logic is: first determine whether the file falls within statutory regulation (personal information, important data, state secrets), then determine the level based on scale and cross-border status, and finally match the processing method. For "sensitive" and "core" level files, the processing method is a hard constraint—once the level is determined, local processing is mandatory with no room for debate about online tools.

III. The Three Major Compliance Risks

Risk 1: Cross-Border Data Transfer Risk

When using overseas online file processing tools, file data is transmitted to overseas servers, constituting a cross-border transfer. Under the DSL and PIPL, the export of important data requires passing a security assessment by the national cyberspace administration, and personal information export requires meeting one of the statutory conditions. Under the GDPR (Chapter V), transfers to third countries require appropriate safeguards such as SCCs or an adequacy decision.

Many employees habitually use foreign online tools to process contracts and reports, unaware that these operations may have triggered cross-border transfer obligations the enterprise never fulfilled. Even when using a domestic provider, if backend servers are deployed overseas or overseas CDN and cloud nodes are used, a cross-border transfer may still occur. The criterion is not "the server the user sees is in their country" but "which jurisdiction the data actually reaches."

Risk 2: Third-Party Processing Risk

Online tools typically upload files to third-party servers. Even if the provider promises "immediate deletion after processing," the enterprise still faces three categories of risk:

Data leakage risk: If the provider's security defenses have vulnerabilities, files may be stolen during storage, transmission, or backup. Several well-known online tool providers have suffered data breaches in recent years.
Retention and reuse risk: Some providers' privacy policies reserve the right to use uploaded content for service improvement or model training; file contents may be retained and repurposed.
Lawful compulsion risk: The laws of the provider's jurisdiction may authorize local law enforcement to access server data, without the enterprise's knowledge.

For sensitive files, handing them to an uncontrollable third party inherently expands the attack surface. Even without an actual leak, the mere fact that "data has left the enterprise's controllable boundary" puts the enterprise on the defensive during a compliance audit.

Risk 3: Log Retention Risk

The DSL requires data processors to retain processing logs for no less than six months. The GDPR similarly requires records of processing activities (Article 30). Enterprises must record "who processed which file and when" to enable tracing if a security incident occurs.

If employees use personal online tools to process enterprise files, these operations fall outside the enterprise's logging scope, creating a compliance blind spot: the operation time cannot be recorded, the processing target cannot be identified, and the operator cannot be correlated. Once a leakage incident occurs, the enterprise cannot prove it fulfilled its security management obligations. The essence of this risk is "loss of controllability"—the processing occurs outside the enterprise boundary, and the enterprise can neither obtain the logs nor guarantee their authenticity.

Compliance Risk Assessment Flow

The diagram below shows the decision flow for a compliance risk assessment of file processing operations:

Risk level rises with file level, and for sensitive-level files and above, there is almost no room for "online tools." Enterprises should embed this flow into their tool selection and usage standards rather than relying on individual employees' judgment.

IV. Local vs Online Compression: A Compliance Comparison

From a compliance perspective, local and online compression differ fundamentally. The table below shows the differences across seven dimensions:

Dimension	Local Compression	Online Compression
Data location	Stays on the local device; never leaves the enterprise boundary	Uploaded to third-party servers; leaves controllable scope
Cross-border transfer	Not involved; no export-filing obligation	May constitute a transfer; server location must be assessed
Third-party risk	No third-party involvement; no leakage or retention risk	Provider leakage, retention, and lawful-compulsion risks exist
Log controllability	Enterprise can fully record time, target, and operator	Operations fall outside enterprise logging; compliance blind spot
Network dependency	No network required; processing unaffected by network	Network required; the network link itself is a risk point
Approval & traceability	Can be incorporated into enterprise workflows; fully traceable	Hard to incorporate; poor traceability
Compliance fit	Suitable for all levels from public to core	Only suitable for public-level files; prohibited for sensitive and above

Online compression has compliance shortfalls in four core dimensions—data location, cross-border transfer, third-party risk, and log controllability—while local compression naturally satisfies the legal requirements in all four. This does not mean online compression is entirely unusable; for public-level files, the convenience has value. But for internal-level files and above, local processing is the only reliable way to avoid compliance risk.

V. Industry Compliance Recommendations

Government and State-Owned Enterprises

Government agencies and state-owned enterprises often handle files involving state secrets, with the strictest compliance requirements. All file processing must be done on internal networks or locally; using any public-network online tool is prohibited. Classified files must be processed with tools that have passed national confidentiality certification. Electronic official documents should preferably use the OFD format and be compressed and converted through local tools.

Financial Services

Banks, insurance companies, and securities firms handle files containing large amounts of customer identity and financial information. Compression and conversion of customer data, loan contracts, and account statements should be done locally. Establish a file processing approval workflow recording the operator, time, target, and parameters of each operation, with logs retained for no less than six months. Use a DLP (data loss prevention) system to monitor whether sensitive files are being transmitted externally.

Healthcare

Medical institutions handle medical records and examination reports that constitute sensitive personal information. Compression and archiving of medical record files should be done locally within the hospital's system; using any third-party online tool to process patient information is prohibited. Electronic medical record transmission should use encrypted channels. Patient information files must not be uploaded to any third-party online tool, to avoid triggering cross-border transfer obligations.

VI. File Processing Tool Selection Compliance Checklist

When selecting a file processing tool, check each of the following compliance points:

Does the tool upload data to a server? Prefer local processing tools that keep files on the device. This is the fundamental measure for avoiding cross-border transfer and third-party risk.
Does processing require a network connection? Offline processing is safer; it eliminates the data transmission step.
Is the service provider registered in your jurisdiction? This affects cross-border transfer assessment; overseas providers require verification of backend server location.
Does the tool support file processing log recording? This satisfies the DSL's six-month log retention requirement (and the GDPR's Article 30 obligation). Logs should include time, filename, operator, and parameters.
Has the tool passed relevant security certifications? Certifications such as ISO 27001 or national cybersecurity level protection assessments reflect the provider's security management capability.
Is a data processing agreement signed? This clarifies the data security responsibilities of both parties, defining data usage, retention period, and deletion method.
Does the tool support batch processing and access control? This meets enterprise management and audit needs, supporting role-based permissions and unified logging of batch operations.

Only a tool that passes all of the above checks may be added to an enterprise's whitelist. Any single item failing means the tool may only be used for public-level files.

VII. FAQ

Q1: The online compression tool says "files are deleted immediately after processing." Is there still a compliance risk?

Yes. Even if the provider promises deletion, the data has already left your device during transmission and may be intercepted, retained, or lawfully compelled. The transmission link, the provider's logging system, backup mechanisms, and disaster-recovery nodes can all leave traces. For sensitive files, "upload is risk"—once data leaves the enterprise's controllable boundary, there is no guarantee it will be completely deleted. Sensitive files should always be processed with a local compression tool.

Q2: Is it compliant for an employee to send files processed with an online compression tool via personal email?

Generally, no. Personal email is not managed by the enterprise, so the processing behavior cannot be recorded in enterprise logs, violating log retention requirements. The email may also transit through overseas servers, creating a cross-border transfer risk. Enterprises should explicitly prohibit such behavior through policy, provide enterprise email and local processing tools as alternatives, and use a DLP system to monitor whether sensitive files flow out through personal channels.

Q3: Does the Data Security Law have specific provisions for file compression?

No. The Data Security Law has no provisions specifically targeting file compression, but file compression constitutes a data processing activity and is subject to general requirements such as data classification, secure processing, and log retention. Whether it is compliant depends on three factors: the data level of the file, whether it involves personal information, and whether the processing involves a cross-border transfer. The law does not distinguish "compression" from other processing methods; it only looks at whether the processing meets general security requirements.

Q4: How can a local compression tool meet the log retention requirement?

Enterprises can require local compression tools to provide a processing log feature that records processing time, filename, operator, and compression parameters, with logs retained for no less than six months. A professional tool should support automatic recording and export of logs for security audits. Enterprises should also integrate logs into a unified log management platform, correlated with the identity authentication system to ensure operators are traceable. Note that logs themselves are data processing records and should be managed at the sensitive level to prevent log leakage from creating secondary risk.

Summary

The implementation of the DSL, PIPL, and GDPR has shifted enterprise file processing from "whatever is convenient" to "whatever is compliant." Returning to the original question—is using an online compression tool a compliance risk?—the answer depends on three variables: the file's classification level, the data's flow path, and whether processing is logged.

The core principles can be summarized as three: classify data, prioritize local processing, keep logs traceable.

Data classification is the prerequisite. Without classification, there is no way to judge whether an online tool may be used. Enterprises must first build a file classification ledger.
Local-first is the baseline. For sensitive-level files and above, local processing is the only reliable way to avoid cross-border transfer, third-party, and logging blind-spot risks. Online tools are acceptable only for public-level files.
Traceable logs are the safeguard. All processing activities must be recordable and traceable, with logs retained for no less than six months. This is a necessary means for an enterprise to prove it has met its obligations.

Achieve these three, and the basic compliance framework for enterprise file processing is in place. File compression is just one link in the file processing chain, but it reflects the enterprise's overall data security governance maturity—compliant tool selection, a clear classification system, and a complete logging framework. All three are indispensable.

Related Reading:

Need to Compress Files? Try SmartSlim

Built on a self-developed Rust compression engine, supporting 10 categories and 40+ formats including PDF, images, video, Office, and OFD, with local compression that keeps your data on-premises.

Free Desktop Download

Enterprise Solutions Inquiry

View SDK Documentation

What Is OFD? China's National Document Standard Explained

Uglypear Data — Fri, 31 Jul 2026 03:51:34 +0000

The Definition and Background of OFD

OFD (Open Fixed-layout Document) is a fixed-layout document format standard issued by the Standardization Administration of the People's Republic of China, with the standard number GB/T 33190-2016. Unlike PDF, which was created by Adobe and later became the ISO 32000 international standard, OFD is a national standard independently developed by China, designed to address the issue of autonomous control in the fixed-layout document domain.

The core characteristic of a fixed-layout document is "locked layout"—the document's layout, fonts, and image positions are fixed at creation time, and it renders identically regardless of the device used to open it. This contrasts with "reflowable documents" such as Word and HTML, which reflow based on screen size, font availability, and other factors. Fixed-layout documents lock the layout, ensuring what you see is what you get.

An OFD file uses XML to describe the document structure and is organized as a ZIP package—essentially a compressed container holding multiple XML files and media resources. It supports text, images, graphics, signatures, seals, and animations, and precisely preserves the document's visual layout. The standard was driven toward industrialization by the Fixed-layout Document Industry Alliance (OFD Alliance) and was published as a national recommended standard in 2016.

Since 2020, OFD has been widely adopted in the electronic official document systems of Party and government organs, gradually becoming the de facto standard for fixed-layout documents in the government sector. Today, OFD files are increasingly common in scenarios such as electronic invoices, electronic licenses, and electronic contracts. Understanding the technical system of OFD has become prerequisite knowledge for handling government documents and interfacing with government-enterprise systems.

OFD vs PDF: A Technical Comparison of Two Fixed-Layout Formats

OFD and PDF are both fixed-layout document formats with similar functional positioning—both are used to precisely preserve document layout and ensure cross-device consistency. However, the two differ significantly in their technical systems and ecosystems. Understanding these differences helps in choosing the right format for different scenarios.

Dimension	OFD	PDF
Format Standard	Chinese national standard GB/T 33190-2016	ISO 32000 international standard (originated from Adobe)
File Structure	XML + ZIP container, open structure	Binary/text hybrid, relatively closed structure
Use Cases	Government documents, electronic licenses, bidding	General documents, contracts, reports, publications
Supported Tools	Few, mainly domestic tools like Suke, Foxit	Mature ecosystem, Adobe, Foxit, and many free tools
Seal/Signature Support	Native electronic seals, compliant with Chinese crypto standards	Digital signatures supported, Chinese crypto needs extra adaptation
International Recognition	Primarily domestic use	Globally accepted

The flowchart below compares the technical paths of OFD and PDF across three dimensions: standard origin, file structure, and application ecosystem:

1. Format Standard

OFD is a Chinese national standard with publicly transparent technical specifications. Its document structure is based on XML, making it convenient for domestic adaptation and integration with Chinese cryptographic algorithms (SM2, SM3, SM4). PDF is an international standard with the most mature ecosystem and global adoption, but in classified and government scenarios, there are concerns regarding autonomous control. The two are not in a replacement relationship; they are parallel standards addressing different compliance requirements and application domains.

2. File Structure

This is the most fundamental technical difference between OFD and PDF. An OFD file is essentially a ZIP archive, with XML internally describing pages, text, images, and other elements—clearly structured and easy to parse. PDF uses a binary/text hybrid format, internally composed of objects, cross-reference tables, and trailer structures—relatively complex. For developers, OFD offers stronger readability and parsability; for end users, the visual appearance of both is similar when opened.

3. Supported Tools

PDF's tool ecosystem is extremely rich, ranging from free browser-embedded viewers to professional Adobe Acrobat, with editing, conversion, and compression tools of all kinds. By comparison, OFD's toolchain is still in its early stages of development. The number of software applications that can view and edit OFD is limited, and tools that support OFD compression are even scarcer. This is OFD's main shortcoming in its current ecosystem.

4. Seal and Signature Support

OFD natively supports electronic seals and seal verification compliant with Chinese cryptographic standards. Seal data is embedded as independent nodes in the document structure, meeting the requirements for authenticity, integrity, and non-repudiation in official document workflows. PDF supports digital signatures based on the PKI system, but requires additional development for Chinese cryptographic algorithm adaptation. In government document scenarios, OFD's seal system aligns more closely with the domestic cryptographic infrastructure.

OFD File Structure: The XML + ZIP Container

Understanding the OFD file structure is the foundation for understanding its compression methods and tool ecosystem. Unlike PDF's binary format, OFD adopts an "XML + ZIP container" architecture similar to modern office documents (such as OOXML, ODF).

Container Organization

An OFD file is essentially a ZIP archive. By changing its file extension to .zip and extracting it, you can see a clear multi-level directory structure. The outermost layer contains an OFD.xml entry file, which declares the document's basic information and physical file paths; internally, content is organized in a hierarchy of "Document (Doc)—Page—Object."

The XML Description Layer

OFD uses XML files to describe the document's logical structure and page content. Document.xml defines the document's page tree, outline, metadata, and so on; each page corresponds to a Content.xml that describes the text, images, graphics, paths, and other layout objects on that page along with their coordinates and styles. This XML-based description approach brings several characteristics:

Strong readability: Any text editor can view OFD's page descriptions, facilitating debugging and auditing
Easy to parse: Standard XML parsers can read the document structure without specialized binary parsers
Good extensibility: Adding new features only requires extending the XML Schema without affecting existing structures

Comparison with PDF's Binary Format

PDF uses a binary/text hybrid format, internally composed of numbered objects linked together through a Cross-Reference Table, with the Trailer at the end of the file pointing to the root object. This design has advantages in file compactness, but the structure is opaque—ordinary users cannot directly view the internal content.

The difference between the two is also evident at the compression level: OFD is already a ZIP container, with internal resources having undergone a layer of compression, so re-compressing the whole file yields limited gains—the optimization focus is on internal resources (images, fonts). PDF's compression, on the other hand, is distributed across individual stream objects, where each stream object can independently choose its compression algorithm.

OFD Applications in Government Documents

OFD's promotion in the government sector has a practical background. Traditional government documents are primarily paper-based, and electronic transformation requires a fixed-layout format that precisely preserves layout, supports electronic seals, and complies with national cryptographic management requirements. Although PDF is powerful, as a foreign standard it raises autonomous control concerns in classified information systems. OFD, as a national standard, fills this gap.

Electronic Official Documents

Electronic official document systems in Party and government organs across many regions of China have adopted OFD as the standard format. The entire workflow of drafting, review, signing, distribution, and archiving uses OFD files as the carrier, combined with Chinese cryptographic electronic seals to achieve tamper resistance and traceability. OFD's fixed-layout characteristic ensures that documents maintain consistent layout across different office systems, without misformatting due to missing fonts or software differences.

Electronic Licenses

Electronic business licenses, real estate registration certificates, electronic ID cards, and other electronic licenses are gradually migrating to the OFD format. OFD files can embed license information, holder photos, issuing authority electronic seals, and support anti-forgery verification. Compared to paper licenses, OFD electronic licenses are more convenient for online verification and cross-department sharing.

Electronic Invoices

Electronic invoices in some regions have adopted the OFD format. OFD invoices contain structured data such as invoice codes, amounts, and tax amounts, while preserving the visual layout—convenient for both human reading and machine parsing. The seal mechanism of OFD invoices ensures authenticity and non-repudiation.

As application scale expands, OFD file size has become apparent—large numbers of scanned documents and high-definition seal images embedded in documents cause official documents to reach tens of MB, creating pressure on storage and transmission, and generating demand for OFD compression.

OFD Compression Methods

The principle of OFD compression is highly similar to PDF compression—the core is optimizing the parts that occupy the most space within the file. Since OFD is already a ZIP container whose file structure has undergone a layer of DEFLATE compression, the key to OFD compression is not adding another compression layer, but optimizing internal resources (images, fonts).

Image Resampling

Scanned documents and seal images in OFD files are typically the largest contributors to file size. A scanned official document may embed multiple 300dpi high-definition scanned pages, each several MB. Reducing high-resolution images to a reasonable resolution (e.g., from 300dpi to 150dpi) can significantly reduce size, with an acceptable impact on screen reading and printing.

Resampling requires balancing resolution and clarity: text content in government documents remains clearly legible at 150dpi, but the fine details of precision seals may require higher resolution. A sensible approach is to set different target resolutions based on image purpose.

Font Subsetting

OFD files embed complete fonts to ensure consistent display across devices, but a document typically uses only a small portion of the characters in a font file. Font subsetting retains only the character glyphs actually used in the document, discarding unused font data.

This technique is especially effective for Chinese documents. A complete Chinese font file typically contains tens of thousands of Chinese character glyphs and can exceed 10MB in size; however, a government document may use only one or two thousand Chinese characters, and subsetting can reduce font data by over 80%.

Cleaning Redundant Objects

An OFD package may contain unreferenced resources and duplicate objects: images deleted during editing may still linger in the resource directory, and multiple saves may produce duplicate metadata. Cleaning redundant objects involves traversing the document structure to remove all resources not referenced by pages, streamlining the document structure tree.

Image Format Optimization

OFD supports embedding multiple image formats, including uncompressed bitmaps (BMP) and compressed formats (JPEG, PNG). Converting uncompressed bitmaps to compressed formats like JPEG can directly reduce image resource usage. For seal graphics with large areas of solid color, PNG may be smaller than JPEG; for color-rich scanned photos, JPEG offers a higher compression ratio.

The following code demonstrates how to unzip an OFD file and analyze its internal resource usage—the first step in compression optimization:

import zipfile
import os
from collections import defaultdict

def analyze_ofd(ofd_path):
    """Unzip and analyze the internal resource usage of an OFD file"""
    with zipfile.ZipFile(ofd_path, 'r') as zf:
        # Tally the size of each file type
        type_sizes = defaultdict(int)
        type_counts = defaultdict(int)
        total_size = 0

        for info in zf.infolist():
            ext = os.path.splitext(info.filename)[1].lower()
            type_sizes[ext] += info.compress_size
            type_counts[ext] += 1
            total_size += info.compress_size

        print(f"Total OFD size: {total_size / 1024 / 1024:.2f} MB\n")
        print(f"{'Type':<12} {'Count':<8} {'Size(MB)':<12} {'Share'}")
        print("-" * 45)
        for ext, size in sorted(type_sizes.items(), key=lambda x: -x[1]):
            pct = size / total_size * 100
            print(f"{ext or '(none)':<12} {type_counts[ext]:<8} "
                  f"{size / 1024 / 1024:<12.2f} {pct:.1f}%")

# Usage example
# analyze_ofd("sample.ofd")

Running this kind of analysis script, you will typically find that over 80% of an OFD file's size comes from image resources (.jpg, .png, .bmp), with font files (.ttf, .otf) second. This confirms the core strategy of OFD compression: prioritize image optimization, then font optimization.

A 50MB OFD scanned official document, after reasonable image resampling and font subsetting, can typically be compressed to 8-12MB, with text and seals still clearly legible—fully meeting the requirements for government workflow and archiving.

The Current State of the OFD Tool Ecosystem

Although OFD's status as a national standard is clear, its tool ecosystem is far less mature than PDF's. Currently, software that can view OFD is mainly domestic tools such as the Suke reader and Foxit OFD; browsers do not natively support OFD preview, and tools that can compress OFD are even scarcer.

The weakness of the tool ecosystem brings two practical problems. First, ordinary users who receive an OFD file may not find a suitable tool to open it and need to download a dedicated reader. Second, when an OFD file is too large, the lack of ready-made compression tools means many users can only store or transmit it as-is, bearing storage and bandwidth costs.

From a technical perspective, OFD's open XML + ZIP structure means the barrier to parsing and processing is theoretically lower than PDF's binary format. The lag in the tool ecosystem is more a matter of market promotion and application scale than a technical obstacle. As OFD applications continue to expand in government and finance, the toolchain is expected to gradually mature.

Comparison Table: OFD vs PDF Compression Methods

Compression Method	OFD Applicability	PDF Applicability	Notes
Image Resampling	High	High	Images dominate size in both; reducing resolution is most effective
Font Subsetting	High	High	Especially effective for Chinese docs; can cut font data by 80%+
Cleaning Redundant Objects	Medium	Medium	Remove unreferenced resources and duplicates, streamline structure
Image Format Optimization	High	High	Convert bitmaps to JPEG; choose optimal format per image type
Whole-File ZIP Compression	Low (already ZIP)	Medium	OFD is already a ZIP container; re-compression yields little
Stream Object Compression	N/A	High	PDF-specific; each Stream can independently choose compression

FAQ

Q1: Can OFD files be converted to PDF?

Yes. Some OFD readers and conversion tools support exporting OFD to PDF. However, note that OFD-specific information such as electronic seals may not be fully preserved during conversion, because OFD's Chinese cryptographic seal system and PDF's PKI signature system do not fully correspond. It is advisable to be cautious when converting in scenarios requiring long-term archiving or legal validity—back up before conversion and verify the seal status of the converted document.

Q2: Does OFD compression affect the validity of electronic seals?

Proper OFD compression only optimizes resources such as images and fonts, without modifying seal data or document hash verification information, so it does not affect the validity of electronic seals. However, note that a seal's integrity verification typically covers specific parts of the document—if the compression process modifies content protected by the seal, seal verification will fail. It is recommended to use professional tools and re-verify the seal status after compression to ensure the legal validity of the document is not compromised.

Q3: Do I need to use OFD for ordinary office scenarios?

For general business documents, PDF is sufficient. OFD is mainly used for government documents, electronic licenses, and other official scenarios. If you are not involved in government interfacing, there is no need to deliberately use OFD. However, if you need to exchange electronic official documents with government agencies or submit OFD-format bidding documents, you must use OFD. The two are not in a replacement relationship; the choice depends on compliance requirements and application scenarios.

Q4: Which is smaller in size, OFD or PDF?

There is no absolute answer. File size depends primarily on internal resources (images, fonts) rather than the container format itself. For identical content, OFD's XML description may be slightly larger than PDF's binary description, but both use compression, and the difference is usually negligible. The key factors determining size are image resolution and font embedding strategy, not the format choice. An OFD official document embedding many high-definition scanned pages will be far larger than a plain-text PDF, and vice versa.

Summary

OFD and PDF are both fixed-layout document formats, but they represent different technical paths and application ecosystems:

OFD, as a Chinese national standard (GB/T 33190-2016), adopts an open XML + ZIP container structure, natively supports Chinese cryptographic electronic seals, and holds a stable position in domestic official scenarios such as government documents, electronic licenses, and electronic invoices. Its tool ecosystem is still in early development, but its open structure provides convenience for domestic adaptation and custom development.
PDF, as an international standard (ISO 32000), adopts a binary/text hybrid format, with the most mature tool ecosystem globally, and is irreplaceable in general office scenarios.

The two are highly similar in compression methods—the core is optimizing embedded image and font resources. OFD compression's four key techniques (image resampling, font subsetting, cleaning redundant objects, image format optimization) share the same lineage as PDF compression, because the size composition principles of fixed-layout documents are universal.

The key to understanding OFD is grasping its "XML + ZIP container" essence: an open, parseable, easily adaptable fixed-layout architecture whose compression focus—like PDF's—is on internal resources, not the container itself.

H.264 vs H.265 vs AV1: Video Codec Evolution and Selection Guide

Uglypear Data — Thu, 30 Jul 2026 02:02:36 +0000

1. The Evolution of Video Codecs

The history of video coding stretches back to the late 1980s. From H.261, designed for video conferencing over ISDN lines, to AV1, engineered for ultra-high-definition streaming, codec standards have evolved over nearly three decades. Each generation pursues the same fundamental goal: reproduce the picture using fewer bits while preserving visual quality.

Video compression is essential because raw, uncompressed video is enormous. A ten-second 1080p clip in raw form can consume several gigabytes—untenable for either storage or transmission. Codecs exploit spatial and temporal redundancy within frames to shrink this data to a manageable size. Spatial redundancy refers to the correlation between neighboring pixels within a single frame; temporal redundancy refers to the similarity between consecutive frames. Virtually every modern codec standard revolves around these two kinds of redundancy.

Concretely, an encoder classifies pictures into three frame types: I-frames (intra-coded frames) record a full picture and can be decoded independently, offering the lowest compression but the strongest error resilience; P-frames (predictive frames) store only the differences from a preceding frame; B-frames (bidirectional frames) reference both preceding and following frames, achieving the highest compression. The group of frames between two I-frames forms a GOP (Group of Pictures)—a longer GOP yields better compression but weaker random access and error recovery. This inter-frame prediction framework has been in place since H.261; successive standards primarily refine block partitioning, prediction accuracy, and entropy coding.

H.261 established the block-based coding framework; MPEG-1 extended it to optical disc storage at roughly 1.5 Mbps, sufficient for VCD-quality video; H.263 optimized for low-bitrate communication, maintaining recognizable pictures below 64 kbps; H.264 opened the era of high-definition streaming; H.265 targeted 4K ultra HD; and AV1 represents the next-generation, royalty-free direction. Each generation attempts to improve compression efficiency over its predecessor while keeping encoding complexity within the reach of real-world hardware.

2. Technical Breakdown of Mainstream Codecs

H.264 (AVC)

Released in 2003, H.264 (Advanced Video Coding) uses macroblock partitioning with a 16×16 base block, supporting sub-blocks from 4×4 to 16×16. Motion estimation reaches quarter-pixel precision, and the standard introduces multi-reference frames, in-loop deblocking, Context-Adaptive Variable-Length Coding (CAVLC), and Context-Adaptive Binary Arithmetic Coding (CABAC).

Compared to its predecessor H.263, H.264 roughly doubles compression efficiency—halving the bitrate at equivalent quality. CABAC saves 5–15% more bitrate than CAVLC but demands more computation, so low-end devices often opt for CAVLC. H.264 also supports multiple slices and Flexible Macroblock Ordering (FMO), facilitating parallel processing and error resilience.

H.264's strength is its mature ecosystem: virtually every modern device, browser, and player supports it natively. Its weakness is insufficient compression efficiency for 4K and beyond, resulting in higher bitrates. Thanks to ubiquitous hardware decoding, H.264 also excels in power efficiency on mobile devices, making it the most compatible codec available today.

H.265 (HEVC)

Released in 2013, H.265 (High Efficiency Video Coding) introduces the Coding Tree Unit (CTU), supporting blocks up to 64×64 with adaptive partitioning down to 8×8 based on picture complexity. At equivalent subjective quality, H.265 reduces bandwidth by roughly 40–50% compared to H.264.

Key improvements include larger CTU sizes that boost coding efficiency, especially for large flat regions; 35 intra prediction directions (versus 9 in H.264) for finer texture representation; advanced motion vector merging (Merge) and Advanced Motion Vector Prediction (AMVP) that reduce the encoding overhead of motion information; Sample Adaptive Offset (SAO) that further suppresses ringing artifacts after in-loop filtering; and parallel processing tools such as Tiles and WPP that ease multi-core encoding.

These features make H.265 well-suited for 4K and 8K content distribution, but they also bring significantly higher computational complexity—H.265 encoding can take several times longer than H.264. Larger block sizes can actually increase overhead at low resolutions, so H.265's advantages only fully emerge at higher resolutions.

VP9

Developed by Google and released in 2013 as part of the WebM project, VP9 uses 64×64 superblocks and supports 32 intra prediction modes, along with advanced inter prediction and adaptive quantization. Its compression efficiency is comparable to H.265, and it is completely royalty-free.

VP9 borrows ideas from H.265 but simplifies the syntax and toolset to lower implementation difficulty. It supports up to three reference frames and allows mixing references of different resolutions to handle variable-resolution scenarios. VP9's main advantage is openness: any vendor can implement it without patent concerns. However, its hardware decoding ecosystem is relatively limited, so it is most active on the web, particularly in Chromium-based browsers. VP9 has been widely adopted by major video platforms as an efficient alternative to H.264, performing reliably in web video distribution.

AV1

Released in 2018 by the Alliance for Open Media (AOMedia), AV1 extends previous-generation techniques with 128×128 superblocks and more sophisticated prediction and transform tools, including intra block copy, affine motion compensation, palette mode, and 1/8-pixel motion precision. Compared to H.265, AV1 saves an additional 20–30% in bitrate at equivalent quality—roughly 60% savings versus H.264.

AOMedia was founded jointly by several industry players specifically to circumvent the fragmented patent licensing of H.265. AV1 builds on VP9, incorporates selected advanced tools from H.265, and adds numerous experimental techniques, giving it a far larger toolset than H.265. It is designed to be entirely royalty-free and has been gradually adopted by major streaming platforms. Encoding complexity is extremely high, and real-time encoding still relies on hardware acceleration. The latest GPUs and mobile chips are beginning to integrate AV1 hardware decoding, and hardware-assisted encoding is progressing. Widespread deployment across both on-demand and live streaming is expected over the next few years.

3. Patent Licensing Comparison

Licensing is a decisive factor in codec adoption. The table below compares the licensing situation of four mainstream formats:

Codec	Licensor	Royalty Cost	Complexity
H.264	MPEG-LA unified pool	Tiered, with annual cap	Low
H.265	Multiple pools (MPEG-LA, HEVC Advance, etc.)	High, no unified cap	Very high
VP9	Open source (BSD license)	Free	None
AV1	Alliance for Open Media	Free	None

Licensing is the primary reason H.265 adoption has lagged. H.264 is managed by MPEG-LA with a clear fee structure and a cap, so content providers can deploy it with predictable costs; H.265, by contrast, involves multiple competing patent pools with patent holders spread across different organizations, uncertain fees, and no unified ceiling. According to public reports, some content providers delayed H.265 deployment because they could not estimate licensing costs. This uncertainty has driven some browser vendors and open-source communities toward VP9 and AV1, and it directly motivated the formation of AOMedia, giving rise to a royalty-free open-source codec ecosystem.

4. Scenario-Based Recommendations

Different use cases have very different codec requirements. The table below summarizes recommendations for common scenarios:

Use Case	Recommended Codec	Reason
Email attachment	H.264	Best compatibility, no extra decoder needed
Web embedding	VP9 / H.264	Native browser support; VP9 saves bandwidth
Mobile playback	H.264 / H.265	Hardware decoding ubiquitous, battery-friendly
Streaming delivery	AV1 / H.265	High compression lowers bandwidth cost
Long-term archival	H.265 / AV1	Saves storage while preserving quality
Live streaming	H.264	Fastest encoding, controllable latency

Always verify the decoding capabilities of the target playback environment first, then weigh compression efficiency against encoding cost. For content that must reach the widest possible audience, H.264 remains the fallback; for bandwidth-sensitive distribution, newer codecs should take priority. In practice, many platforms adopt a multi-bitrate, multi-codec strategy: serving H.264 to cover all clients while providing AV1 or H.265 versions for devices that support newer codecs, balancing compatibility and efficiency.

5. FFmpeg Compression Comparison

Using the same 1080p source video, compress with each codec. Lower CRF (Constant Rate Factor) values yield higher quality; similar values generally indicate comparable quality.

# H.264 (CRF 23, medium preset)
ffmpeg -i input.mp4 -c:v libx264 -crf 23 -preset medium -an output_h264.mp4

# H.265 (CRF 28, medium preset)
ffmpeg -i input.mp4 -c:v libx265 -crf 28 -preset medium -x265-params log-level=error -an output_h265.mp4

# VP9 (CRF 32, unrestricted bitrate)
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 32 -b:v 0 -an output_vp9.webm

# AV1 (CRF 30, unrestricted bitrate)
ffmpeg -i input.mp4 -c:v libaom-av1 -crf 30 -b:v 0 -cpu-used 4 -an output_av1.mkv

Parameter notes: -crf controls quality, -preset/-cpu-used balance speed against compression ratio, -b:v 0 lets the encoder allocate bitrate based on CRF, and -an discards audio to focus the comparison on video compression. Note that the CRF scales are not uniform across encoders: H.264's CRF 23 and H.265's CRF 28 are roughly comparable in subjective quality, but not an exact match—rigorous comparisons should rely on objective metrics such as VMAF or SSIM.

In practice, at equivalent subjective quality, H.265, VP9, and AV1 produce progressively smaller files—but encoding time increases accordingly. AV1 encoding can take tens of times longer than H.264 and requires hardware or parallel optimization to be practical. In production batch transcoding environments, encoding speed is often just as important as compression ratio, so many platforms use AV1 only for offline transcoding while keeping H.264 for real-time scenarios.

6. Frequently Asked Questions

Q1: H.265 is technically superior—why has adoption been slower than H.264?

Primarily due to licensing. H.265 involves multiple patent pools with complex, uncapped fee structures, deterring browser vendors and content platforms. Some browser makers explicitly declined to support H.265 natively because of licensing concerns. Limited early hardware decoding support also slowed mobile adoption. Only in recent years have mainstream chips generally integrated H.265 hardware decoding, but web-side support remains inconsistent. By contrast, H.264 licensing is handled centrally by MPEG-LA with predictable fees, making it the most widely deployed codec standard to date.

Q2: When will AV1 fully replace H.265?

AV1 decoding is supported in next-generation hardware and major browsers, but real-time encoding remains computationally expensive. On-demand delivery has begun scaling up, but live streaming demands low encoding latency and still relies mainly on H.264. Widespread use in live streaming is expected only after hardware encoders mature and encoding speed optimizations improve. In the short term, a more likely scenario is AV1 and H.265 coexisting, with adaptive switching based on client decoding capability. AV1 encoding speed is also improving steadily—open-source encoders such as SVT-AV1 have substantially reduced encoding times, and further gains are expected.

Q3: Why is VP9 mostly confined to the web?

VP9's hardware decoding coverage is relatively limited, relying mainly on Chromium-based browsers' software decoding. On desktop players and native mobile apps, H.264 and H.265 enjoy better hardware support and lower power consumption. VP9 never built a cross-platform hardware decoding ecosystem like H.264, so it remains primarily a web video format. However, VP9 paved the way for AV1—many of its technical ideas continue and are upgraded in AV1. Some platforms serve both VP9 and AV1 versions simultaneously, gradually transitioning from VP9 to AV1.

7. Conclusion

Video codec development is a constant balance between compression efficiency and implementation cost. H.264, with its comprehensive compatibility, remains the safest choice today; H.265 excels in bandwidth-sensitive UHD scenarios, though licensing constraints limit adoption; VP9 and AV1 represent the royalty-free direction, with AV1 emerging as the core of next-generation streaming.

When selecting a codec, consider the target playback environment, bandwidth cost, storage capacity, and computational overhead rather than chasing the newest standard. Finding the right balance between compatibility, efficiency, and cost is the engineering optimum. Understanding the applicable boundaries of each codec is more valuable than remembering which one is newest. As hardware support for AV1 matures and encoding tools continue to optimize, the cost gap between generations will narrow—yet the fundamental tradeoffs between compatibility, efficiency, and licensing will persist. Staying informed about these dynamics is the best way to make sound codec decisions over time.

FFmpeg Video Compression: CRF vs CBR vs VBR Explained

Uglypear Data — Wed, 29 Jul 2026 08:08:56 +0000

Bitrate Control: Starting From the Basics

To understand the difference between CRF, CBR, and VBR, we first need to talk about bitrate.

Bitrate is the amount of video data produced per unit of time, usually measured in kbps (kilobits per second) or Mbps (megabits per second). It directly determines the clarity, smoothness, and final file size of a video. With the same resolution, frame rate, and codec, a higher bitrate preserves more detail but produces a larger file, while a lower bitrate shrinks the file at the cost of blur, banding, and blocking artifacts.

So why do we need bitrate control at all? The reason is that the complexity of video frames is constantly changing. Static backgrounds and flat color areas can be reconstructed losslessly with very little data, whereas fast motion, rich textures, and abrupt changes require far more bits to stay sharp. If you force a constant bitrate across the entire video, simple scenes waste storage space, and complex scenes starve for bits and visibly degrade.

The core goal of bitrate control is to allocate bits to each frame intelligently, given a fixed storage or bandwidth budget, and to strike the best balance between quality and size. The three modes, CRF, CBR, and VBR, are essentially three different allocation strategies.

The Three Bitrate Control Modes Explained

CRF (Constant Rate Factor)

The guiding principle of CRF is constant quality. The encoder dynamically allocates bits based on the complexity of each frame: it spends fewer bits on simple scenes and more bits on complex scenes, keeping the perceived quality consistent throughout the video. Whether the frame is a still landscape or an action-packed sequence, the viewer experiences roughly the same visual quality.

CRF is controlled by an integer between 0 and 51 (for the x264 and x265 encoders). A lower value means higher quality and a larger file; a higher value means lower quality and a smaller file. A value of 0 produces lossless encoding, 51 is the lowest quality, and 23 is typically the default visually lossless reference point.

The biggest advantage of CRF is simplicity combined with stable quality. You do not need to set a target bitrate in advance, because the encoder automatically distributes bits according to scene complexity. For this reason, CRF is the preferred mode for most non-real-time tasks such as offline compression, file storage, and web publishing.

CBR (Constant Bitrate)

CBR keeps the bitrate fixed for the entire duration. No matter how complex or simple a scene is, the encoder outputs the same amount of data per second. This means that in simple scenes the allocated bits are redundant or even wasted, while in complex scenes the bits may be insufficient, causing visible artifacts.

The strength of CBR lies in its predictability and buffer friendliness. These properties make it especially suited for live streaming and real-time video transmission, where bandwidth stability is critical. In these scenarios the network bandwidth is often fixed, and large bitrate fluctuations can overflow the buffer or congest the network, causing stutter. CBR sacrifices some quality in exchange for transmission stability.

VBR (Variable Bitrate)

VBR allows the bitrate to fluctuate within a specified range. The encoder weighs scene complexity and a target quality, adjusting the bitrate between a configured minimum and maximum. VBR typically requires setting a target bitrate (or target quality) along with upper and lower bounds, making it more flexible than CBR while adding a bitrate constraint that CRF lacks.

VBR is common in scenarios that need to control overall file size while preserving quality, such as streaming segment encoding. It is worth noting that the x264 encoder also offers a 2-pass VBR mode: the first pass scans the entire video to analyze the complexity distribution, and the second pass allocates bits precisely based on that analysis. This mode controls the target size accurately while optimizing bit distribution, but it takes roughly twice as long as a single-pass encode, making it ideal for offline tasks with strict size requirements.

Comparing Bitrate Fluctuations Across the Three Modes

The chart below visualizes how the three modes behave on the same video. CBR is a flat horizontal line, while CRF and VBR both rise and fall with scene complexity. CRF fluctuates more freely, whereas VBR is bounded by its configured floor and ceiling.

How CRF Value Affects Quality and Size

Choosing the right CRF value is the most important step when using CRF mode. The chart below shows how quality and file size change as the CRF value goes from 18 to 30. A lower CRF means higher quality and a larger file, while a higher CRF means a smaller file at the cost of quality.

FFmpeg Command Examples: Three Common Scenarios

Here are three typical compression scenarios, each addressing a different balance of quality and size.

Scenario 1: Email (H.264 + 720p + CRF 28)

Email attachments usually have size limits, so the video needs to be compressed aggressively. 720p resolution with CRF 28 keeps the file small while remaining watchable.

ffmpeg -i input.mp4 \
  -c:v libx264 -preset medium -crf 28 \
  -vf "scale=-2:720" \
  -c:a aac -b:a 128k \
  output.mp4

Scenario 2: Web Publishing (H.264 + 1080p + CRF 26)

Web video needs to balance clarity with loading speed. 1080p with CRF 26 is a common combination that also offers the widest compatibility.

ffmpeg -i input.mp4 \
  -c:v libx264 -preset medium -crf 26 \
  -vf "scale=-2:1080" \
  -c:a aac -b:a 128k \
  output.mp4

Scenario 3: Archival (H.265 + Original Resolution + CRF 20)

Archival prioritizes quality and compression efficiency. Using the H.265 encoder while keeping the original resolution, CRF 20 achieves near-visually-lossless quality at a smaller size than H.264.

ffmpeg -i input.mp4 \
  -c:v libx265 -preset medium -crf 20 \
  -c:a aac -b:a 192k \
  output.mp4

A quick note on the key parameters: -c:v sets the video codec, where libx264 maps to H.264 and libx265 maps to H.265; -preset trades encoding speed for compression efficiency, where slower presets yield better compression; -crf sets the quality level; -vf "scale=-2:720" scales the height to 720 pixels while computing the width proportionally and keeping it even; and -c:a aac -b:a 128k encodes audio as AAC at 128 kbps.

CRF Value Selection Guide

The table below lists common CRF ranges for quick reference. In practice, start with the middle of the recommended range for your scenario, test-encode a short clip, and fine-tune from there.

CRF Range	Quality Level	File Size	Use Case
18-22	High quality	Large	Archival, source backup, high-quality playback
23-26	Standard	Moderate	Web publishing, on-demand streaming, mobile playback
27-30	Extreme compression	Small	Email, instant messaging, bandwidth-constrained delivery

CRF vs CBR vs VBR Comparison

The table below compares the three modes across quality, size, encoding speed, and use case to help you choose based on your needs.

Aspect	CRF	CBR	VBR
Quality stability	High (consistent throughout)	Low (artifacts in complex scenes)	Medium (bounded by bitrate limits)
File size	Unpredictable	Predictable	Roughly predictable (2-pass more precise)
Encoding speed	Fast (single pass)	Fast	Slower (2-pass about twice as long)
Use case	Offline compression, storage, web publishing	Live streaming, real-time transmission	Streaming delivery, size-constrained tasks

Frequently Asked Questions

1. How do I choose the right CRF value?

There is no universally correct CRF value; it depends on your trade-off between quality and size. A good starting point is CRF 23-26. If the quality looks good, gradually increase the value to shrink the file further; if the quality is insufficient, decrease the value. For archival needs, 18-22 works well, and for size-sensitive delivery, 27-30 is appropriate.

2. What should I do if the compressed video looks blurry?

Blur is usually caused by insufficient bitrate or over-aggressive downscaling. Check a few things: first, lower the CRF value, for example from 28 to 24, to give the encoder more bits; second, verify that you have not shrunk the resolution too much, and raise it if needed; third, switch to a slower preset (such as slow or slower) to trade encoding time for better compression quality; fourth, confirm that you have not applied overly strong denoise or deinterlace filters, which can also soften the image when misconfigured.

3. Should I choose H.264 or H.265?

H.264 has the widest compatibility, supported by virtually all modern devices, operating systems, and browsers, making it ideal for broad distribution such as web and mobile playback. H.265 (HEVC) is more efficient, producing files roughly 30%-50% smaller than H.264 at the same quality, but it encodes more slowly and is not supported by some older devices and browsers. If your goal is archival or distribution to modern devices, prefer H.265; if maximum compatibility is the priority, choose H.264.

Summary

Bitrate control is the heart of video compression, and choosing the right mode often matters more than tuning specific parameters. CRF prioritizes constant quality and is the first choice for offline compression and file storage; CBR prioritizes bitrate stability and suits live streaming and real-time transmission; VBR strikes a balance between the two and fits distribution scenarios with overall size constraints.

In practice, mastering three points covers most needs: choose the bitrate control mode that matches your scenario, pick a sensible CRF value and fine-tune after a test encode, and select between H.264 and H.265 based on compatibility requirements. Combined with the preset knob for trading speed against efficiency, FFmpeg lets you produce videos that balance quality and size across any scenario.