Processing 10,000+ Wedding Photos: My Image Pipeline

Processing 10,000+ Wedding Photos: My Image Pipeline Architecture

Quick Answer: A scalable image pipeline for wedding photography uses asynchronous job queues, multi-format conversion (WebP/AVIF), intelligent thumbnail generation, and CDN edge caching to process 10,000+ photos efficiently while maintaining visual quality and fast delivery.

Introduction

Wedding photography generates massive amounts of image data. A single wedding with two photographers can easily produce 5,000–10,000 RAW files. When I built WedPlanner, a platform for wedding vendors, I quickly realized that naively uploading full-resolution JPEGs to S3 wasn't going to cut it.

Photographers needed fast gallery previews. Couples wanted to share images instantly. And mobile users on 3G connections couldn't wait for 15MB files to load. The challenge wasn't just storage—it was building a pipeline that could process, optimize, and deliver images at scale without breaking the budget or the user experience.

This is the architecture I ended up with, built in public over six months of iteration.

Why Image Processing at Scale Is Hard

Before diving into the solution, let's understand the problem. Wedding photos present unique challenges:

• Volume: 3,000–15,000 images per event
• Resolution: 24MP–45MP cameras produce enormous files
• Formats: RAW, JPEG, HEIC, and now JPEG XL
• Use cases: Full-resolution downloads, web galleries, thumbnails, social sharing
• Latency expectations: Users expect galleries to load in under 2 seconds Traditional approaches like processing images synchronously during upload fail spectacularly at this scale. Uploading 10,000 images and waiting for each to process before showing a confirmation screen? That's a recipe for timeouts and abandoned uploads.

The Pipeline Architecture

My pipeline follows a fan-out pattern with four distinct stages:

Stage 1: Upload Buffer

Images land in a temporary S3 bucket via presigned URLs. This decouples upload from processing and lets photographers start uploading immediately. No waiting, no blocking. The client gets an instant "Upload complete" message while the heavy lifting happens asynchronously.

I use S3 multipart uploads for files over 5MB, which dramatically improves reliability on flaky connections—essential when photographers upload from venue Wi-Fi.

Stage 2: Queue and Fan-Out

Once uploaded, a Lambda function triggers from S3 event notifications and drops a message into an SQS queue. From there, the work fans out to multiple parallel processors:

• Thumbnail generator: Creates 5 sizes (150px, 300px, 600px, 1200px, 2000px wide)
• Format converter: Generates WebP and AVIF variants for each size
• Metadata extractor: Pulls EXIF data (camera settings, GPS, timestamps)
• Duplicate detector: Uses perceptual hashing (pHash) to catch burst-mode duplicates This fan-out pattern is critical. Processing 10,000 images sequentially would take hours. Running each image through four parallel jobs reduces total processing time to roughly 20–30 minutes.

Stage 3: Optimization Strategy

Raw JPEGs from professional cameras are surprisingly unoptimized. I apply several techniques:

Progressive JPEG encoding: Loads a low-resolution preview first, then fills in detail. Essential for perceived performance.

Chroma subsampling 4:2:0: Reduces file size by ~30% with minimal visual impact on web viewing.

MozJPEG: Facebook's JPEG encoder produces files 10–20% smaller than standard libjpeg at the same quality. For galleries viewed on phones and laptops, quality 85 with MozJPEG hits the sweet spot.

WebP and AVIF: These modern formats deliver 25–50% smaller files than JPEG. I generate both and serve them via content negotiation—the browser requests what it supports.

Here's a real comparison from our production pipeline:

FormatAverage Size (24MP)QualityBrowser SupportJPEG (baseline)8.2 MB85UniversalJPEG (MozJPEG)6.1 MB85UniversalWebP4.3 MB8594%+AVIF3.1 MB8085%+The AVIF savings are real, but encoding is CPU-intensive—about 4x slower than WebP. I generate AVIF asynchronously and serve it as a progressive enhancement.

Stage 4: CDN and Intelligent Delivery

All processed images live in a final S3 bucket with CloudFront in front. But CDN alone isn't enough—you need smart delivery logic.

I built a small Node.js service that acts as an image origin. It handles:

• Content negotiation: Inspects Accept headers to serve WebP/AVIF when supported
• Responsive sizing: Uses URL parameters (?w=600) to serve the closest pre-generated size
• Client hints: DPR and Width hints from supported browsers eliminate guesswork
• Lazy loading prep: Generates blurhash placeholders for instant visual feedback Blurhash is particularly effective for wedding galleries. Users see a colorful approximation of the image immediately while the full version loads. It feels instant even on slow connections.

Storage Strategy and Cost Control

Storing 10,000 wedding photos across multiple formats gets expensive fast. My tiered approach:

1. Hot storage (S3 Standard): Original RAWs and active gallery images for 90 days
2. Warm storage (S3 Intelligent-Tiering): Frequently accessed thumbnails and medium sizes
3. Cold storage (S3 Glacier): Originals after 90 days, accessible within minutes For a typical 8,000-image wedding, this breaks down to roughly:

• RAW files ( backup only): ~120 GB → Glacier after 90 days
• Full-resolution JPEGs: ~40 GB → Standard for 90 days
• Thumbnails and web sizes: ~15 GB → Intelligent-Tiering
• CDN cache: ~50% hit rate, reducing origin requests significantly Total first-month storage cost: approximately $12–18 per wedding. After 90 days, that drops to under $5 with Glacier.

Thumbnails: The Secret Performance Weapon

Thumbnails are where most image pipelines fall short. Serving a 2,000px wide image when the user only needs 300px wastes bandwidth and kills perceived performance.

My thumbnail strategy generates five sizes at upload time:

• Micro (150px): Grid previews, fast initial render
• Small (300px): Mobile gallery view
• Medium (600px): Tablet and desktop thumbnails
• Large (1200px): Lightbox preview
• XLarge (2000px): Full-screen viewing, retina displays I use the <picture> element with srcset to let browsers choose the optimal size. This single change improved our Lighthouse performance score from 62 to 94.

One subtle optimization: thumbnail sharpening. Downsampling destroys fine detail. Applying unsharp mask during resize (radius 0.3, amount 80%) keeps edges crisp without looking artificial.

Common Mistakes I Made (So You Don't Have To)

Mistake 1: Processing during upload

My first iteration processed images synchronously. A photographer uploading 500 images would stare at a progress bar for 20 minutes. Users hated it. Moving to async queue processing was the single biggest UX improvement.

Mistake 2: Ignoring memory limits

Sharp (the Node.js image library I use) loads entire images into memory. Processing a 45MP RAW requires 200+ MB per image. Running 50 concurrent Lambda functions caused OOM kills. I now cap concurrency at CPU cores × 2 and use streams where possible.

Mistake 3: Forgetting about color profiles

Professional photographers use Adobe RGB or ProPhoto RGB. Serving these to web browsers without conversion to sRGB causes washed-out colors on most displays. Always convert to sRGB for web delivery—photographers will notice if you don't.

Mistake 4: Not handling orientation metadata

iPhone photos store orientation in EXIF rather than actually rotating pixels. Libraries like Sharp handle this automatically, but forgetting to check caused some of our early thumbnails to appear sideways. Always read and apply EXIF orientation.

Technology Stack

Here's what actually powers this pipeline:

• Sharp (libvips): Fast Node.js image processing. Handles resize, format conversion, and optimization.
• AWS Lambda + SQS: Event-driven processing with automatic scaling.
• S3 + CloudFront: Storage and CDN with custom origin logic.
• Redis: Job status tracking and deduplication.
• PostgreSQL: Image metadata and gallery organization. Sharp is the workhorse here. It's approximately 4x faster than ImageMagick for typical resize operations and has excellent memory efficiency. The npm package sees over 3 million weekly downloads, which speaks to its reliability.

Performance Results

After implementing this pipeline, our metrics improved dramatically:

• Gallery load time: 8.2s → 1.4s (mobile, 3G)
• Time to first image visible: 4.1s → 0.3s (blurhash)
• Storage cost per wedding: $45 → $12 (first month)
• Upload completion rate: 73% → 97%
• CDN hit ratio: 67% (warm cache), 89% (after 24 hours) Most importantly, photographer churn dropped by 40%. When the tool actually works reliably, people stick around.

Key Takeaways

• Process images asynchronously using job queues—never block uploads with synchronous processing
• Generate multiple sizes and formats upfront; content negotiation at delivery time eliminates waste
• Use blurhash placeholders for instant perceived performance on slow connections
• Tier your storage strategy: hot for active content, cold for archives
• Always convert color profiles to sRGB and respect EXIF orientation for web delivery
• Monitor memory usage carefully; image processing is memory-intensive and can crash servers
• Sharp + AWS Lambda provides a cost-effective, scalable foundation for image pipelines
• Thumbnails are not optional—invest in multiple sizes and responsive image markup

Frequently Asked Questions

How long does it take to process 10,000 wedding photos?

With parallel processing across multiple Lambda workers, a typical pipeline processes 10,000 images in 20–30 minutes. This depends on how many sizes and formats you generate. Generating only JPEG thumbnails takes 10–15 minutes; adding WebP, AVIF, and EXIF extraction extends this.

What's the best image format for web galleries?

AVIF offers the best compression (25–50% smaller than JPEG), but WebP has broader browser support and faster encoding. The practical answer: serve WebP as your primary format with JPEG fallback, and add AVIF as progressive enhancement for supported browsers.

How much does image processing infrastructure cost?

For 10,000 images per wedding with 5 thumbnail sizes and 2 formats, expect roughly $12–18 in storage costs during the first 90 days. Processing costs via AWS Lambda range from $5–15 per wedding depending on concurrency settings. CDN costs add $3–8 for typical gallery traffic.

Should I store RAW files or just JPEGs?

Store RAW files if photographers require them for re-editing, but move them to cold storage immediately. Most wedding photography platforms only need JPEGs for gallery delivery. RAW files are 3–5x larger and unnecessary for web viewing.

What's the difference between Sharp and ImageMagick?

Sharp uses libvips under the hood and is significantly faster (up to 4x) with lower memory usage than ImageMagick. Sharp also has a simpler API and better Node.js integration. ImageMagick supports more obscure formats, but for standard web processing, Sharp is the better choice.

How do you handle duplicate photos from burst mode?

I use perceptual hashing (pHash) to generate a fingerprint for each image. Images with similar hashes are flagged as potential duplicates. For burst-mode shots, I keep the sharpest image (determined by variance of Laplacian) and offer the rest as a "similar photos" group.

Is JPEG XL worth implementing?

Not yet. JPEG XL offers excellent compression but lacks browser support (only Safari as of 2026). Until Chrome and Firefox add support, stick with WebP and AVIF. Monitor caniuse.com for JPEG XL adoption.

How do you prevent storage costs from growing forever?

Implement lifecycle policies: delete thumbnails after galleries expire, archive originals to Glacier after 90 days, and use S3 Intelligent-Tiering for unpredictable access patterns. Most wedding photos see 80% of their lifetime views in the first 30 days.

What's the ideal thumbnail size for wedding galleries?

Generate at least five sizes: 150px for grids, 300px for mobile, 600px for tablets, 1200px for lightboxes, and 2000px for retina displays. Use responsive images with srcset so browsers download only what they need.

Can this pipeline handle video files too?

This pipeline focuses on images. Video requires different tools (FFmpeg) and much more processing power. If you need video, run a separate pipeline. Mixing video into image queues causes unpredictable processing times and can stall your entire pipeline.

Conclusion

Building an image pipeline that handles 10,000+ wedding photos isn't about finding one magic tool—it's about combining the right pieces with smart architecture decisions. Asynchronous processing, format optimization, intelligent thumbnails, and tiered storage work together to create a system that's fast for users and affordable to operate.

If you're building something similar, start simple. Get uploads working with async processing first. Add formats and optimizations later. The biggest wins come from not blocking users during upload and serving appropriately sized images.

Want to see how this pipeline fits into a complete wedding photography platform? Check out WedPlanner for more backend architecture articles, or browse our photography tips archive and tech stack deep-dives for related reads.

Have questions about image processing at scale? Drop them in the comments—I read every one.