Building a Real-Time Photo Gallery That Handles 500+ Simultaneous Uploads

The Problem That Started It All

It started at a hackathon. We built a photo wall — a simple real-time gallery where attendees could upload photos and see them instantly on a big screen. The MVP worked great with 20 people. Then 200 showed up. The server melted. Uploads timed out, the WebSocket connection dropped every few seconds, and the gallery showed more spinners than photos.

That night, I made a decision: I was going to build this properly and document every step. This is that story — the architecture decisions, the dead ends, the unexpected wins, and the final system that handles 500+ simultaneous uploads without breaking a sweat.

Picshots is where this gallery engine ultimately landed — a real-time photo platform that powers live event galleries, community walls, and interactive displays. But the journey to get there was anything but straightforward.

Architecture Overview: What We Built

Here's the high-level architecture we converged on after three iterations:

• Upload Gateway: A dedicated Node.js service that accepts multipart uploads, validates them, and enqueues processing jobs — never blocks on image processing.
• Message Queue: Redis Streams for job distribution. Each upload becomes a job with a unique ID, priority, and metadata payload.
• Worker Pool: Stateless workers (auto-scaled) that consume jobs from Redis, process images (resize, thumbnail, optimize), and upload to object storage.
• Real-Time Layer: WebSocket server (Socket.io) that broadcasts gallery updates to connected clients whenever a new photo finishes processing.
• Object Storage: S3-compatible storage (we used Cloudflare R2 for egress-cost reasons) for original and processed images.
• Database: PostgreSQL for metadata (photo records, gallery state, user associations) with read replicas for the gallery query path. The key insight was separation of concerns: the upload path does nothing but receive and enqueue. The processing path does nothing but transform and store. The delivery path does nothing but query and broadcast. No service in the critical path does more than one thing.

Iteration 1: The Naive Approach (And Why It Failed)

The first version was a single Express server. Upload came in, Multer handled it, Sharp processed it on the same thread, and the result was saved to disk. A Socket.io broadcast told clients to refresh.

At 50 concurrent uploads, response times jumped from 200ms to 8 seconds. At 100, the server was effectively dead — CPU pinned at 100% on image processing, event loop blocked, new connections timing out.

The problem was clear: image processing on the request thread is a death sentence for concurrency. Sharp is fast, but synchronous processing of a 12MB photo (resize, compress, generate thumbnail) takes 150-400ms per image. With 100 uploads hitting simultaneously, that's 15-40 seconds of CPU time queued up with nowhere to go.

Iteration 2: Offloading to a Worker Queue

The second version split the monolith. Uploads went to the Express server, which saved the raw file to a temp directory and pushed a job to Redis. A separate worker process picked up jobs, ran Sharp, uploaded to R2, and updated PostgreSQL.

This was dramatically better — the upload server stayed responsive because it did almost no work. But we hit two new problems:

• WebSocket updates were unreliable. The worker finished processing but had no direct connection to the client. We tried having the worker publish to a Redis pub/sub channel that the web server subscribed to, but messages got lost under load.
• Single worker bottleneck. One worker process could process maybe 3-4 images per second. At 500 concurrent uploads arriving over 10 seconds, that's a 50-image backlog that takes 12-15 seconds to clear. Acceptable, but not great for a "live" gallery experience. The fix for the WebSocket issue was making the real-time layer its own service. A dedicated Socket.io server with Redis adapter (sticky sessions via Redis for horizontal scaling) subscribed to the job completion channel. When a job finished, the worker published a photo:processed event with the photo ID, gallery ID, and CDN URL. The real-time server pushed it to every connected client in that gallery room.

Event photo galleries on Picshots now use exactly this pattern — each gallery is a Socket.io room, and clients only receive updates for the gallery they're viewing.

Iteration 3: Horizontal Scaling and the 500-Upload Target

To hit 500+ simultaneous uploads, we needed horizontal scaling at every layer. Here's what we did:

Upload Gateway Scaling

We ran 3 instances of the upload gateway behind a round-robin load balancer (nginx with least_conn strategy). Each gateway instance was capped at 200 concurrent connections. With 3 instances, that's 600 concurrent upload slots — headroom above our 500 target.

File size limits mattered more than I expected. We capped uploads at 15MB (rejecting anything larger at the nginx layer before it even hit Node). This prevented a single massive upload from consuming connection time and memory.

Worker Pool Auto-Scaling

Workers were deployed as a Docker Swarm service with auto-scaling rules based on Redis stream length. When the pending job count exceeded 20, a new worker container spun up. When it dropped below 5, workers scaled down. We capped at 8 workers to avoid CPU contention on the host.

Each worker processed one image at a time (Sharp is CPU-bound, so parallelism within a worker doesn't help much). With 8 workers averaging 3 images/second each, that's 24 images/second throughput — enough to clear a 500-image burst in about 21 seconds.

Database Connection Pooling

PostgreSQL connection pooling via PgBouncer was non-negotiable. Without it, each worker and gateway instance opened its own connections, and Postgres choked on 40+ simultaneous connections from a 4GB VPS. PgBouncer in transaction mode kept the actual database connections to 12 while serving 40+ clients.

Real-Time Layer

Two Socket.io server instances with the Redis adapter behind nginx (sticky sessions via ip_hash). Each instance handled up to 5,000 concurrent connections. For a gallery with 500 viewers, this was massive overkill — but it meant the real-time layer would never be the bottleneck.

The Image Processing Pipeline

Here's what happens to each photo once a worker picks it up:

1. Validation: Verify the file is actually an image (check magic bytes, not just extension). Reject anything that's not JPEG, PNG, or WebP.
2. Exif stripping: Remove EXIF data for privacy (location coordinates especially) using Sharp's withExif: false.
3. Master resize: Resize the longest edge to 1920px while preserving aspect ratio. This becomes the "full" version displayed in the gallery lightbox.
4. Thumbnail generation: 400px wide thumbnail at 70% JPEG quality for the grid view.
5. Format optimization: Convert to WebP for the thumbnail (40-50% size reduction vs JPEG at similar quality). Keep full version as JPEG for compatibility.
6. Upload to R2: Upload both versions with content-type headers. Use multipart upload for files > 5MB.
7. Database insert: Insert photo record with gallery_id, original_filename, full_url, thumbnail_url, dimensions, file_size, and created_at.
8. Publish event: Publish photo:processed to Redis with the photo payload. The whole pipeline takes 200-500ms per image depending on original size. The bottleneck is almost always the resize step — Sharp's libvips backend is fast, but a 12MP image still takes 150ms+ to resize on a single core.

Live Gallery Updates: The Real-Time Experience

The gallery frontend is a React app with an infinite scroll grid. When a client connects, it joins a Socket.io room named after the gallery ID. The initial load fetches existing photos via a paginated REST endpoint. After that, all updates come through the WebSocket.

When a photo:processed event arrives, the client prepends the new photo to the grid with a fade-in animation. The effect is magical at events — you upload a photo from your phone and watch it appear on the big screen gallery within seconds.

One thing we learned the hard way: don't send the full photo object through the WebSocket. Early on, we sent the base64 thumbnail data in the event payload. With 50 new photos arriving in a burst, that was 50 × ~60KB = 3MB of data hitting every connected client simultaneously. Now we send only the CDN URL and metadata — clients fetch the thumbnail image from the CDN, which handles concurrent requests far better than a WebSocket connection.

Real-time photo sharing on Picshots follows exactly this pattern — metadata over WebSocket, image bytes over CDN.

Handling Failure Gracefully

At 500+ concurrent uploads, things will fail. The question isn't how to prevent failure — it's how to handle it without degrading the experience.

• Upload failures: Client-side retry with exponential backoff (1s, 2s, 4s, max 3 attempts). If all retries fail, show a retry button on the failed upload.
• Worker crashes: Redis Streams consumer groups with pending entries lists (PEL) ensure unprocessed jobs are re-delivered to another worker. We set an acknowledgement timeout of 60 seconds — if a worker doesn't ACK a job in 60s, it's considered dead and the job is requeued.
• Database failures: Workers retry DB inserts 3 times with 1s backoff. If all retries fail, the job is moved to a dead letter queue for manual inspection. The photo doesn't appear in the gallery, but the upload isn't lost.
• WebSocket disconnections: Clients auto-reconnect with backoff. On reconnect, they fetch any photos they missed via a REST endpoint that accepts ?after_timestamp= as a query parameter.

Performance Numbers

Here's where we landed after all three iterations:

• Upload throughput: 500 concurrent uploads accepted and enqueued in under 8 seconds.
• Processing latency: P50 of 3.2 seconds from upload completion to photo appearing in the gallery. P95 of 12 seconds under full load.
• WebSocket fan-out: 500 connected clients receive a gallery update in under 50ms (Redis adapter + Socket.io rooms).
• Memory per worker: ~80MB (Sharp's libvips is memory-efficient for single-image processing).
• Total infrastructure cost: ~$45/month for a setup that handles 500 concurrent uploads (3 gateway instances, 2-8 workers, Redis, Postgres, R2 storage).

What I'd Do Differently

If I were starting over, three things would change:

1. Use a managed queue from day one. We started with in-process queueing, then moved to Redis lists, then to Redis Streams. If we'd started with BullMQ or even SQS, we'd have saved two weeks of rework on retry logic and dead letter handling.
2. Stream uploads directly to object storage. We save the raw upload to disk, then the worker reads it from disk. If we'd streamed directly from the upload gateway to R2 (presigned URLs), we'd eliminate the disk I/O bottleneck entirely. This is on the roadmap.
3. Measure earlier. We didn't add proper monitoring (Prometheus + Grafana) until iteration 3. Having P50/P95 latency dashboards from the start would have caught the event-loop blocking problem in iteration 1 much faster.

FAQ

Why Redis Streams instead of RabbitMQ or Kafka?

Redis Streams gave us consumer groups, pending entry tracking, and dead letter handling with minimal infrastructure. We already needed Redis for Socket.io's adapter, so using it for the queue too meant one less system to operate. For 500 uploads/second, Redis is more than sufficient. Kafka would make sense at 50,000+/second.

How do you handle duplicate uploads?

We generate a hash of the file content (SHA-256) on the upload gateway before enqueuing. The worker checks this hash against PostgreSQL before processing. If a match exists, the worker skips processing and returns the existing photo URL. This handles accidental double-submits from flaky mobile connections.

What about image moderation?

We integrated AWS Rekognition's content moderation API as an optional pipeline step. When enabled for a gallery, each processed photo is scanned before publishing. Photos flagged with high confidence are held for manual review. This adds ~200ms per image but is worth it for public-facing galleries.

Can this architecture handle 5,000+ uploads?

The architecture scales horizontally — more gateway instances, more workers, bigger Redis instance. The real constraint at 5,000+ is the database write path. We'd likely need to batch insert photo records or move to a write-optimized store (like ClickHouse) for the metadata layer. The queue and processing layers would scale fine.

Wrapping Up

Building a real-time photo gallery that handles 500+ simultaneous uploads taught me more about system design than any tutorial ever could. The journey from a single Express server that died at 50 uploads to a horizontally-scaled architecture handling 500+ was three months of trial, error, and incremental improvement.

The principles that got us there are universal: separate the hot path from the slow path, use queues to absorb bursts, scale workers horizontally, keep WebSocket payloads tiny, and handle failure explicitly at every layer. If you're building something similar, I hope this saves you a few of the dead ends I hit.

The gallery engine I built is now powering live photo experiences on Picshots. If you want to see it in action or have questions about the architecture, reach out — I love talking about this stuff.