Part 6 of 8 · week 6 of the learning path
A video platform is two pipelines joined by storage: an upload pipeline that turns each file into many playable versions once, and a watch path that serves those versions from caches near the viewer.
Learning objectives
- Estimate storage and bandwidth for a media product and see why they dominate cost.
- Design resumable upload, a parallel transcoding pipeline and adaptive streaming.
- Explain CDN tiers, counters and how a viral video is kept from overwhelming the origin.
1. Requirements
- Functional: upload videos, process them for any device and bandwidth, watch with fast start and smooth playback, search, likes, comments and view counts. Recommendations are acknowledged but out of scope here.
- Non-functional: playback starts in about two seconds, very few stalls, uploads that survive bad networks, high availability, and cost control since storage and bandwidth are the big bills.
2. Estimates. The inputs are public statements that YouTube receives more than 500 hours of video a minute and as of 2017 sees more than a billion hours watched a day; storage per hour is an assumption.
| Quantity | Calculation | Result |
|---|---|---|
| Hours uploaded per day | 500 times 60 times 24 | 720,000 hours |
| Storage per day at an assumed 5 GB per hour across all versions | 720,000 times 5 GB | about 3.6 PB |
| Storage per year | 3.6 PB times 365 | about 1.3 EB |
| Average concurrent viewers | 1 billion hours times 3,600 divided by 86,400 | about 42 million |
| Average egress at 2.5 Mbit per second each | 42 million times 2.5 Mbit per second | about 104 Tbit per second |
| Watching versus uploading | 1 billion divided by 720,000 | about 1,400 hours watched per hour uploaded |
The lesson: nothing can serve over 100 Tbit per second from one place, so almost all traffic must come from caches near users, and the system is overwhelmingly read-heavy.
3. API and data model
- Upload: create an upload session, send numbered chunks, then mark it complete.
- Watch: get video metadata, then get a manifest (HLS or DASH) that lists the segments at each quality.
- Other: search by query, like, comment.
- Data: video record (owner, title, status, duration, available qualities), comments and counters in the metadata store; original files, segments and manifests in object storage.
4. High-level design
Creator app
| (1) resumable upload
v
Upload service --(2)--> Raw storage
|
v (3) transcode job per rendition
Transcode farm
|
v (4) write segments + manifest
Rendition storage Metadata DB
|
v (5) cache fill on first request
CDN edge <-----(6) play: manifest, then segments---- Viewer
Step numbers in the diagram match the list below.
The numbers match the steps below; the highlighted transcode farm is where the upload becomes many playable versions.
- The creator's app uploads the file in chunks and can resume after a failure.
- The upload service writes the original file to raw storage.
- A new-upload event triggers the transcoding farm.
- The farm splits the video into short segments, encodes each at several qualities in parallel, and writes segments and manifests to rendition storage; the metadata store marks the video ready and search is updated.
- The first viewer request pulls segments through the CDN edge cache; later viewers get them from the cache.
- The player downloads the manifest and picks a quality for each segment.
5. Deep dives
- Resumable upload: chunks with checksums, resume from the last acknowledged chunk, and upload straight to storage with signed URLs so upload servers stay light.
- Transcoding pipeline: a graph of tasks (split, encode each quality, package, thumbnails, captions, safety and copyright checks) run by queue-fed workers that autoscale. Tasks are idempotent so a failed one is simply retried. Segment-level parallelism cuts the time to publish.
- Adaptive bitrate streaming: each video is cut into segments of a few seconds at several bitrates, and the player measures throughput and buffer to switch quality per segment.
- CDN tiers: edge cache, then a regional shield, then origin. Popular videos are cached everywhere; the long tail mostly comes from the regional tier or origin. Signed, expiring URLs control access.
- Metadata and counters: a sharded relational store with replicas and a cache for metadata. View counts flow as events into an aggregator, shown approximately and corrected later after fraud filtering.
- Search: an index fed asynchronously from metadata and captions.
- Cost control: encode into more efficient codecs only for videos that are watched enough to pay back, and move cold content to cheaper storage.
6. Trade-offs: more qualities improve playback but cost storage and compute; pre-encoding everything is simple but wasteful for videos nobody watches; eventually consistent counters are cheap but lag.
7. Failures and monitoring. A crashed transcoder's task is retried on another worker; a video that keeps failing goes to a dead-letter queue for review. If a video goes viral and misses the cache at many edges, request coalescing and the shield tier turn thousands of misses into one origin fetch. Watch upload success, time to publish, startup time, rebuffer ratio, CDN hit ratio, queue depth and egress cost.
Real-world notes: YouTube built Vitess to scale MySQL by sharding, and Netflix places its own cache servers inside internet providers' networks; both show how far the read path is pushed toward the viewer.
Design exercise: add live streaming to this design. Say what changes in the pipeline (segments are produced continuously), where latency comes from, and what you would do to protect the origin during a major live event.
Quiz 6 (5 questions, pass mark 4)
- Why can't origin storage serve all viewers directly?
- A) Videos are too short
- B) Disks cannot be read
- C) Average viewing needs on the order of 100 Tbit per second, so most traffic must come from caches near users
- D) Players do not support it
- What does adaptive bitrate streaming do?
- A) Cuts video into short segments at several qualities so the player can switch quality per segment
- B) Compresses audio only
- C) Stores the video in the browser
- D) Encrypts the manifest
- Why split a video into segments before transcoding?
- A) To hide it
- B) To shorten file names
- C) To avoid using storage
- D) So segments can be encoded in parallel across workers, cutting the time to publish
- A video goes viral and misses cache at many edges. What protects the origin?
- A) Delete the video
- B) Request coalescing and a shield tier, so many misses become one origin fetch
- C) Longer segments
- D) Turn caching off
- Why do uploads use chunks with resume?
- A) Large uploads on mobile networks often fail, so only the missing chunks are resent
- B) To encrypt the file
- C) To reduce storage
- D) To skip transcoding
Quiz 6 answers
Sources and further reading
- More than 500 hours of content are now uploaded to YouTube every minute, Tubefilter, 2019, reporting YouTube's figure.
- YouTube viewers watch 1 billion hours a day, CNBC, 2017, reporting YouTube's blog post.
- Vitess history, on how YouTube sharded MySQL.
- Open Connect overview from Netflix, on caches placed inside internet provider networks.
- HTTP Live Streaming from Apple, the reference for segmented adaptive streaming.
The upload and transcoding pipeline is a teaching design built on these facts, not YouTube's published architecture.
Part 6 of 8 in the System Design Learning Path series. Next: part 7.
Top comments (0)