System Design: Design YouTube

Key Takeaways

Upload video, watch video. Sounds simple. YouTube processes 500 hours of video every minute. That's 30,000 hours of content uploaded every hour. The engineering behind this interface is complex. Let's design YouTube. Not the billion user version, but a smaller system that we can learn from. This video covers the core components. We'll focus on video upload and streaming. There's much more to YouTube. search, recommendations, comments, monetization. Each could be its own deep dive. Today, we'll build a foundation you can extend. Here are the requirements we'll focus on. People upload massive videos. A 10-minute 4K video ranges from 1.5 GB highly compressed to 30 GB in ProRes. When uploading huge videos, users expect resumable uploads. For viewing, playback must adapt to network condition in real time. When your connection drops from 25 megabit per second to 2 megabit per second, playback shouldn't stop. The upload challenge reveals the first design decision. We could route large videos through our API servers, configure streaming instead of buffering, increase time out from 30 seconds to 30 minutes, handle partial uploads is all solvable. But why make our servers handle gigabytes of pass through traffic when they could be serving actual API requests? There's a better pattern. use pre-signed URLs. Our API server generates a temporary signed URL that grants direct upload permission to blob storage. The client uploads straight to storage. Our server stay free to handle other work. Blob storage gives us another benefit. We get multiport uploads built in. The client splits the videos into chunks. Each chunk is 5 to 10 megabyte. It's small enough to upload quickly on slower connections. Large enough to minimize overhead. Each chunk get a SH 256 fingerprint. The client uploads chunks in parallel. Six chunks uploading simultaneously is common. This pattern appears in Dropbox, Google Drive, and backup services. Any system handling large files uses similar approaches. The processing pipeline presents a different challenge. Video transcoding burns massive compute cycles. User uploads videos in many different formats. iPhone record in HGVC. Android phones uses H.264. Someone uploads a 4K Pro file from Final Cut Pro. We need these videos playable on every device. Old Android phones run ancient Android. Smart TVs from 2018, web browsers that haven't updated in years. The solution is a processing pipeline that converts one video into many versions. We generate multiple resolutions from 2160p down to 240p. We need so many because network conditions and device speeds vary widely. Someone on fiber with a fast device needs 4K. Someone on 3G with an ancient phone needs 240p. Then consider codecs. H264 works everywhere but uses more bandwidth. VP9 saves bandwidth but older devices can decode it. AV1 saves even more bandwidth but needs powerful hardware. We encode in all three. Next, we package these in containers. MP4 for maximum compatibility. Webb for web optimization. Now one upload becomes 15 to 20 files. How do we process efficiently? Model the workflow as a DAG. DAG stands for the directed as cyclic graph. Each processing step is a note. Dependencies are edges. The asyclic part matters. It ensures task can complete without circular dependencies. First split the video into segments. Videos have key frames every two to 10 seconds. These frames send alone without referencing others. Split a key frames now use segment processes independently. The workflow splits into multiple streams. Video, audio, and metadata each take their own path through the system. Video segments fan out to hundreds of workers while one machine transcodes segment one to 1080p. Another handles segment 2 to 720p. Audio processing runs in parallel on different hardware. Thumbnail generation and subtitle extraction happen on their own dedicated workers. This is the power of modeling work as a DAG. One video becomes hundreds of parallel tasks across a worker farm. As each task completes, results flows to the next stage. Sequential processing would take hours. Parallel processing completes in minutes. Now streaming. Modern video streaming uses adaptive bitray streaming. The video player doesn't download one file. It downloads segments, small chunks of videos, each a few seconds long. When network bandwidth is high, the player fetches 1080p segment. When bandwidth drops, it switches to 480p segments. The transition is usually seamless. This work through manifest files. The primary manifest lists all available formats. Each format then has its own media manifest with URLs for every segment. The player reads manifests, monitors bandwidth using the download speed of the recent segments, and fetches the probate segments. All this happened invisibly. The player makes HTTP range requests. Give me 1,000 to 2,000 of segment five of the 720p version. This enables instance seeking to jump to minute 47. The player calculates which segments to fetch. These segments are stores in CDN's, content delivery networks. Popular videos cache across edge servers worldwide. A viewer in Tokyo get segments from Tokyo, not California. Geographic proximity means lower latency, better streaming. We've just scratched the surface. Several areas deserve deeper exploration. The hot video problems challenges every video platform. When one video goes viral, millions request it simultaneously. We need metadata caching and database hotspot prevention. Cost optimization requires trade-offs. Do we transcode everything immediately or popular formats first? Should rare formats use ondemand transcoding? When do we migrate to code storage? Pipeline optimization can improve latency. Stop processing segments as they arrive instead of waiting for complete upload. Pipeline to upload and processing for faster availability. We could explore geographic CDN placement, readwrite ratios or lazy transcoding strategies. Each topic could be its own deep dive, but the fundamentals remain the same. What makes this design work at scale? Direct uploads keep servers free for actual logic. DAX transforms sequential bottlenecks into parallel workflow. Adaptive streaming ensures smooth playback regardless of network conditions. Once we understand these principles, we see them everywhere. Large file sharing, machine learning pipelines, and live streaming all build on these same foundations. Ready to ace your next technical interview? Join our community where we offer comprehensive courses on system design, coding, behavioral questions, machine learning, and object-oriented design. Learn more at bitebico.com.