Summary
Highlights
The video begins by introducing the complexity of designing a YouTube-like application, highlighting that users can upload and watch videos, unlike platforms such as Netflix. While YouTube has many features (search, recommendations, comments, advertising), the core functional requirements for this design will be uploading and watching videos.
Key non-functional requirements include reliability (videos should not be corrupted or deleted), high availability over strong consistency (accepting slightly stale data for continuous service), and low latency for video playback. The discussion estimates supporting a billion daily active users, with a ratio of 100 video views to 1 upload, leading to 50 million video uploads and 5 billion video watches per day. Most views are concentrated on a small percentage of videos.
Uploading a video starts with a user request routed through a load balancer to application servers. Videos are stored in object storage (like AWS S3) for reliability and handling large files. Metadata for each video (title, description, tags, user info, reference to the video file) is stored in a NoSQL database (e.g., MongoDB). Data is denormalized to favor reads, potentially duplicating user info within video documents, with asynchronous updates for changes like profile pictures.
Video encoding is an asynchronous task. Raw uploaded videos are added to a message queue and processed by a dedicated encoding service, which can scale horizontally. Encoded videos are then stored back in object storage. This ensures reliability and proper formatting for playback.
The video discusses capacity planning for encoding workers. With 50 million uploads per day, averaging 500 uploads per second, and assuming each video takes one minute to encode, more than 500 workers are needed. It's calculated that approximately 30,000 workers would be required to handle the volume and prevent backlog in the queue.
To optimize video watching, encoded videos are distributed globally via a Content Delivery Network (CDN) to ensure low latency. Video metadata is fetched rapidly using an in-memory cache (like an LRU cache) in front of the database, as a small percentage of videos receive most views. The video demonstrates how YouTube streams videos in small chunks (using HTTP requests built on TCP) rather than downloading the entire file at once, allowing for instant playback and skipping without buffering. Audio and video are often fetched separately.
The video explains why TCP is favored for video streaming over UDP for pre-recorded content. While UDP is better for live streaming due to its speed, TCP ensures reliability and that all parts of the video are received without gaps, which is crucial for watching a complete video. YouTube uses HTTP requests, which are built on TCP.
Other aspects like rate limiting uploads, search, and recommendation systems are mentioned as needing auxiliary services. The video then delves into YouTube's historical use of MySQL (a relational database) rather than NoSQL. Due to initial scale and eventual growth, YouTube implemented read-only replicas and sharding, leading to complex application-level routing. This led to the development of Vitess, an open-source database clustering system that decouples the application layer from sharding logic, allowing MySQL to scale to YouTube's demands.