Summary
Highlights
The video introduces the ambitious project of connecting five Mac Studios to form a super powerful AI cluster. The goal is to run the largest AI models, specifically the Llama 3.1 405B model, which typically requires cloud-based super powerful AI clusters. The host aims to achieve this with Mac Studios, leveraging new software called XO Labs for AI clustering.
The video explains AI model parameters, where 'B' stands for billions of parameters, representing learned knowledge. More parameters mean a smarter model but also require more resources, especially Video RAM (VRAM). Various Llama models are discussed, from 1B (requiring 4GB VRAM) to 70B (requiring 48GB VRAM), and the ultimate target, Llama 3.1 405B, which demands 1TB of VRAM, traditionally needing high-end NVIDIA GPUs like H100s or A100s.
The concept of quantization is introduced, which allows larger AI models to fit on smaller GPUs by reducing precision. Mac Studios, with their unified memory architecture where system and GPU memory are shared, offer a potential advantage. The host speculates that five Mac Studios with 64GB of RAM each could provide 320GB of shared GPU-addressable RAM, offering a cost-effective and power-efficient alternative to traditional GPU setups despite Apple's MLX not being as mature as NVIDIA's CUDA.
The video highlights networking as a critical bottleneck for AI clusters. Initially, 10 Gigabit Ethernet is used, but it's noted that enterprise AI networking often uses 400-800 Gigabit per second connections. The Mac Studios split model downloads, and efficient communication is vital. Thunderbolt (40 Gigabit per second) is explored as a faster alternative, offering direct PCIe access, though it introduces its own challenges in clustering multiple devices.
The host demonstrates the installation of XO Labs, which simplifies AI clustering by allowing automatic discovery of nodes. Initial tests with a single Mac Studio running a 1B Llama model achieve about 117 tokens per second. However, clustering all five Macs with 10 Gigabit Ethernet for the same model significantly degrades performance to 29 tokens per second, confirming the networking bottleneck.
A sponsored segment for NordVPN discusses its benefits: anonymity online by masking IP addresses, accessing geo-restricted content (like different Netflix libraries), and enhancing security on public Wi-Fi with threat protection and ad blocking capabilities.
Transitioning to Thunderbolt networking, the host sets up a spoke-and-hub configuration. While slightly improving performance over Ethernet, the cluster still experiences bottlenecks. Running the Llama 3.3 70B model on a single Mac Studio is slow, taking 15 seconds to the first token. When distributed across five Mac Studios via 10 Gigabit Ethernet, it achieves about 15 tokens per second, and with Thunderbolt, it reaches 11 tokens per second, indicating network limitations persist.
The host attempts to run the Llama 3.1 405B model, which is extremely large. Running it on a single Mac Studio quickly causes high RAM usage and swap memory engagement, rendering it impractical. When distributed across five Mac Studios via 10 Gigabit Ethernet, the model takes a long time to load but eventually generates text at a mere 0.8 tokens per second. Trying it with Thunderbolt doesn't yield better performance, solidifying networking as the main constraint.
The video briefly compares XO Labs' performance with Ollama, a popular tool for running local AI models. Ollama shows significantly better performance on a single Mac Studio running the 70B model, utilizing the full GPU without swap. The host ponders creating another video to cluster NVIDIA-based PCs. The video concludes that while XO Labs is very cool for Mac and MLX, networking is still a major bottleneck, and more work is needed to optimize performance.