Summary
Highlights
This section introduces the challenges of deploying AI systems to production and discusses how to containerize an AI application for Kubernetes deployment using AWS. It assumes basic knowledge of Docker, Kubernetes, and automated deployment pipelines, setting the stage for a patient healthcare history bot use case.
The video highlights the significant difference in artifact size between traditional backend applications and AI applications. AI applications require not only code and dependencies but also ML frameworks (like PyTorch, Nvidia CUDA) and substantial model weights, which can be tens of gigabytes.
A key strategy discussed is keeping model weights out of the Docker image, storing them in mounted volumes or caches. This reduces image size and speeds up deployment but introduces challenges in managing distribution, integrity, and cold start latency. This approach also facilitates A/B testing of different models.
This part focuses on Kubernetes deployment challenges for AI, including startup time due to model pulling, and the critical role of GPUs. It explains why GPUs are essential for AI workloads (parallel matrix multiplications) and contrasts their cost and performance with CPUs, noting the 100x cost difference for equivalent AI serving capacity.
The video details Kubernetes rolling update configurations (max surge, max unavailable) to ensure no downtime and handle cold start times during updates. This strategy allows new pods to become ready before old ones are terminated, preventing service disruption.
This section covers setting memory and CPU requests and limits for AI pods in Kubernetes. It emphasizes the importance of balancing efficient resource placement with cost optimization, warning against over-provisioning which can lead to higher cloud bills. Monitoring real usage with tools like Prometheus is recommended for iterative tuning.
The discussion moves to defining application readiness and liveliness using Kubernetes probes. Readiness probes determine if a pod should receive traffic, while liveness probes check if a pod is alive or needs restarting. The video illustrates a detailed timeline of probe execution and failure thresholds.
HPA is introduced as a mechanism to scale pods based on metrics like CPU utilization and requests per second. This automated scaling helps manage fluctuating loads, saving costs by scaling down when not in use and scaling up during high demand.
The concept of warm pods is explained as a crucial optimization for LLM serving. Warm pods are pre-scheduled, with containers started, dependencies ready, and model weights loaded into memory (RAM or GPU VRAM), significantly reducing cold start latency. This is achieved by setting a minimum number of replicas in HPA.
The video concludes by discussing blue-green and canary deployment strategies for AI models. Blue-green deployment involves deploying a new version (V2) alongside the old (V1) and then switching traffic entirely. Canary deployment gradually shifts traffic to the new version. Shadow traffic, where requests are mirrored to the new version without affecting user responses, is also mentioned as a safe testing method.