DeepSpeed ZeRO Tutorial: Fine-Tune LLMs Across Multiple GPUs
About this lesson
In this video, we walk through how to fine-tune a 3B parameter language model across multiple GPUs using DeepSpeed ZeRO. You'll learn how to: - Configure DeepSpeed for multi-GPU training - Understand the trade-offs between ZeRO-2 and ZeRO-3 - Fix out-of-memory errors by sharding model states - Use Hugging Face Trainer with DeepSpeed - Evaluate performance and memory utilization - Compare runtime between DDP, ZeRO-2, and ZeRO-3 Timestamps: 0:00 - Intro: What we’ll cover in this DeepSpeed demo 0:28 - ZeRO-1, ZeRO-2, ZeRO-3: What they shard and why it matters 1:11 - Overview of config files for ZeRO-2 and ZeRO-3 2:30 - Running full fine-tuning with ZeRO-2 (OOM error) 4:22 - Switching to ZeRO-3 and successful training 5:07 - Comparing training time and performance 6:03 - Does full fine-tuning outperform QLoRA? 7:02 - Memory analysis and why ZeRO-2 failed Watch this if you’re training large language models that don’t fit on a single GPU, and want to avoid expensive hardware upgrades. This demo shows how DeepSpeed helps shard memory across devices and make full fine-tuning feasible even with limited resources. This video is part of the LLM Engineering and Deployment Certification Program by Ready Tensor. ✅ Enroll Now: https://app.readytensor.ai/certifications/llm-engineering-and-deployment-DAROCXlj About Ready Tensor: Ready Tensor helps AI/ML professionals build and evaluate intelligent, goal-driven systems and showcase them through certifications, competitions, and real-world project publications. 🌐 Learn more: https://www.readytensor.ai/ 👍 Like the video? Subscribe and tell us what you'd like to see next.
DeepCamp AI