Lightning Talk: Optimizing AI Workload Scheduling at Scale: Practical Lessons Using Kueu... P. Matam

Lightning Talk: Optimizing AI Workload Scheduling at Scale: Practical Lessons Using Kueue on Kubernetes - Prashanthi Matam As large-scale AI training and inference workloads increasingly run on shared Kubernetes clusters, efficient job queuing and resource management have become critical for performance, cost, and fairness. Kueue, an open-source job queueing and workload orchestration system for Kubernetes, enables organizations to manage compute-intensive ML workloads more intelligently — reducing idle GPU time while improving cluster utilization. This talk explores how Kueue integrates with existing Kubernetes batch systems (like Job, RayJob, or custom CRDs) to provide enterprise-grade workload management. Drawing from real-world deployment patterns, we’ll examine how Kueue helps orchestrate AI training jobs, preempt lower-priority runs, and align scheduling with business or research priorities. Key Takeaways: Understand what Kueue is, and how it extends native Kubernetes scheduling for AI/ML workloads. Learn to configure queues, cohorts, and resource flavors to optimize GPU utilization. Discover how to integrate Jupyter notebooks and Kubeflow pipelines with Kueue for managed job submission. Explore real-world examples of Kueue in production and common pitfalls when scaling distributed workloads.