Andrej Jovanovic -Communication efficient training for foundation models through federated learning

Training at large scales, as is the case with today’s frontier foundation models, poses a significant engineering challenge. It is no longer sufficient to train on a single node, but instead we require multi-node setups, often across different data centres, to orchestrate training in a “divide and conquer fashion”. Methods such as DDP, TP, PP and variants thereof are frequently employed to manage the computational requirements for training constrained by a given budget. Whilst these solutions solve the memory overhead (to an extent), the amount of communication overhead increases substantially with the number of nodes in the configuration due to both inter-GPU and inter-data centre communication costs. However, a new paradigm of training, derived from the federated learning literature, has emerged as a promising, communication-efficient solution whilst not comprising on performance. Essentially, these methods alleviate communication costs by communicating infrequently. In this talk, I hope to introduce you to the promise of federated learning for large scale training, starting from seminal works, to state-of-the-art systems that are competitive with their centralised training counterparts. Andrej Jovanović is currently a research assistant and an incoming PhD student, with the machine learning systems group (CaMLSys), supervised by Prof. Nic Lane, at the University of Cambridge. His research interests are centred around distributed, privacy-preserving, efficient and collaborative machine learning. In particular, he is interested in paradigms such as federated learning and neural network compression, and how these interact with optimisation theory. This session is brought to you by the Cohere Labs Open Science Community - a space where ML researchers, engineers, linguists, social scientists, and lifelong learners connect and collaborate with each other. We'd like to extend a special thank you to Harsha Nelaturu, Viraat Aryabum, Srishti Gurejai and Bhavnick Minhas