Fireside Chat: Mike Lewis

Cohere For AI Fireside Chats bring together leading researchers and rising stars in the field of machine learning to discuss their research learning journeys. Research is inherently a human endeavour, and this discussion series provides insights from beginning to breakthrough. This Fireside Chat features Mike Lewis, Research Scientist at Meta AI. Ahmet Üstün, Research Scientist at Cohere For AI, will sit down with Mike for a conversation on "Exploring the State of Foundation Models" 00:01 - Introduction 03:06 - How did you get interested in computers, engineering or maybe linguistics? 05:29 - How was your graduate studies in Oxford and Edinburgh? Did you know what you were going to do after those or you kind of also explored as you as you go at the time? 08:01 - So I see that you did a post-doc after the PhD. What was your decision process not going to industry but doing a bit more like a academic research in the university? 10:24 - Tell us about your thoughts on formal Synthetic parsing in the form of categorical grammar with the formal semantics and combining with some distributional semantics. How relevant is this now? 15:24 - What was the starting point for you in experimenting, pre-training, generative language models , ect? 24:30 - You are also working on non-parametric language modelling, knn-languge models and retrieval augmented generations. So where do you position these type of work in LLMs space? 32:00 - How do you see the interest around MoE models and you you think framework such as BTM would be one of the solution at very large scale? 41:02 - In relation to size of the training data set and models. Do you think that the compute is problematic at this point? 44:02 - You say actually pretraining is one of the most important parts, even a really small set of instructions you can actually align pretty well. So maybe can you briefly tell what is your experience with that 47:46 - How do you manage your time and how do you decide on which project or what