Imbue - training a 70B model from scratch! (w/ Bowei - head of infra)

The AI Epiphany ยท Advanced ยท๐Ÿง  Large Language Models ยท1y ago
Become a Patreon: https://www.patreon.com/theaiepiphany ๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆ Join our Discord community: https://discord.gg/peBrCpheKE Bowei joined us from Imbue to talk about Imbue's latest endeavor: building the infra to support a 70B model training! They wrote up an amazing blog post with a lot of details describing the grind that was needed to set everything up from scratch. :) โ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌ Blog post: https://imbue.com/research/70b-infrastructure/ (From bare metal to a 70B model: infrastructure set-up and scripts) โ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌโ–ฌ โŒš๏ธ Timetable: 00:00 - 00:45 Intro 00:45 - โ€ฆ
Watch on YouTube โ†— (saves to browser)

Chapters (7)

00:45 Intro
0:45 02:25 Hyperstack GPUs (sponsored)
2:25 11:30 Bowei's background
11:30 18:30 More on Imbue, their research, their focus
18:30 26:20 Training a 70B model
26:20 45:40 Building a cluster from scratch
45:40 59:25 Anecdotes, Q&A

Playlist

Uploads from Aleksa Gordiฤ‡ - The AI Epiphany ยท Aleksa Gordiฤ‡ - The AI Epiphany ยท 0 of 60

โ† Previous Next โ†’
1 Intro | Neural Style Transfer #1
Intro | Neural Style Transfer #1
Aleksa Gordiฤ‡ - The AI Epiphany
 2 Basic Theory | Neural Style Transfer #2
Basic Theory | Neural Style Transfer #2
Aleksa Gordiฤ‡ - The AI Epiphany
 3 Optimization method | Neural Style Transfer #3
Optimization method | Neural Style Transfer #3
Aleksa Gordiฤ‡ - The AI Epiphany
 4 Advanced Theory | Neural Style Transfer #4
Advanced Theory | Neural Style Transfer #4
Aleksa Gordiฤ‡ - The AI Epiphany
 5 Anyone can make deepfakes now!
Anyone can make deepfakes now!
Aleksa Gordiฤ‡ - The AI Epiphany
 6 What is Computer Vision? | The Art of Creating Seeing Machines
What is Computer Vision? | The Art of Creating Seeing Machines
Aleksa Gordiฤ‡ - The AI Epiphany
 7 Feed-forward method | Neural Style Transfer #5
Feed-forward method | Neural Style Transfer #5
Aleksa Gordiฤ‡ - The AI Epiphany
 8 Alan Turing | Computing Machinery and Intelligence
Alan Turing | Computing Machinery and Intelligence
Aleksa Gordiฤ‡ - The AI Epiphany
 9 Feed-forward method (training) | Neural Style Transfer #6
Feed-forward method (training) | Neural Style Transfer #6
Aleksa Gordiฤ‡ - The AI Epiphany
 10 What is Google Deep Dream? (Basic Theory) | Deep Dream Series #1
What is Google Deep Dream? (Basic Theory) | Deep Dream Series #1
Aleksa Gordiฤ‡ - The AI Epiphany
 11 Semantic Segmentation in PyTorch | Neural Style Transfer #7
Semantic Segmentation in PyTorch | Neural Style Transfer #7
Aleksa Gordiฤ‡ - The AI Epiphany
 12 How to get started with Machine Learning
How to get started with Machine Learning
Aleksa Gordiฤ‡ - The AI Epiphany
 13 How to learn PyTorch? (3 easy steps) | 2021
How to learn PyTorch? (3 easy steps) | 2021
Aleksa Gordiฤ‡ - The AI Epiphany
 14 PyTorch or TensorFlow?
PyTorch or TensorFlow?
Aleksa Gordiฤ‡ - The AI Epiphany
 15 3 Machine Learning Projects For Beginners (Highly visual) | 2021
3 Machine Learning Projects For Beginners (Highly visual) | 2021
Aleksa Gordiฤ‡ - The AI Epiphany
 16 Machine Learning Projects (Intermediate level) | 2021
Machine Learning Projects (Intermediate level) | 2021
Aleksa Gordiฤ‡ - The AI Epiphany
 17 Cheapest (0$) Deep Learning Hardware Options | 2021
Cheapest (0$) Deep Learning Hardware Options | 2021
Aleksa Gordiฤ‡ - The AI Epiphany
 18 How to learn deep learning? (Transformers Example)
How to learn deep learning? (Transformers Example)
Aleksa Gordiฤ‡ - The AI Epiphany
 19 How do transformers work? (Attention is all you need)
How do transformers work? (Attention is all you need)
Aleksa Gordiฤ‡ - The AI Epiphany
 20 Developing a deep learning project (case study on transformer)
Developing a deep learning project (case study on transformer)
Aleksa Gordiฤ‡ - The AI Epiphany
 21 Vision Transformer (ViT) - An image is worth 16x16 words | Paper Explained
Vision Transformer (ViT) - An image is worth 16x16 words | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 22 GPT-3 - Language Models are Few-Shot Learners | Paper Explained
GPT-3 - Language Models are Few-Shot Learners | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 23 Google DeepMind's AlphaFold 2 explained! (Protein folding, AlphaFold 1, a glimpse into AlphaFold 2)
Google DeepMind's AlphaFold 2 explained! (Protein folding, AlphaFold 1, a glimpse into AlphaFold 2)
Aleksa Gordiฤ‡ - The AI Epiphany
 24 Attention Is All You Need (Transformer) | Paper Explained
Attention Is All You Need (Transformer) | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 25 Graph Attention Networks (GAT) | GNN Paper Explained
Graph Attention Networks (GAT) | GNN Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 26 Graph Convolutional Networks (GCN) | GNN Paper Explained
Graph Convolutional Networks (GCN) | GNN Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 27 Graph SAGE - Inductive Representation Learning on Large Graphs | GNN Paper Explained
Graph SAGE - Inductive Representation Learning on Large Graphs | GNN Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 28 PinSage - Graph Convolutional Neural Networks for Web-Scale Recommender Systems | Paper Explained
PinSage - Graph Convolutional Neural Networks for Web-Scale Recommender Systems | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 29 OpenAI CLIP - Connecting Text and Images | Paper Explained
OpenAI CLIP - Connecting Text and Images | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 30 Temporal Graph Networks (TGN) | GNN Paper Explained
Temporal Graph Networks (TGN) | GNN Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 31 Graph Neural Network Project Update! (I'm coding GAT from scratch)
Graph Neural Network Project Update! (I'm coding GAT from scratch)
Aleksa Gordiฤ‡ - The AI Epiphany
 32 Graph Attention Network Project Walkthrough
Graph Attention Network Project Walkthrough
Aleksa Gordiฤ‡ - The AI Epiphany
 33 How to get started with Graph ML? (Blog walkthrough)
How to get started with Graph ML? (Blog walkthrough)
Aleksa Gordiฤ‡ - The AI Epiphany
 34 DQN - Playing Atari with Deep Reinforcement Learning | RL Paper Explained
DQN - Playing Atari with Deep Reinforcement Learning | RL Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 35 AlphaGo - Mastering the game of Go with deep neural networks and tree search | RL Paper Explained
AlphaGo - Mastering the game of Go with deep neural networks and tree search | RL Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 36 DeepMind's AlphaGo Zero and AlphaZero | RL paper explained
DeepMind's AlphaGo Zero and AlphaZero | RL paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 37 OpenAI - Solving Rubik's Cube with a Robot Hand | RL paper explained
OpenAI - Solving Rubik's Cube with a Robot Hand | RL paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 38 MuZero - Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model | RL Paper explained
MuZero - Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model | RL Paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 39 EfficientNetV2 - Smaller Models and Faster Training | Paper explained
EfficientNetV2 - Smaller Models and Faster Training | Paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 40 Implementing DeepMind's DQN from scratch! | Project Update
Implementing DeepMind's DQN from scratch! | Project Update
Aleksa Gordiฤ‡ - The AI Epiphany
 41 MLP-Mixer: An all-MLP Architecture for Vision | Paper explained
MLP-Mixer: An all-MLP Architecture for Vision | Paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 42 DeepMind's Android RL Environment - AndroidEnv
DeepMind's Android RL Environment - AndroidEnv
Aleksa Gordiฤ‡ - The AI Epiphany
 43 When Vision Transformers Outperform ResNets without Pretraining | Paper Explained
When Vision Transformers Outperform ResNets without Pretraining | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 44 Non-Parametric Transformers | Paper explained
Non-Parametric Transformers | Paper explained
Aleksa Gordiฤ‡ - The AI Epiphany
 45 Chip Placement with Deep Reinforcement Learning | Paper Explained
Chip Placement with Deep Reinforcement Learning | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 46 Text Style Brush - Transfer of text aesthetics from a single example | Paper Explained
Text Style Brush - Transfer of text aesthetics from a single example | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 47 Graphormer - Do Transformers Really Perform Bad for Graph Representation? | Paper Explained
Graphormer - Do Transformers Really Perform Bad for Graph Representation? | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 48 GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation | Paper Explained
GANs N' Roses: Stable, Controllable, Diverse Image to Image Translation | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 49 VQ-VAEs: Neural Discrete Representation Learning | Paper + PyTorch Code Explained
VQ-VAEs: Neural Discrete Representation Learning | Paper + PyTorch Code Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 50 VQ-GAN: Taming Transformers for High-Resolution Image Synthesis | Paper Explained
VQ-GAN: Taming Transformers for High-Resolution Image Synthesis | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 51 Multimodal Few-Shot Learning with Frozen Language Models | Paper Explained
Multimodal Few-Shot Learning with Frozen Language Models | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 52 Focal Transformer: Focal Self-attention for Local-Global Interactions in Vision Transformers
Focal Transformer: Focal Self-attention for Local-Global Interactions in Vision Transformers
Aleksa Gordiฤ‡ - The AI Epiphany
 53 AudioCLIP: Extending CLIP to Image, Text and Audio | Paper Explained
AudioCLIP: Extending CLIP to Image, Text and Audio | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 54 RMA: Rapid Motor Adaptation for Legged Robots | Paper Explained
RMA: Rapid Motor Adaptation for Legged Robots | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 55 DALL-E: Zero-Shot Text-to-Image Generation | Paper Explained
DALL-E: Zero-Shot Text-to-Image Generation | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 56 Channel Update: vacation, leaving Microsoft, approaching 10k subs and more!
Channel Update: vacation, leaving Microsoft, approaching 10k subs and more!
Aleksa Gordiฤ‡ - The AI Epiphany
 57 DETR: End-to-End Object Detection with Transformers | Paper Explained
DETR: End-to-End Object Detection with Transformers | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 58 DINO: Emerging Properties in Self-Supervised Vision Transformers | Paper Explained!
DINO: Emerging Properties in Self-Supervised Vision Transformers | Paper Explained!
Aleksa Gordiฤ‡ - The AI Epiphany
 59 DeepMind DetCon: Efficient Visual Pretraining with Contrastive Detection | Paper Explained
DeepMind DetCon: Efficient Visual Pretraining with Contrastive Detection | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
 60 Do Vision Transformers See Like Convolutional Neural Networks? | Paper Explained
Do Vision Transformers See Like Convolutional Neural Networks? | Paper Explained
Aleksa Gordiฤ‡ - The AI Epiphany
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Next Up
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)