Energy-Based Transformers explained | How EBTs and EBMs work

AI Coffee Break · Advanced ·📄 Research Papers Explained ·7mo ago
I always wanted to know how energy-based models (EBMs) work. In this video, we break down EBMs — what they are, how they work, and how they’re different from standard neural networks. ► Then we zoom in on the Energy-Based Transformers (EBTs) paper by Gladstone et al. 2025, showing how the authors combined EBMs with transformers to create models that can refine their guesses, self-verify, and potentially adapt how much computation they use depending on the difficulty of the problem. AI Coffee Break Merch! 🛍️ https://aicoffeebreak.creator-spring.com/ Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏 Vignesh Valliappan, Ivan Janov, Sunny Dhiana, Andy Ma 📃 Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, and Tariq Iqbal. "Energy-Based Transformers are Scalable Learners and Thinkers." (2025 https://arxiv.org/abs/2507.02092 ) Outline: 00:00 Energy-Based Transformers 00:47 EBT paper 01:07 Energy-based models explained 03:56 EBM training 06:37 EBM inference 07:55 Energy-based LLM 09:07 Stabilising Training via Noise 09:59 Replay Buffer 10:34 Randomising Step Size 10:26 Results 12:54 Energy-based Transformers for other Modalities ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕ Patreon: https://www.patreon.com/AICoffeeBreak Ko-fi: https://ko-fi.com/aicoffeebreak Join this channel as a Bean Member to get access to perks: https://www.youtube.com/channel/UCobqgqE4i5Kf7wrxRxhToQA/join ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ 🔗 Links: AICoffeeBreakQuiz: https://www.youtube.com/c/AICoffeeBreak/community Twitter / X: https://twitter.com/AICoffeeBreak LinkedIn: https://www.linkedin.com/in/letitia-parcalabescu/ Threads: https://www.threads.net/@ai.coffee.break Bluesky: https://bsky.app/profile/aicoffeebreak.bsky.social Reddit: https://www.reddit.com/r/AICoffeeBreak/ YouTube: https://www.youtube.com/AICoffeeBreak Substack: https://aicoffeebreakwl.substack.c
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from AI Coffee Break with Letitia · AI Coffee Break with Letitia · 0 of 60

← Previous Next →
1 AI Coffee Break - Channel Trailer
AI Coffee Break - Channel Trailer
AI Coffee Break with Letitia
 2 How to check if a neural network has learned a specific phenomenon?
How to check if a neural network has learned a specific phenomenon?
AI Coffee Break with Letitia
 3 A brief history of the Transformer architecture in NLP
A brief history of the Transformer architecture in NLP
AI Coffee Break with Letitia
 4 Our paper at CVPR 2020 - MUL Workshop and ACL 2020 - ALVR Workshop
Our paper at CVPR 2020 - MUL Workshop and ACL 2020 - ALVR Workshop
AI Coffee Break with Letitia
 5 The Transformer neural network architecture EXPLAINED. “Attention is all you need”
The Transformer neural network architecture EXPLAINED. “Attention is all you need”
AI Coffee Break with Letitia
 6 Transformer combining Vision and Language? ViLBERT - NLP meets Computer Vision
Transformer combining Vision and Language? ViLBERT - NLP meets Computer Vision
AI Coffee Break with Letitia
 7 Pre-training of BERT-based Transformer architectures explained – language and vision!
Pre-training of BERT-based Transformer architectures explained – language and vision!
AI Coffee Break with Letitia
 8 GPT-3 explained with examples. Possibilities, and implications.
GPT-3 explained with examples. Possibilities, and implications.
AI Coffee Break with Letitia
 9 Adversarial Machine Learning explained! | With examples.
Adversarial Machine Learning explained! | With examples.
AI Coffee Break with Letitia
 10 BERTology meets Biology | Solving biological problems with Transformers
BERTology meets Biology | Solving biological problems with Transformers
AI Coffee Break with Letitia
 11 Can a neural network tell if an image is mirrored? – Visual Chirality
Can a neural network tell if an image is mirrored? – Visual Chirality
AI Coffee Break with Letitia
 12 The ultimate intro to Graph Neural Networks. Maybe.
The ultimate intro to Graph Neural Networks. Maybe.
AI Coffee Break with Letitia
 13 Can language models understand? Bender and Koller argument.
Can language models understand? Bender and Koller argument.
AI Coffee Break with Letitia
 14 GANs explained | Generative Adversarial Networks video with showcase!
GANs explained | Generative Adversarial Networks video with showcase!
AI Coffee Break with Letitia
 15 What nobody tells you about MULTIMODAL Machine Learning! 🙊 THE definition.
What nobody tells you about MULTIMODAL Machine Learning! 🙊 THE definition.
AI Coffee Break with Letitia
 16 Multimodal Machine Learning models do not work. Here is why. Part 1/2 – The SYMPTOMS
Multimodal Machine Learning models do not work. Here is why. Part 1/2 – The SYMPTOMS
AI Coffee Break with Letitia
 17 Why Multimodal Machine Learning models do not work. Part 2/2 – The CAUSES
Why Multimodal Machine Learning models do not work. Part 2/2 – The CAUSES
AI Coffee Break with Letitia
 18 An image is worth 16x16 words: ViT | Vision Transformer explained
An image is worth 16x16 words: ViT | Vision Transformer explained
AI Coffee Break with Letitia
 19 AI understanding language!? A roadmap to natural language understanding.
AI understanding language!? A roadmap to natural language understanding.
AI Coffee Break with Letitia
 20 "What Can We Do to Improve Peer Review in NLP?" 👀
"What Can We Do to Improve Peer Review in NLP?" 👀
AI Coffee Break with Letitia
 21 The curse of dimensionality. Or is it a blessing?
The curse of dimensionality. Or is it a blessing?
AI Coffee Break with Letitia
 22 PCA explained with intuition, a little math and code
PCA explained with intuition, a little math and code
AI Coffee Break with Letitia
 23 Data-efficient Image Transformers EXPLAINED! Facebook AI's DeiT paper
Data-efficient Image Transformers EXPLAINED! Facebook AI's DeiT paper
AI Coffee Break with Letitia
 24 OpenAI's DALL-E explained. How GPT-3 creates images from descriptions.
OpenAI's DALL-E explained. How GPT-3 creates images from descriptions.
AI Coffee Break with Letitia
 25 Leaking training data from GPT-2. How is this possible?
Leaking training data from GPT-2. How is this possible?
AI Coffee Break with Letitia
 26 OpenAI’s CLIP explained! | Examples, links to code and pretrained model
OpenAI’s CLIP explained! | Examples, links to code and pretrained model
AI Coffee Break with Letitia
 27 Transformers can do both images and text. Here is why.
Transformers can do both images and text. Here is why.
AI Coffee Break with Letitia
 28 UMAP explained | The best dimensionality reduction?
UMAP explained | The best dimensionality reduction?
AI Coffee Break with Letitia
 29 NVIDIA Jarvis (now NVIDIA Riva) meets Ms. Coffee Bean
NVIDIA Jarvis (now NVIDIA Riva) meets Ms. Coffee Bean
AI Coffee Break with Letitia
 30 Transformer in Transformer: Paper explained and visualized | TNT
Transformer in Transformer: Paper explained and visualized | TNT
AI Coffee Break with Letitia
 31 [RANT] Adversarial attack on OpenAI’s CLIP? Are we the fools or the foolers?
[RANT] Adversarial attack on OpenAI’s CLIP? Are we the fools or the foolers?
AI Coffee Break with Letitia
 32 Pattern Exploiting Training explained! | PET, iPET, ADAPET
Pattern Exploiting Training explained! | PET, iPET, ADAPET
AI Coffee Break with Letitia
 33 Deep Learning for Symbolic Mathematics!? | Paper EXPLAINED
Deep Learning for Symbolic Mathematics!? | Paper EXPLAINED
AI Coffee Break with Letitia
 34 FNet: Mixing Tokens with Fourier Transforms – Paper Explained
FNet: Mixing Tokens with Fourier Transforms – Paper Explained
AI Coffee Break with Letitia
 35 Are Pre-trained Convolutions Better than Pre-trained Transformers? – Paper Explained
Are Pre-trained Convolutions Better than Pre-trained Transformers? – Paper Explained
AI Coffee Break with Letitia
 36 "Please Commit More Blatant Academic Fraud" – A fellow PhD student's response.
"Please Commit More Blatant Academic Fraud" – A fellow PhD student's response.
AI Coffee Break with Letitia
 37 Scaling Vision Transformers? How much data can a transformer get? #Shorts
Scaling Vision Transformers? How much data can a transformer get? #Shorts
AI Coffee Break with Letitia
 38 How cross-modal are vision and language models really? 👀 Seeing past words. [Own work]
How cross-modal are vision and language models really? 👀 Seeing past words. [Own work]
AI Coffee Break with Letitia
 39 Charformer: Fast Character Transformers via Gradient-based Subword Tokenization +Tokenizer explained
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization +Tokenizer explained
AI Coffee Break with Letitia
 40 Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.
Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.
AI Coffee Break with Letitia
 41 Adding vs. concatenating positional embeddings & Learned positional encodings
Adding vs. concatenating positional embeddings & Learned positional encodings
AI Coffee Break with Letitia
 42 Self-Attention with Relative Position Representations – Paper explained
Self-Attention with Relative Position Representations – Paper explained
AI Coffee Break with Letitia
 43 Saddle points vs. local minima in high dimensional spaces | ❓ #AICoffeeBreakQuiz #Shorts
Saddle points vs. local minima in high dimensional spaces | ❓ #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 44 What is the model identifiability problem? | Explained in 60 seconds! | ❓ #AICoffeeBreakQuiz #Shorts
What is the model identifiability problem? | Explained in 60 seconds! | ❓ #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 45 Data leakage during data preparation? | Using AntiPatterns to avoid MLOps Mistakes
Data leakage during data preparation? | Using AntiPatterns to avoid MLOps Mistakes
AI Coffee Break with Letitia
 46 Is today's AI smarter than YOU? #Shorts
Is today's AI smarter than YOU? #Shorts
AI Coffee Break with Letitia
 47 Convolution vs Cross-Correlation. How most CNNs do not compute convolutions. | ❓ #Shorts
Convolution vs Cross-Correlation. How most CNNs do not compute convolutions. | ❓ #Shorts
AI Coffee Break with Letitia
 48 Why do we care about cross-correlations vs convolutions | ❓ #AICoffeeBreakQuiz #Shorts
Why do we care about cross-correlations vs convolutions | ❓ #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 49 The convolution is not shift invariant. | Invariance vs Equivariance | ❓ #AICoffeeBreakQuiz #Shorts
The convolution is not shift invariant. | Invariance vs Equivariance | ❓ #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 50 How to increase the receptive field in CNNs? | #AICoffeeBreakQuiz #Shorts
How to increase the receptive field in CNNs? | #AICoffeeBreakQuiz #Shorts
AI Coffee Break with Letitia
 51 What is tokenization and how does it work? Tokenizers explained.
What is tokenization and how does it work? Tokenizers explained.
AI Coffee Break with Letitia
 52 Foundation Models | On the opportunities and risks of calling pre-trained models “Foundation Models”
Foundation Models | On the opportunities and risks of calling pre-trained models “Foundation Models”
AI Coffee Break with Letitia
 53 How modern search engines work – Vector databases explained! | Weaviate open-source
How modern search engines work – Vector databases explained! | Weaviate open-source
AI Coffee Break with Letitia
 54 Eyes tell all: How to tell that an AI generated a face?
Eyes tell all: How to tell that an AI generated a face?
AI Coffee Break with Letitia
 55 Swin Transformer paper animated and explained
Swin Transformer paper animated and explained
AI Coffee Break with Letitia
 56 Data BAD | What Will it Take to Fix Benchmarking for NLU?
Data BAD | What Will it Take to Fix Benchmarking for NLU?
AI Coffee Break with Letitia
 57 SimVLM explained | What the paper doesn’t tell you
SimVLM explained | What the paper doesn’t tell you
AI Coffee Break with Letitia
 58 Generalization – Interpolation – Extrapolation in Machine Learning: Which is it now!?
Generalization – Interpolation – Extrapolation in Machine Learning: Which is it now!?
AI Coffee Break with Letitia
 59 Do Transformers process sequences of FIXED or of VARIABLE length? | #AICoffeeBreakQuiz
Do Transformers process sequences of FIXED or of VARIABLE length? | #AICoffeeBreakQuiz
AI Coffee Break with Letitia
 60 The efficiency misnomer | Size does not matter | What does the number of parameters mean in a model?
The efficiency misnomer | Size does not matter | What does the number of parameters mean in a model?
AI Coffee Break with Letitia

Related AI Lessons

⚡
The ABCs of reading medical research and review papers these days
Learn to critically evaluate medical research papers by accepting nothing at face value, believing no one blindly, and checking everything
Medium · LLM
 ⚡
#1 DevLog Meta-research: I Got Tired of Tab Chaos While Reading Research Papers.
Learn to manage research paper tabs efficiently and apply meta-research techniques to improve productivity
Dev.to AI
 ⚡
How to Set Up a Karpathy-Style Wiki for Your Research Field
Learn to set up a Karpathy-style wiki for your research field to organize and share knowledge effectively
Medium · AI
 ⚡
The Non-Optimality of Scientific Knowledge: Path Dependence, Lock-In, and The Local Minimum Trap
Scientific knowledge may be stuck in a local minimum, hindering optimal progress, and understanding this concept is crucial for advancing research
ArXiv cs.AI

Chapters (11)

Energy-Based Transformers
0:47 EBT paper
1:07 Energy-based models explained
3:56 EBM training
6:37 EBM inference
7:55 Energy-based LLM
9:07 Stabilising Training via Noise
9:59 Replay Buffer
10:34 Randomising Step Size
10:26 Results
12:54 Energy-based Transformers for other Modalities
Up next
Microsoft Research Forum | Season 2, Episode 4
Microsoft Research
Watch →