Applied Deep Learning 2025 - Lecture 8 - Transformers

Alexander Pacha · Beginner ·🧬 Deep Learning ·7mo ago

About this lesson

In this lecture, we're diving into Transformers, which not only sound really cool, but have become a real game-changer, especially for sequence modeling like machine translation or text generation. The main driver behind transformers is the attention mechanism, which we will explore in detail. Some even say "Attention is all you need" - Let me know in the comments, if you agree with this statement, or rather with my reference. Complete Playlist: https://www.youtube.com/watch?v=vlTnIjhhmzA&list=PLNsFwZQ_pkE8H1o874cZbiwnNRJ6hCDJI 00:00:00 - Start 00:00:40 - Recap 00:03:53 - Why Transformers? 00:09:36 - What is a Transformer? 00:11:10 - Input Embedding and Positional Encoding 00:14:20 - Attention 00:17:12 - Implementing Attention 00:21:34 - Masked Attention 00:24:06 - Cross-Attention 00:25:51 - The final layers 00:27:38 - Architecture Variations 00:30:11 - Transformers for Object Detection 00:36:28 - The Detection Transformer (DETR) 00:38:02 - Bipartite Matching Loss 00:47:33 - Object Queries 00:55:14 - Advances in Transformers 01:01:48 - Summary == Literature == 1. Halthor, Transformer Neural Networks Explained, 2020 2. Vaswani et al., Attention Is All You Need, 2017 3. Carion et al. End-to-End Object Detection with Transformers, 2020 4. Kilcher, End-to-End Object Detection with Transformers (Paper explanation), 2020 5. Pacha et al., A Baseline for General Music Object Detection with Deep Learning, 2018 6. Parmar et al. Image Transformer, 2018 7. Kilcher, Attention Is All You Need (Explained), 2017 8. Phi, Illustrated Guide to Transformers: Step by Step Explanation, 2020 9. Olah et al. Attention and Augmented Recurrent Neural Networks, 2016 10. Peters et al. Deep contextualized word representations, 2018 11. Howard et al. Universal Language Model Fine-tuning for Text Classification, 2018 12. Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018 13. Kitaev et al. Reformer: The Efficient Transformer, 2020 14. Wang et a

Original Description

In this lecture, we're diving into Transformers, which not only sound really cool, but have become a real game-changer, especially for sequence modeling like machine translation or text generation. The main driver behind transformers is the attention mechanism, which we will explore in detail. Some even say "Attention is all you need" - Let me know in the comments, if you agree with this statement, or rather with my reference. Complete Playlist: https://www.youtube.com/watch?v=vlTnIjhhmzA&list=PLNsFwZQ_pkE8H1o874cZbiwnNRJ6hCDJI 00:00:00 - Start 00:00:40 - Recap 00:03:53 - Why Transformers? 00:09:36 - What is a Transformer? 00:11:10 - Input Embedding and Positional Encoding 00:14:20 - Attention 00:17:12 - Implementing Attention 00:21:34 - Masked Attention 00:24:06 - Cross-Attention 00:25:51 - The final layers 00:27:38 - Architecture Variations 00:30:11 - Transformers for Object Detection 00:36:28 - The Detection Transformer (DETR) 00:38:02 - Bipartite Matching Loss 00:47:33 - Object Queries 00:55:14 - Advances in Transformers 01:01:48 - Summary == Literature == 1. Halthor, Transformer Neural Networks Explained, 2020 2. Vaswani et al., Attention Is All You Need, 2017 3. Carion et al. End-to-End Object Detection with Transformers, 2020 4. Kilcher, End-to-End Object Detection with Transformers (Paper explanation), 2020 5. Pacha et al., A Baseline for General Music Object Detection with Deep Learning, 2018 6. Parmar et al. Image Transformer, 2018 7. Kilcher, Attention Is All You Need (Explained), 2017 8. Phi, Illustrated Guide to Transformers: Step by Step Explanation, 2020 9. Olah et al. Attention and Augmented Recurrent Neural Networks, 2016 10. Peters et al. Deep contextualized word representations, 2018 11. Howard et al. Universal Language Model Fine-tuning for Text Classification, 2018 12. Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018 13. Kitaev et al. Reformer: The Efficient Transformer, 2020 14. Wang et a
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

Chapters (17)

Start
0:40 Recap
3:53 Why Transformers?
9:36 What is a Transformer?
11:10 Input Embedding and Positional Encoding
14:20 Attention
17:12 Implementing Attention
21:34 Masked Attention
24:06 Cross-Attention
25:51 The final layers
27:38 Architecture Variations
30:11 Transformers for Object Detection
36:28 The Detection Transformer (DETR)
38:02 Bipartite Matching Loss
47:33 Object Queries
55:14 Advances in Transformers
1:01:48 Summary
Up next
RNNs Explained in 60 Seconds #ai #coding #machinelearning
Ascent
Watch →