Transformer Architecture

Machine Learning Studio · Beginner ·🧠 Large Language Models ·3y ago

About this lesson

Full architecture of the transformer model as published in "Attention Is All You Need" by Vaswani et al. In this video, I will describe the encoder-decoder transformer architecture , looking into the the details of their sublayers, the position embedding, residual connection, and the feed-forward network. I will wrap-up the video with an important note on training the transformers, and that is to start with a very small learning-rate, and increase it gradually with a warmup stage. ========= Errata ========= Typo in the equation for the Feed-Forward Network. The max function that represent ReLU activation is missing a 0. The correct formula is as follows: max(0, x W1 + b1)W2 + b2

Original Description

Full architecture of the transformer model as published in "Attention Is All You Need" by Vaswani et al. In this video, I will describe the encoder-decoder transformer architecture , looking into the the details of their sublayers, the position embedding, residual connection, and the feed-forward network. I will wrap-up the video with an important note on training the transformers, and that is to start with a very small learning-rate, and increase it gradually with a warmup stage. ========= Errata ========= Typo in the equation for the Feed-Forward Network. The max function that represent ReLU activation is missing a 0. The correct formula is as follows: max(0, x W1 + b1)W2 + b2
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

📰
Beyond the Hype: Why Google’s Open Knowledge Format is Not the "RAG Killer"
Learn why Google's Open Knowledge Format isn't a replacement for RAG and how to build a dual-engine context system in .NET 9
Medium · AI
 📰
The Kimi K3 Shockwave: How One Open-Weight Model Rewired the Industry in 48 Hours
Learn how Moonshot AI's open-weight model, Kimi K3, disrupted the industry in 48 hours and understand the implications for AI development
Medium · AI
 📰
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics
Learn to build production-grade LLM evaluation pipelines to catch hallucinations before deployment
Dev.to AI
 📰
Open-Weight LLM API Integration: A Developer's Guide to Building with Accessible AI
Learn to integrate open-weight LLM APIs for flexible and cost-effective AI development
Dev.to AI
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →