LLM Subword Tokenizer Explained: Byte-Pair Encoding (BPE) with HuggingFace and OpenAI

AI Simply Explained with Tiyasa · Beginner ·🧠 Large Language Models ·2mo ago

About this lesson

00:00 Introduction (Quick Recap) 00:13 What is BPE 00:27 Step-by-Step BPE Algorithm Example 01:08 Why BPE Works 02:28 Hugging Face Tokenizer Demo 03:22 OpenAI (TikToken) Demo 03:55 Final Thoughts Tokenization breaks text into pieces — but how does it actually work under the hood? In this video, we dive deep into Byte-Pair Encoding (BPE) — the algorithm behind modern LLM tokenizers. You’ll learn: • The intuition behind BPE • Step-by-step how the algorithm works • How tokens are built from raw text • Why BPE is efficient for large language models • Real implementations using Hugging Face & OpenAI’s tiktoken We’ll also walk through practical demos so you can see exactly how your text gets converted into tokens and numbers. By the end, you’ll have a clear, working understanding of how LLM tokenization actually happens. 👍 Like, Subscribe & Hit the Bell 🔔 🌐 Connect with me: LinkedIn: https://www.linkedin.com/in/tiyasamukherjee Let’s learn and grow together 🚀 #BPE #Tokenization #LLM #GPT #Claude #MachineLearning #GenerativeAI #AIExplained

Original Description

00:00 Introduction (Quick Recap) 00:13 What is BPE 00:27 Step-by-Step BPE Algorithm Example 01:08 Why BPE Works 02:28 Hugging Face Tokenizer Demo 03:22 OpenAI (TikToken) Demo 03:55 Final Thoughts Tokenization breaks text into pieces — but how does it actually work under the hood? In this video, we dive deep into Byte-Pair Encoding (BPE) — the algorithm behind modern LLM tokenizers. You’ll learn: • The intuition behind BPE • Step-by-step how the algorithm works • How tokens are built from raw text • Why BPE is efficient for large language models • Real implementations using Hugging Face & OpenAI’s tiktoken We’ll also walk through practical demos so you can see exactly how your text gets converted into tokens and numbers. By the end, you’ll have a clear, working understanding of how LLM tokenization actually happens. 👍 Like, Subscribe & Hit the Bell 🔔 🌐 Connect with me: LinkedIn: https://www.linkedin.com/in/tiyasamukherjee Let’s learn and grow together 🚀 #BPE #Tokenization #LLM #GPT #Claude #MachineLearning #GenerativeAI #AIExplained
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

📰
I Built an Entire Application with ChatGPT 5.5. That Isn’t What Changed Me.
Building an app with ChatGPT 5.5 was a breakthrough, but the real change came from learning to think with AI
Medium · ChatGPT
 📰
ferrovec: a Tiny Rust HNSW Vector Index That Runs Semantic Search Inside the Browser Tab
Learn how to use ferrovec, a tiny Rust HNSW vector index for semantic search in the browser, and why it matters for serverless applications
Dev.to · AI Explore
 📰
Beyond the Prompt: 5 Surprising Lessons in Making LLMs Smarter and More Efficient
Learn 5 surprising lessons to make LLMs smarter and more efficient, including the cost of overthinking and the importance of structured input and reasoning history
Medium · AI
 📰
The Quiet Part of the Mind
Discover how logic and language are processed in different neural areas, and what this means for AI and cognitive science
Medium · AI

Chapters (7)

Introduction (Quick Recap)
0:13 What is BPE
0:27 Step-by-Step BPE Algorithm Example
1:08 Why BPE Works
2:28 Hugging Face Tokenizer Demo
3:22 OpenAI (TikToken) Demo
3:55 Final Thoughts
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →