07 Gen AI Interview Preparation: What is Tokenization in LLM

Before an LLM can understand a single word you type, it must first convert that text into tokens. Most beginners assume LLMs read language the way humans do — but that assumption leads to a lot of confusion in interviews and in production. In this video, we break down tokenization from the ground up in a structured interview Q&A format built for AI Engineer and GenAI interview preparation. We cover what tokenization is and why raw text cannot be fed directly into a model, the complete pipeline from text to tokens to numbers to model output, the difference between word-level and subword tokeniz…