Tokenization Explained Simply | How AI Reads Text

Key Takeaways

In this lecture, we'll explore tokenization. The process of converting text into smaller units called tokens before sending them to a language model. Let's start with a simple example. Suppose we have a sentence as input. The tokenizer's job is to break that sentence into tokens. Each token can represent a word, part of a word, or even a single character depending on the tokenizer's design. Imagine the model allows a maximum token length of four characters. A short word like this fits within that limit, but a longer word such as wards might be split into smaller parts. For instance, war and Diaz. The tokenizer performs this division automatically based on the character limit. Once the sentence is tokenized, the tokens are converted into numeric representations known as embeddings. Neural networks can only process numbers, not raw text. So this conversion is a crucial step. These embeddings are then passed to the language model, which may use attention or mass attention mechanisms to analyze the relationships between tokens and understand their context. To summarize, the flow text is tokenized into smaller segments transformed into numerical vectors and then processed by the model to find patterns and predict the next word in a sequence. Now let's look at a few variations of tokenization. If the tokenizer divides the sentence by words, we call it word level tokenization. However, many modern models use subword tokenization where longer words are split into smaller units based on length limits or frequent character patterns. For instance, a word like preceded might be divided into two smaller subword tokens. There's also character level tokenization where each character is treated as an individual token. Some systems even go further and use bite level tokenization, breaking text into bytes that can represent any character, symbol or space. Different models use different tokenization methods. Subword tokenization is the most common approach in large language models because it balances vocabulary size and efficiency. Character and bite level methods are also used when handling diverse languages or symbols. So tokenization converts text into manageable numerical pieces that models can understand forming the foundation for embeddings and ultimately language generation.