I Visualized a Decoder-Only Transformer

I traced a single token through a decoder-only Transformer—end to end—so you can finally “see” what happens inside an LLM when it generates the next token. We follow one token from tokenization → embeddings + positional info → LayerNorm → multi-head self-attention (Q/K/V, causal mask, softmax) → residual connections → MLP → logits → sampling, and then show how KV cache speeds up the next step of generation. If you’ve ever wondered how ChatGPT picks the next token, this is the complete, visual walkthrough—every step, no hand-waving. transformer token journey one token decoder-only transformer how transformers work how llms work llm inference autoregressive decoding next token prediction tokenization byte pair encoding sentencepiece tokenizer embedding layer positional embeddings rotary positional embeddings layer normalization self-attention causal self-attention attention mechanism multi-head attention qkv scaled dot-product attention softmax attention residual connection feedforward network mlp transformer logits lm head sampling temperature sampling top-k sampling top-p sampling kv cache key value cache transformer architecture gpt architecture decoder transformer explained attention vs mlp token vs word llm tokenization explained transformer forward pass how chatgpt generates text visual explanation transformers manim transformer llm visualization machine learning deep learning natural language processing 00:00 Breaking Images into Patches 00:42 Flattening and Linear Projection 01:23 2D Structure and Positional Embeddings 02:14 The CLS (Classification) Token 02:54 Bidirectional Self-Attention Mechanism 03:57 Residual Connections, Layer Norm, and MLPs 04:57 Stacking Layers: From Raw Pixels to High-Level Semantics 05:45 How the CLS Token Aggregates Global Information 06:18 Final Classification and Softmax 06:51 ViTs vs. CNNs (Convolutional Neural Networks) 07:12 Evolution of ViTs: Swin Transformers and DeiT