Tokenisation Explained: How AI Actually Reads Text

This video describes the fundamental process through which artificial intelligence/ Gen AI interprets and generates human language. Rather than reading complete sentences, AI systems break text into tokens, which can represent whole words, sub-words, or even punctuation marks.