๐‹๐‹๐Œ ๐“๐จ๐ค๐ž๐ง๐ข๐ณ๐š๐ญ๐ข๐จ๐ง (AI) Explained: How ChatGPT Understands Text

Data Engineering ยท Beginner ยท๐Ÿ”„ Data Engineering ยท11mo ago

About this lesson

In this video, I explain what tokens are in large language models (LLMs) like ChatGPT and GPT-4. Youโ€™ll learn how tokenizers work, what vocab is, how token IDs are assigned, and what happens when a word isnโ€™t in the vocab. Includes a live Python code demo too! โœ… Topics covered: What is a token in AI What is a tokenizer and how it works Vocab and token ID explained Token splitting logic with examples Code demo using GPT tokenizer ๐Ÿ“Œ Subscribe for more AI + NLP tutorials! ๐€๐ฅ๐ฅ ๐…๐ซ๐ž๐ž ๐‚๐จ๐ฎ๐ซ๐ฌ๐ž๐ฌ ----------------------------- ๐†๐ž๐ง ๐€๐ˆ ๐๐ฅ๐š๐ฒ ๐‹๐ข๐ฌ๐ญ - https://bit.ly/3EmIqn9 ๐๐ข๐  ๐ƒ๐š๐ญ๐š ๐…๐ฎ๐ฅ๐ฅ ๐‚๐จ๐ฎ๐ซ๐ฌ๐ž ๐„๐ง๐ ๐ฅ๐ข๐ฌ๐ก - https://youtu.be/Tyg1FVNq40g ๐’๐๐‹ ๐Œ๐š๐ฌ๐ญ๐ž๐ซ ๐‚๐ฅ๐š๐ฌ๐ฌ ๐๐ฅ๐š๐ฒ๐ฅ๐ข๐ฌ๐ญ - https://bit.ly/3WikqGI ๐ƒ๐š๐ญ๐š ๐„๐ง๐ ๐ข๐ง๐ž๐ž๐ซ๐ข๐ง๐  ๐๐ซ๐จ๐ฃ๐ž๐œ๐ญ๐ฌ ๐๐ฅ๐š๐ฒ๐‹๐ข๐ฌ๐ญ - https://bit.ly/3DxUkKb ๐ƒ๐š๐ญ๐š ๐„๐ง๐ ๐ข๐ง๐ž๐ž๐ซ๐ข๐ง๐  ๐‰๐จ๐› ๐๐ฅ๐š๐ฒ๐‹๐ข๐ฌ๐ญ - https://bit.ly/4agAfEq ๐’๐๐‹ ๐ˆ๐ง๐ญ๐ž๐ซ๐ฏ๐ข๐ž๐ฐ ๐๐ฎ๐ž๐ฌ๐ญ๐ข๐จ๐ง ๐„๐ง๐ ๐ฅ๐ข๐ฌ๐ก - https://bit.ly/4e0sXFS ๐๐ฒ๐ญ๐ก๐จ๐ง ๐๐ซ๐จ๐ฃ๐ž๐œ๐ญ ๐•๐ข๐๐ž๐จ๐ฌ - https://bit.ly/4iJStRQ ๐’๐จ๐œ๐ข๐š๐ฅ๐ฌ ๐ŸŽฅ๐˜๐จ๐ฎ๐“๐ฎ๐›๐ž - https://www.youtube.com/@dataengineeringvideos ๐Ÿ“ธ๐ˆ๐ง๐ฌ๐ญ๐š๐ ๐ซ๐š๐ฆ - https://instagram.com/thedatatech.in ๐Ÿ’ผ๐‹๐ข๐ง๐ค๐ž๐๐ˆ๐ง - https://www.linkedin.com/in/sbgowtham/ ๐ŸŒ๐–๐ž๐›๐ฌ๐ข๐ญ๐ž - https://codewithgowtham.blogspot.com ๐Ÿ’ป๐†๐ข๐ญ๐‡๐ฎ๐› - http://github.com/Gowthamdataengineer ๐Ÿ’ฌ๐–๐ก๐š๐ญ๐ฌ ๐€๐ฉ๐ฉ - https://lnkd.in/g5JrHw8q ๐Ÿ“ง๐„๐ฆ๐š๐ข๐ฅ - atozknowledge.com@gmail.com ๐Ÿ“ฑ๐€๐ฅ๐ฅ ๐Œ๐ฒ ๐’๐จ๐œ๐ข๐š๐ฅ๐ฌ - https://lnkd.in/gf8k3aCH Technology in Tamil & English #DataEngineering #BigData #DataPipeline #ETL #DataProcessing #DataScience #DataAnalytics #DataWrangling #DataOps #DataArchitecture #DataIntegration #DataTransformation #DataStorage #DataManagement #DataPlatform #CloudDataEngineering #AWS #Azure #GCP #DataCloud #CloudComputing #CloudDataPipeline #DataStreaming #Kafka #Spark #Hadoop #NoSQL #DataModeling #DataGovernance #DataLake #DataWarehouse #Redshift #BigQuery #Snowflake #Da

Original Description

In this video, I explain what tokens are in large language models (LLMs) like ChatGPT and GPT-4. Youโ€™ll learn how tokenizers work, what vocab is, how token IDs are assigned, and what happens when a word isnโ€™t in the vocab. Includes a live Python code demo too! โœ… Topics covered: What is a token in AI What is a tokenizer and how it works Vocab and token ID explained Token splitting logic with examples Code demo using GPT tokenizer ๐Ÿ“Œ Subscribe for more AI + NLP tutorials! ๐€๐ฅ๐ฅ ๐…๐ซ๐ž๐ž ๐‚๐จ๐ฎ๐ซ๐ฌ๐ž๐ฌ ----------------------------- ๐†๐ž๐ง ๐€๐ˆ ๐๐ฅ๐š๐ฒ ๐‹๐ข๐ฌ๐ญ - https://bit.ly/3EmIqn9 ๐๐ข๐  ๐ƒ๐š๐ญ๐š ๐…๐ฎ๐ฅ๐ฅ ๐‚๐จ๐ฎ๐ซ๐ฌ๐ž ๐„๐ง๐ ๐ฅ๐ข๐ฌ๐ก - https://youtu.be/Tyg1FVNq40g ๐’๐๐‹ ๐Œ๐š๐ฌ๐ญ๐ž๐ซ ๐‚๐ฅ๐š๐ฌ๐ฌ ๐๐ฅ๐š๐ฒ๐ฅ๐ข๐ฌ๐ญ - https://bit.ly/3WikqGI ๐ƒ๐š๐ญ๐š ๐„๐ง๐ ๐ข๐ง๐ž๐ž๐ซ๐ข๐ง๐  ๐๐ซ๐จ๐ฃ๐ž๐œ๐ญ๐ฌ ๐๐ฅ๐š๐ฒ๐‹๐ข๐ฌ๐ญ - https://bit.ly/3DxUkKb ๐ƒ๐š๐ญ๐š ๐„๐ง๐ ๐ข๐ง๐ž๐ž๐ซ๐ข๐ง๐  ๐‰๐จ๐› ๐๐ฅ๐š๐ฒ๐‹๐ข๐ฌ๐ญ - https://bit.ly/4agAfEq ๐’๐๐‹ ๐ˆ๐ง๐ญ๐ž๐ซ๐ฏ๐ข๐ž๐ฐ ๐๐ฎ๐ž๐ฌ๐ญ๐ข๐จ๐ง ๐„๐ง๐ ๐ฅ๐ข๐ฌ๐ก - https://bit.ly/4e0sXFS ๐๐ฒ๐ญ๐ก๐จ๐ง ๐๐ซ๐จ๐ฃ๐ž๐œ๐ญ ๐•๐ข๐๐ž๐จ๐ฌ - https://bit.ly/4iJStRQ ๐’๐จ๐œ๐ข๐š๐ฅ๐ฌ ๐ŸŽฅ๐˜๐จ๐ฎ๐“๐ฎ๐›๐ž - https://www.youtube.com/@dataengineeringvideos ๐Ÿ“ธ๐ˆ๐ง๐ฌ๐ญ๐š๐ ๐ซ๐š๐ฆ - https://instagram.com/thedatatech.in ๐Ÿ’ผ๐‹๐ข๐ง๐ค๐ž๐๐ˆ๐ง - https://www.linkedin.com/in/sbgowtham/ ๐ŸŒ๐–๐ž๐›๐ฌ๐ข๐ญ๐ž - https://codewithgowtham.blogspot.com ๐Ÿ’ป๐†๐ข๐ญ๐‡๐ฎ๐› - http://github.com/Gowthamdataengineer ๐Ÿ’ฌ๐–๐ก๐š๐ญ๐ฌ ๐€๐ฉ๐ฉ - https://lnkd.in/g5JrHw8q ๐Ÿ“ง๐„๐ฆ๐š๐ข๐ฅ - atozknowledge.com@gmail.com ๐Ÿ“ฑ๐€๐ฅ๐ฅ ๐Œ๐ฒ ๐’๐จ๐œ๐ข๐š๐ฅ๐ฌ - https://lnkd.in/gf8k3aCH Technology in Tamil & English #DataEngineering #BigData #DataPipeline #ETL #DataProcessing #DataScience #DataAnalytics #DataWrangling #DataOps #DataArchitecture #DataIntegration #DataTransformation #DataStorage #DataManagement #DataPlatform #CloudDataEngineering #AWS #Azure #GCP #DataCloud #CloudComputing #CloudDataPipeline #DataStreaming #Kafka #Spark #Hadoop #NoSQL #DataModeling #DataGovernance #DataLake #DataWarehouse #Redshift #BigQuery #Snowflake #Da
Watch on YouTube โ†— (saves to browser)
Sign in to unlock AI tutor explanation ยท โšก30

Related Reads

๐Ÿ“ฐ
I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer
Learn how to build a production-ready ETL pipeline with Python, Docker, PostgreSQL, and Kestra by thinking like a data engineer
Towards Data Science
 ๐Ÿ“ฐ
JuiceFS Sync for PB-Scale Data Transfers: Resumable Sync, Encryption, and Bandwidth Control
Learn how to efficiently transfer large volumes of data using JuiceFS Sync, which offers resumable sync, encryption, and bandwidth control, ideal for PB-scale data transfers.
Dev.to AI
 ๐Ÿ“ฐ
How Airflow is using AI to make data engineering more resilient, not more complex
Airflow uses AI to make data engineering more resilient by detecting data drift, resuming failed pipelines, and fixing issues automatically, reducing complexity and improving reliability.
Medium ยท AI
 ๐Ÿ“ฐ
What Can We Do When Memory Becomes the New Bottleneck in Data Engineering?
Learn how to overcome memory bottlenecks in data engineering using Pandas chunking, Dask, and Polars, and why it matters for processing large datasets
Towards Data Science
Up next
A Moment Frozen in Time | Arnav Iyengar | TEDxJenks Youth
TEDx Talks
Watch โ†’