Compressing Large Language Models (LLMs) | w/ Python Code

Shaw Talebi · Beginner ·🧠 Large Language Models ·1y ago

Key Takeaways

Demonstrates compressing large language models using quantization, pruning, and knowledge distillation with Python code

Original Description

🤝 Work with me: https://aibuilder.academy/yt/FLkUOkeMd5M 🚀 Ship AI apps in weeks, not months: https://aibuilder.academy/courses/yt/FLkUOkeMd5M Here, I discuss 3 ways to do model compression on LLMs (i.e. Quantization, Pruning, and Knowledge Distillation/Model Distillation) with example Python code. Resources: 📰 Blog: https://medium.com/towards-data-science/compressing-large-language-models-llms-9f406eea5b5e?sk=2e8a9606c78e60eab725c9a5d4170299 🎥 Training the Teacher Model: https://youtu.be/4QHg8Ix8WWQ 💻 GitHub Repo: https://github.com/ShawhinT/YouTube-Blog/tree/main/LLMs/model-compression 👩‍🏫 Teacher Model: https://huggingface.co/shawhin/bert-phishing-classifier_teacher 🧑‍🎓 Student Model: https://huggingface.co/shawhin/bert-phishing-classifier_student 👾 4-bit Student Model: https://huggingface.co/shawhin/bert-phishing-classifier_student_4bit 💿 Dataset: https://huggingface.co/datasets/shawhin/phishing-site-classification References: [1] https://arxiv.org/abs/2001.08361 [2] https://arxiv.org/abs/1710.09282 [3] https://machinelearning.apple.com/research/model-compression-in-practice [4] https://arxiv.org/abs/1710.09282 [5] https://arxiv.org/abs/2308.07633 [6] https://arxiv.org/abs/2402.17764 [7] https://arxiv.org/abs/1710.01878 [8] https://arxiv.org/abs/1503.02531 [9] https://crfm.stanford.edu/2023/03/13/alpaca.html [10] https://arxiv.org/abs/2305.14314 [11] https://www.researchgate.net/publication/372248458_ChatGPT_in_the_Age_of_Generative_AI_and_Large_Language_Models_A_Concise_Survey Intro - 0:00 "Bigger is Better" - 0:40 The Problem - 1:35 Model Compression - 2:14 1) Quantization - 3:11 2) Pruning - 5:44 3) Knowledge Distillation - 8:04 Example: Compressing a model with KD + Quantization - 11:10
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from Shaw Talebi · Shaw Talebi · 0 of 60

← Previous Next →
1 biometricDashboard2 DEMO
biometricDashboard2 DEMO
Shaw Talebi
 2 biometricDahboard3 DEMO
biometricDahboard3 DEMO
Shaw Talebi
 3 Time Series, Signals, & the Fourier Transform | Introduction
Time Series, Signals, & the Fourier Transform | Introduction
Shaw Talebi
 4 The Fast Fourier Transform | How does it (actually) work?
The Fast Fourier Transform | How does it (actually) work?
Shaw Talebi
 5 The Wavelet Transform | Introduction & Example Code
The Wavelet Transform | Introduction & Example Code
Shaw Talebi
 6 Principal Component Analysis (PCA) | Introduction & Example (Python) Code
Principal Component Analysis (PCA) | Introduction & Example (Python) Code
Shaw Talebi
 7 Independent Component Analysis (ICA) | EEG Analysis Example Code
Independent Component Analysis (ICA) | EEG Analysis Example Code
Shaw Talebi
 8 Kmeans-based Blink Detecter DEMO
Kmeans-based Blink Detecter DEMO
Shaw Talebi
 9 Shit Happens, Stay Solution Oriented
Shit Happens, Stay Solution Oriented
Shaw Talebi
 10 Why Conflict Is Good & How You Can Use It
Why Conflict Is Good & How You Can Use It
Shaw Talebi
 11 Causality: An Introduction | How (naive) statistics can fail us
Causality: An Introduction | How (naive) statistics can fail us
Shaw Talebi
 12 Causal Inference | Answering causal questions
Causal Inference | Answering causal questions
Shaw Talebi
 13 Causal Discovery | Inferring causality from observational data
Causal Discovery | Inferring causality from observational data
Shaw Talebi
 14 How to Be Antifragile | 7 Practical Tips
How to Be Antifragile | 7 Practical Tips
Shaw Talebi
 15 Multi-kills: How to Do More With Less (no, not by multi-tasking)
Multi-kills: How to Do More With Less (no, not by multi-tasking)
Shaw Talebi
 16 Topological Data Analysis (TDA) | An introduction
Topological Data Analysis (TDA) | An introduction
Shaw Talebi
 17 The Mapper Algorithm | Overview & Python Example Code
The Mapper Algorithm | Overview & Python Example Code
Shaw Talebi
 18 Persistent Homology | Introduction & Python Example Code
Persistent Homology | Introduction & Python Example Code
Shaw Talebi
 19 What Is Data Science & How To Start? | A Beginner's Guide
What Is Data Science & How To Start? | A Beginner's Guide
Shaw Talebi
 20 How to do MORE with LESS - multikills
How to do MORE with LESS - multikills
Shaw Talebi
 21 Causal Effects | An introduction
Causal Effects | An introduction
Shaw Talebi
 22 Causal Effects via Propensity Scores | Introduction & Python Code
Causal Effects via Propensity Scores | Introduction & Python Code
Shaw Talebi
 23 Causal Effects via the Do-operator | Overview & Example
Causal Effects via the Do-operator | Overview & Example
Shaw Talebi
 24 Causal Effects via DAGs | How to Handle Unobserved Confounders
Causal Effects via DAGs | How to Handle Unobserved Confounders
Shaw Talebi
 25 Smoothing Crypto Time Series with Wavelets | Real-world Data Project
Smoothing Crypto Time Series with Wavelets | Real-world Data Project
Shaw Talebi
 26 Causal Effects via Regression w/ Python Code
Causal Effects via Regression w/ Python Code
Shaw Talebi
 27 5 Reasons Why Every Data Scientist Should Consider Freelancing
5 Reasons Why Every Data Scientist Should Consider Freelancing
Shaw Talebi
 28 An Introduction to Decision Trees | Gini Impurity & Python Code
An Introduction to Decision Trees | Gini Impurity & Python Code
Shaw Talebi
 29 10 Decision Trees are Better Than 1 | Random Forest & AdaBoost
10 Decision Trees are Better Than 1 | Random Forest & AdaBoost
Shaw Talebi
 30 Dimensionality Reduction & Segmentation with Decision Trees | Python Code
Dimensionality Reduction & Segmentation with Decision Trees | Python Code
Shaw Talebi
 31 How to Make a Data Science Portfolio With GitHub Pages (2025)
How to Make a Data Science Portfolio With GitHub Pages (2025)
Shaw Talebi
 32 My $100,000+ Data Science Resume (what got me hired)
My $100,000+ Data Science Resume (what got me hired)
Shaw Talebi
 33 How to Create a Custom Email Signature in Gmail (2025)
How to Create a Custom Email Signature in Gmail (2025)
Shaw Talebi
 34 I Spent $675.92 Talking to Top Data Scientists on Upwork—Here’s what I learned
I Spent $675.92 Talking to Top Data Scientists on Upwork—Here’s what I learned
Shaw Talebi
 35 Lessons from Spending $675.92 to Talk to Top Data Scientists on Upwork #freelance #datascience
Lessons from Spending $675.92 to Talk to Top Data Scientists on Upwork #freelance #datascience
Shaw Talebi
 36 A Practical Introduction to Large Language Models (LLMs)
A Practical Introduction to Large Language Models (LLMs)
Shaw Talebi
 37 The OpenAI (Python) API | Introduction & Example Code
The OpenAI (Python) API | Introduction & Example Code
Shaw Talebi
 38 The Hugging Face Transformers Library | Example Code + Chatbot UI with Gradio
The Hugging Face Transformers Library | Example Code + Chatbot UI with Gradio
Shaw Talebi
 39 Why I Quit My $150,000 Data Science Job
Why I Quit My $150,000 Data Science Job
Shaw Talebi
 40 Prompt Engineering: How to Trick AI into Solving Your Problems
Prompt Engineering: How to Trick AI into Solving Your Problems
Shaw Talebi
 41 The REALITY of entrepreneurship. #entrepreneurship #startup #smallbusiness
The REALITY of entrepreneurship. #entrepreneurship #startup #smallbusiness
Shaw Talebi
 42 Fine-tuning Large Language Models (LLMs) | w/ Example Code
Fine-tuning Large Language Models (LLMs) | w/ Example Code
Shaw Talebi
 43 How to Build an LLM from Scratch | An Overview
How to Build an LLM from Scratch | An Overview
Shaw Talebi
 44 I Have 90 Days to Make $10k/mo—Here's my plan
I Have 90 Days to Make $10k/mo—Here's my plan
Shaw Talebi
 45 I Spent $716.46 Talking to Data Scientists on Upwork—Here’s what I learned.
I Spent $716.46 Talking to Data Scientists on Upwork—Here’s what I learned.
Shaw Talebi
 46 Pareto, Power Laws, and Fat Tails
Pareto, Power Laws, and Fat Tails
Shaw Talebi
 47 Do NOT become an entrepreneur #entrepreneurship
Do NOT become an entrepreneur #entrepreneurship
Shaw Talebi
 48 Detecting Power Laws in Real-world Data | w/ Python Code
Detecting Power Laws in Real-world Data | w/ Python Code
Shaw Talebi
 49 How I’d learn data analytics (if I had to start over in 2024) #dataanalytics
How I’d learn data analytics (if I had to start over in 2024) #dataanalytics
Shaw Talebi
 50 4 Ways to Measure Fat Tails with Python (+ Example Code)
4 Ways to Measure Fat Tails with Python (+ Example Code)
Shaw Talebi
 51 Fine-tuning EXPLAINED in 40 sec #generativeai
Fine-tuning EXPLAINED in 40 sec #generativeai
Shaw Talebi
 52 How Much YouTube Paid Me in My First 6 Months of Monetization (as a Data Science Creator)
How Much YouTube Paid Me in My First 6 Months of Monetization (as a Data Science Creator)
Shaw Talebi
 53 5 Questions Every Data Scientist Should Hardcode into Their Brain
5 Questions Every Data Scientist Should Hardcode into Their Brain
Shaw Talebi
 54 AI for Business: A (non-technical) introduction
AI for Business: A (non-technical) introduction
Shaw Talebi
 55 LLMs EXPLAINED in 60 seconds #ai
LLMs EXPLAINED in 60 seconds #ai
Shaw Talebi
 56 3 Ways to Make a Custom AI Assistant | RAG, Tools, & Fine-tuning
3 Ways to Make a Custom AI Assistant | RAG, Tools, & Fine-tuning
Shaw Talebi
 57 What is #ai? — Simply Explained
What is #ai? — Simply Explained
Shaw Talebi
 58 QLoRA—How to Fine-tune an LLM on a Single GPU (w/ Python Code)
QLoRA—How to Fine-tune an LLM on a Single GPU (w/ Python Code)
Shaw Talebi
 59 How to Improve LLMs with RAG (Overview + Python Code)
How to Improve LLMs with RAG (Overview + Python Code)
Shaw Talebi
 60 Text Embeddings, Classification, and Semantic Search (w/ Python Code)
Text Embeddings, Classification, and Semantic Search (w/ Python Code)
Shaw Talebi

Related AI Lessons

⚡
Building HITL Feedback RAG: Embeddings, Retrieval, and Reranking
Learn to build a Human-in-the-Loop (HITL) Feedback RAG system using embeddings, retrieval, and reranking to improve model performance
Medium · AI
 ⚡
Building HITL Feedback RAG: Embeddings, Retrieval, and Reranking
Learn to build a Human-in-the-Loop (HITL) Feedback RAG system using embeddings, retrieval, and reranking to improve LLM performance
Medium · LLM
 ⚡
The 2026 AI Model Release Race: Every Major LLM Launch You Need to Know
Stay updated on the 2026 AI model release race, including major LLM launches like Claude Sonnet 5 and GPT-5.6, to leverage the latest advancements in AI technology
Dev.to AI
 ⚡
Call GPT, Claude, and Gemini from one API key — a 3-step setup
Access GPT, Claude, and Gemini through one API key with a 3-step setup using Modelishub
Dev.to AI
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →